MongoDB集群同步如何实现为？
摘要：实现 MongoDB Cluster-to-Cluster 即集群同步的工具是：mongosync
详情可参考如下官方文档：
https:www.mongodb.comzh-cndocscluste
实现 MongoDB Cluster-to-Cluster 即集群同步的工具是：mongosync
详情可参考如下官方文档：
https://www.mongodb.com/zh-cn/docs/cluster-to-cluster-sync/current/quickstart/
以上这个地址的文档一看就是机器翻译的，可能有不恰当的地方，但基本可参考使用。
以下是本次在某项目地配置集群同步的简要步骤，可参考使用。
一、首先在目的集群安装同步工具mongosync
本次使用的是版本匹配的1.5.0版本：
https://fastdl.mongodb.org/tools/mongosync/mongosync-rhel80-x86_64-1.5.0.tgz
以下是我习惯使用的安装脚本：
KDR=/u01
cd ${KDR}
TGZ=mongosync-rhel80-x86_64-1.5.0
#cp_unzip_chown_ln:
cp /u01/nfs/MongoDB/${TGZ}.tgz .
tar zxvf ${TGZ}.tgz
chown -R mongod:mongod ${TGZ}
ln -s ${KDR}/${TGZ}/bin/* /usr/local/bin/
二、集群规划信息配置
在/etc/hosts文件追加host定义(mongodb25~33)，方便用于MongoDB集群同步脚本调用[以下代码已做脱敏处理]
cat >> /etc/hosts << EOF
# 生产集群
2409:xx:1a mongodb25
2409:xx:1b mongodb26
2409:xx:1c mongodb27
# 容灾集群
2409:xx:1d mongodb28
2409:xx:1e mongodb29
2409:xx:1f mongodb30
# 历史数据集群
2409:xx:20 mongodb31
2409:xx:21 mongodb32
2409:xx:22 mongodb33
EOF
三、mongosync 执行
LOGFILE=mongosync.liking.`date +%Y%m%d-%H:%M`.log
mongosync \
--logPath /var/log/mongosync \
--cluster0 mongodb://admin:passwd@mongodb28:20000,mongodb29:20000,mongodb30:20000/?authSource=admin \
--cluster1 mongodb://admin:passwd@mongodb31:20000,mongodb32:20000,mongodb33:20000/?authSource=admin \
--verbosity INFO > ${LOGFILE} 2>&1 &
启用同步服务后，可以在日志目录/var/log/mongosync查看日志信息。
四、POST 同步命令 start
■ 同步整个集群【本次使用】
curl "http://localhost:27182/api/v1/start" -XPOST --data '{
"source": "cluster0",
"destination": "cluster1"
}'
■ 同步指定的数据库
curl "http://localhost:27182/api/v1/start" -XPOST --data '{
"source": "cluster0",
"destination": "cluster1"
"includeNameSpaces": [
{
"database": "flowtest"
}
]
}'
■ 同步指定的集合
curl "http://localhost:27182/api/v1/start" -XPOST --data '{
"source": "cluster0",
"destination": "cluster1",
"includeNameSpaces": [
{
"database": "flowtest",
"collections": [ "activityConfiguration", "history_task", "processConfiguration", "processDataObj", "snapshotResource" ]
}
]
}'
■ 同步时的运维操作
curl "http://localhost:27182/api/v1/progress" -XGET
curl "http://localhost:27182/api/v1/pause" -XPOST --data '{}'
curl "http://localhost:27182/api/v1/resume" -XPOST --data '{}'
五、全量同步日志1
本次全量同步的数据规模如下：
[mongos] testdb> show databases
admin 252.00 KiB
config 35.56 MiB
flowtest 624.74 GiB
同步开始阶段的重要信息如下：
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","uri":"/api/v1/start","method":"POST","body":"{\n \"source\": \"cluster0\",\n \"destination\": \"cluster1\"\n}","clientIP":"::1","traceID":"fec8eb4b-8123-4a33-a1a5-d6cf57b431ea","time":"2024-04-18T16:47:49.511390067+08:00","message":"received request"}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","source":"cluster0","destination":"cluster1","reversible":false,"enableUserWriteBlocking":false,"time":"2024-04-18T16:47:49.511783755+08:00","message":"Start handlercalled"}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","sourceURI":"mongodb28:20000,mongodb29:20000,mongodb30:20000","dstURI":"mongodb31:20000,mongodb32:20000,mongodb33:20000","time":"2024-04-18T16:47:49.511812312+08:00","message":"Creating clients and connecting to clusters."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","clusterType":"src","operation description":"Pinging deployment at mongodb://admin:<REDACTED>@mongodb28:20000,mongodb29:20000,mongodb30:20000/?authSource=admin to establish initial connection.","attemptNumber":0,"durationSoFarSecs":0,"durationLimitSecs":10,"time":"2024-04-18T16:47:49.512143259+08:00","message":"Trying operation."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","clusterType":"dst","operation description":"Pinging deployment at mongodb://admin:<REDACTED>@mongodb31:20000,mongodb32:20000,mongodb33:20000/?authSource=admin to establish initial connection.","attemptNumber":0,"durationSoFarSecs":0,"durationLimitSecs":10,"time":"2024-04-18T16:47:49.530184069+08:00","message":"Trying operation."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","time":"2024-04-18T16:47:49.545274189+08:00","message":"Successfully initialized source and destination clients"}
全量同步完毕阶段的重要日志信息如下：
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"Collection Copy","time":"2024-04-18T20:29:55.31864523+08:00","message":"All Collection Copy components have stopped."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"Collection Copy","time":"2024-04-18T20:29:55.318660079+08:00","message":"Completed writing 6487 / 6487 partitions to destination cluster"}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"CHECKPOINT","totalBytes":2717579843528,"copiedBytes":2717920929608,"time":"2024-04-18T20:29:55.318684889+08:00","message":"Finished collection copy phase."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"CHECKPOINT","time":"2024-04-18T20:29:55.330386515+08:00","message":"Waiting for collection copy phase to complete."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"CHECKPOINT","time":"2024-04-18T20:29:55.351147335+08:00","message":"Running change event application phase."}
{"level":"info","serverID":"bb927128","mongosyncID":"coordinator","componentName":"Change Event Application","time":"2024-04-18T20:29:55.365007664+08:00","message":"Starting index checker service which runs a cycle every 60 seconds."}
以上日志可见：
620G数据，同步耗时3.5小时左右，性能比之前测试稍慢。
再次印证了mongodb本身的压缩算法较好，实际fs层面的数据大小如下：
"totalBytes":2717579843528,"copiedBytes":2717920929608,"time":"2024-04-18T20:29:55.318684889+08:00","message":"Finished collection copy phase."
六、全量同步日志2
当晚实际生产集群全量同步的数据规模如下：
flowtest 684.20 GiB
{"estimatedTotalBytes":2972595055214,"estimatedCopiedBytes":2973012240862}
实际耗时2h55m，可见同样硬件、网络环境夜间性能稍好。
七、思考
通过以上实践，显而易见，MongoDB集群同步技术使得我们后续迁移数据库变得轻而易举，实际割接时间可控制在几分钟以内。