开启掘金成长之旅！这是我参与「掘金日新计划 · 2 月更文挑战」的第 5 天，点击查看活动详情
MongoDB 单节点迁移副本集高可用集群
背景
因为历史原因，公司某内部系统使用单节点部署 MongoDB，随着系统的业务增长，考虑到内部系统对于公司的重要性，避免由于单节点数据库故障造成系统停用，所以需要搭建MongoDB 副本集，通过把原有数据导入到新集群，并将系统的数据操作请求一同切换，保证数据库和业务系统的高可用。
为验证方案可行性，以下均为模拟操作。
基本信息
- 数据库版本：mongodb-3.0.7
- 服务器信息：
| 编号 | IP地址 | 备注 |
|---|---|---|
| 1 | 192.168.56.102:27017 | 原单节点 |
| 2 | 192.168.56.103:27017 | 新集群主节点 |
| 3 | 192.168.56.104:27017 | 新集群从节点1 |
| 4 | 192.168.56.105:27017 | 新集群从节点2 |
-
MongoDB 副本集高可用架构
-
MongoDB 自动故障转移示意图
迁移思路
- 通过三台新的服务器搭建 MongoDB 副本集。
- 原单节点 MongoDB 备份数据（业务系统需要停止写入，保证数据一致性）。
- 全量备份数据导入新 MongoDB 副本集群。
- 验证新集群复制功能
- 验证新集群自动故障转移功能。
- 业务系统切换数据库地址完成迁移。
操作过程
1. 准备阶段 在单节点 MongoDB 数据库上，创建测试数据（192.168.56.102）
[root@master-2 bin]# ./mongo
MongoDB shell version: 3.0.7
connecting to: test
> for(var i=0;i<60000; i++)db.test.insert({"name": "lxc"+i, "age": 22, "sex": "man"})
> db.test.count()
60000
通过 Mongo Shell 执行 for 循环，在 test.test 集合中插入 6w 条文档
2. 搭建 MongoDB 副本集 2.1 安装 MongoDB（下载并解压业务系统提供的MongoDB软件包,192.168.56.[103-105]三台都操作）
$ unzip mongodb.zip
$ mv mongodb /usr/local/
$ chown -R mongodb.mongodb /usr/local/mongodb
2.2 关闭 THP（三台都设置）
$ echo "never" >> /sys/kernel/mm/transparent_hugepage/enabled
$ echo "never" >> /sys/kernel/mm/transparent_hugepage/defrag
# 加入 /etc/rc.local（三台都设置）
$ echo "never" >> /sys/kernel/mm/transparent_hugepage/enabled
$ echo "never" >> /sys/kernel/mm/transparent_hugepage/defrag
2.3 编辑 MongoDB 配置（bind 参数对应填入各服务器IP地址）
$ vim /usr/local/mongodb/conf/mongodb.conf
storage:
dbPath: "/usr/local/mongodb/data"
indexBuildRetry: true
journal:
enabled: true
directoryPerDB: true
syncPeriodSecs: 60
engine: wiredTiger
wiredTiger:
engineConfig:
cacheSizeGB: 1
journalCompressor: zlib
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
systemLog:
destination: file
path: "/usr/local/mongodb/logs/mongodb.log"
logAppend: true
timeStampFormat: iso8601-local
processManagement:
fork: true
net:
bindIp: 192.168.56.<103-105>,127.0.0.1 # 改成各自IP地址
port: 27017
replication:
oplogSizeMB: 4096
replSetName: rs
- 尽量调大
oplogSizeMB参数，确保 oplog 有足够大的空间来保存写操作，以便于节点异常重启后能够顺利的恢复数据。oplog 类似于 MySQL 的 redo log。- mongoDB v3.0 配置文件参考
2.4 启动 mongod（三台都设置）
$ numactl --interleave=all /usr/local/mongodb/bin/mongod --config /usr/local/mongodb/conf/mongodb.conf
2.5 登录任意一个节点，创建副本集配置
[root@master-3 bin]# ./mongo
MongoDB shell version: 3.0.7
connecting to: test
Welcome to the MongoDB shell.
> use admin
switched to db admin
# 自定义副本集配置，rs 为副本集群的名称。
> config={"_id":"rs","members":[{"_id":0,"host":"192.168.56.103:27017"},{"_id":1,"host":"192.168.56.104:27017"},{"_id":2,"host":"192.168.56.105:27017"}]}
{
"_id" : "rs",
"members" : [
{
"_id" : 0,
"host" : "192.168.56.103:27017"
},
{
"_id" : 1,
"host" : "192.168.56.104:27017"
},
{
"_id" : 2,
"host" : "192.168.56.105:27017"
}
]
}
# 执行初始化
> rs.initiate(config)
{ "ok" : 1 }
# 查看副本集状态
rs:OTHER> rs.status()
{
"set" : "rs",
"date" : ISODate("2022-12-05T12:06:36.459Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.56.103:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 5992,
"optime" : Timestamp(1670241986, 1),
"optimeDate" : ISODate("2022-12-05T12:06:26Z"),
"electionTime" : Timestamp(1670241988, 1),
"electionDate" : ISODate("2022-12-05T12:06:28Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 1,
"name" : "192.168.56.104:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 9,
"optime" : Timestamp(1670241986, 1),
"optimeDate" : ISODate("2022-12-05T12:06:26Z"),
"lastHeartbeat" : ISODate("2022-12-05T12:06:34.933Z"),
"lastHeartbeatRecv" : ISODate("2022-12-05T12:06:35.331Z"),
"pingMs" : 0,
"configVersion" : 1
},
{
"_id" : 2,
"name" : "192.168.56.105:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 9,
"optime" : Timestamp(1670241986, 1),
"optimeDate" : ISODate("2022-12-05T12:06:26Z"),
"lastHeartbeat" : ISODate("2022-12-05T12:06:34.933Z"),
"lastHeartbeatRecv" : ISODate("2022-12-05T12:06:35.367Z"),
"pingMs" : 0,
"configVersion" : 1
}
],
"ok" : 1
}
经过选举投票，192.168.56.103 成为主节点（PRIMARY），其余两台成为从节点（SECONDARY）
3. 单节点 MongoDB 数据备份
# 使用数据库自带的 mongodump 工具进行备份
[root@master-2 bin]# ./mongodump --host 192.168.56.102 --port 27017
2022-12-02T23:46:30.507+0800 writing test.system.indexes to dump/test/system.indexes.bson
2022-12-02T23:46:30.507+0800 writing test.test to dump/test/test.bson
2022-12-02T23:46:30.725+0800 writing test.test metadata to dump/test/test.metadata.json
2022-12-02T23:46:30.726+0800 done dumping test.test (60000 documents)
# mongodump 工具会将数据备份到当前的 dump 目录
[root@master-2 bin]# ls
dump
- 由于备份会对数据库造成性能影响，对于单节点的情况，应选择空闲时段或停机进行全量备份。（上面模拟空闲时段系统停止读写后进行备份，停机备份则可使用
mongodump --dbpath /data/db）- 副本集服务器在备份时，需要添加
--oplog选项，来跟踪备份期间的增量数据，并在恢复备份时能进行重放操作，可从源服务器得到某一个时刻的一致性快照。
4. 备份数据导入副本集 4.1 找到副本集群的主节点（192.168.56.103）
# 验证此节点是主节点（PRIMARY）
rs:PRIMARY> rs.isMaster()
{
"setName" : "rs",
"setVersion" : 1,
"ismaster" : true,
"secondary" : false,
"hosts" : [
"192.168.56.103:27017",
"192.168.56.104:27017",
"192.168.56.105:27017"
],
"primary" : "192.168.56.103:27017",
"me" : "192.168.56.103:27017",
"electionId" : ObjectId("638ddec45fea951e7c9bd755"),
"maxBsonObjectSize" : 16777216,
"maxMessageSizeBytes" : 48000000,
"maxWriteBatchSize" : 1000,
"localTime" : ISODate("2022-12-05T15:48:18.928Z"),
"maxWireVersion" : 3,
"minWireVersion" : 0,
"ok" : 1
}
4.2 从单节点服务器，将备份数据导入副本集群（192.168.56.102）
# 使用数据库自带的 mongorestore 工具进行数据恢复导入
[root@master-2 bin]# ./mongorestore -h 192.168.56.103:27017 --dir dump
2022-12-02T23:52:56.739+0800 building a list of dbs and collections to restore from dump dir
2022-12-02T23:52:56.740+0800 reading metadata file from dump/test/test.metadata.json
2022-12-02T23:52:56.740+0800 restoring test.test from file dump/test/test.bson
2022-12-02T23:52:59.739+0800 [###########.............] test.test 1.9 MB/3.8 MB (49.9%)
2022-12-02T23:53:01.906+0800 restoring indexes for collection test.test from metadata
2022-12-02T23:53:01.907+0800 finished restoring test.test (60000 documents)
2022-12-02T23:53:01.907+0800 done
副本集备份数据，在导入恢复时需指定
--oplogReplay获取某一时刻的快照进行重放。（对应 mongodump 的--oplog）
5. 数据验证阶段 5.1 在主节点查看导入的数据，验证导入成功（192.168.56.103）
rs:PRIMARY> show dbs;
local 0.001GB
test 0.001GB
rs:PRIMARY> use test
switched to db test
rs:PRIMARY> db.test.count()
60000
5.2 在从节点查看数据是否复制，验证复制成功（192.168.56.104）
[root@master-4 bin]# ./mongo
rs:SECONDARY> show dbs;
2022-12-02T00:03:06.934+0800 E QUERY Error: listDatabases failed:{ "note" : "from execCommand", "ok" : 0, "errmsg" : "not master" }
at Error (<anonymous>)
at Mongo.getDBs (src/mongo/shell/mongo.js:47:15)
at shellHelper.show (src/mongo/shell/utils.js:630:33)
at shellHelper (src/mongo/shell/utils.js:524:36)
at (shellhelp2):1:1 at src/mongo/shell/mongo.js:47
rs:SECONDARY> rs.slaveOk() # 允许从节点只读
rs:SECONDARY> show dbs;
local 0.001GB
test 0.001GB
rs:SECONDARY> use test
switched to db test
rs:SECONDARY> db.test.count()
60000
6. 故障转移验证阶段 6.1 在主节点操作关停服务，模拟主节点宕机，触发自动关机故障转移，验证节点选举切换，即 Automatic Failover（192.168.56.103）
[root@master-3 bin]# ./mongo
rs:PRIMARY> use admin
switched to db admin
rs:PRIMARY> db.shutdownServer()
2022-12-03T16:44:49.577+0800 I NETWORK DBClientCursor::init call() failed
server should be down...
2022-12-03T16:44:49.580+0800 I NETWORK trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2022-12-03T16:44:49.580+0800 I NETWORK reconnect 127.0.0.1:27017 (127.0.0.1) ok
2022-12-03T16:44:49.580+0800 I NETWORK DBClientCursor::init call() failed
2022-12-03T16:44:49.584+0800 I NETWORK trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2022-12-03T16:44:49.584+0800 I NETWORK reconnect 127.0.0.1:27017 (127.0.0.1) ok
2022-12-03T16:44:50.367+0800 I NETWORK Socket recv() errno:104 Connection reset by peer 127.0.0.1:27017
2022-12-03T16:44:50.368+0800 I NETWORK SocketException: remote: 127.0.0.1:27017 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:27017]
2022-12-03T16:44:50.368+0800 I NETWORK DBClientCursor::init call() failed
6.2 切换到从节点，查看选举日志，成功将 192.168.56.104 选为主节点（192.168.56.104）
2022-12-03T16:44:51.073+0800 I REPL [ReplicationExecutor] Error in heartbeat request to 192.168.56.103:27017; Location18915 Failed attempt to connect to 192.168.56.103:27017; couldn't connect to server 192.168.56.103:27017 (192.1
68.56.103), connection attempt failed
2022-12-03T16:44:51.073+0800 I REPL [ReplicationExecutor] Standing for election
2022-12-03T16:44:51.074+0800 I REPL [ReplicationExecutor] replSet possible election tie; sleeping 147ms until 2022-12-06T16:44:51.221+0800
2022-12-03T16:44:51.221+0800 I REPL [ReplicationExecutor] Standing for election
2022-12-03T16:44:51.222+0800 I REPL [ReplicationExecutor] replSet info electSelf
2022-12-03T16:44:51.223+0800 I REPL [ReplicationExecutor] received vote: 1 votes from 192.168.56.105:27017
2022-12-03T16:44:51.223+0800 I REPL [ReplicationExecutor] replSet election succeeded, assuming primary role
2022-12-03T16:44:51.223+0800 I REPL [ReplicationExecutor] transition to PRIMARY
2022-12-03T16:44:51.940+0800 I REPL [rsSync] transition to primary complete; database writes are now permitted
2
6.3 查看副本集状态，192.168.56.104 变成主节点，192.168.56.103 health 为 0 ，无法连接
rs:PRIMARY> rs.status()
{
"set" : "rs",
"date" : ISODate("2022-12-06T08:46:03.481Z"),
"myState" : 1,
"members" : [
{
"_id" : 0,
"name" : "192.168.56.103:27017",
"health" : 0,
"state" : 8,
"stateStr" : "(not reachable/healthy)",
"uptime" : 0,
"optime" : Timestamp(0, 0),
"optimeDate" : ISODate("1970-01-01T00:00:00Z"),
"lastHeartbeat" : ISODate("2022-12-06T08:46:03.337Z"),
"lastHeartbeatRecv" : ISODate("2022-12-06T08:44:48.181Z"),
"pingMs" : 0,
"lastHeartbeatMessage" : "Failed attempt to connect to 192.168.56.103:27017; couldn't connect to server 192.168.56.103:27017 (192.168.56.103), connection attempt failed",
"configVersion" : -1
},
{
"_id" : 1,
"name" : "192.168.56.104:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 75039,
"optime" : Timestamp(1670255583, 4114),
"optimeDate" : ISODate("2022-12-05T15:53:03Z"),
"electionTime" : Timestamp(1670316291, 1),
"electionDate" : ISODate("2022-12-06T08:44:51Z"),
"configVersion" : 1,
"self" : true
},
{
"_id" : 2,
"name" : "192.168.56.105:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 74376,
"optime" : Timestamp(1670255583, 4114),
"optimeDate" : ISODate("2022-12-05T15:53:03Z"),
"lastHeartbeat" : ISODate("2022-12-06T08:46:03.222Z"),
"lastHeartbeatRecv" : ISODate("2022-12-06T08:46:03.221Z"),
"pingMs" : 0,
"configVersion" : 1
}
],
"ok" : 1
}
6.4 恢复 192.168.56.103 服务，查看日志已变成副本节点。(192.168.56.103)
[root@master-3 bin]# numactl --interleave=all /usr/local/mongodb/bin/mongod --config /usr/local/mongodb/conf/mongodb.conf
[root@master-3 bin]# less /usr/local/mongodb/mongodb.log
2022-12-06T16:59:38.743+0800 I REPL [ReplicationExecutor] transition to STARTUP2
2022-12-06T16:59:38.743+0800 I REPL [ReplicationExecutor] Starting replication applier threads
2022-12-06T16:59:38.743+0800 I REPL [ReplicationExecutor] transition to RECOVERING
2022-12-06T16:59:38.744+0800 I REPL [ReplicationExecutor] Member 192.168.56.105:27017 is now in state SECONDARY
2022-12-06T16:59:38.744+0800 I REPL [ReplicationExecutor] Member 192.168.56.104:27017 is now in state PRIMARY
2022-12-06T16:59:38.744+0800 I REPL [ReplicationExecutor] transition to SECONDARY
7. 业务系统的数据库地址切换 根据业务系统的代码语言所提供的不同 driver，将数据库调用方式对应修改为副本集形式即可。
结论
在保证数据一致性的前提下，通过将单节点 MongoDB 的备份数据导入新的 MongoDB 副本集，可以确保数据完整，并满足数据库的高可用性需求。
开启掘金成长之旅！这是我参与「掘金日新计划 · 2 月更文挑战」的第 5 天，点击查看活动详情
