glusterfs集群的搭建和使用这里就不再赘述了 , 可以看以前的教程文档。本文主要聊的是随着服务使用量的增加 , 那么存储集群势必要扩充空间。服务器迁移 , 需要先扩容后缩容等等。正常的扩容有两种, 横向扩容和纵向扩容。横向扩容指的是添加新的节点(物理机/虚拟机)以扩展存储池。纵向扩容时增加brick(逻辑卷)扩展卷容量。所以本文的主旨是聊glusterfs集群的横向优化 ; 扩容和缩容。
这里简单用以下命令演示纵向扩容的大致步骤
# 扩展逻辑卷 ( 在 Brick 所在节点 )
lvextend -L +10G /dev/vg_gluster/brick1
xfs_growfs /data/brick1 # 调整文件系统大小
# 更新卷信息 ( 无需 rebalance )
gluster volume heal vol1 info # 检查状态这里需要说明一个命令:volume rebalance <VOLNAME> {{fix-layout start} | {start [force]|stop|status}} - rebalance operations。volume rebalance 命令用于在 GlusterFS 卷 的存储节点(Brick)发生变更 ( 如扩容或缩容 ) 后，重新平衡数据分布，确保数据均匀分布在所有 Brick 上。
子命令使用详解
fix-layout start作用 : 更新卷的 目录结构元数据 ( 如哈希分布表 ) ，确保新加入的
Brick被正确识别，但 不迁移实际数据。典型场景 :
添加新
Brick后，希望目录结构立即生效，但暂不迁移旧数据。修复因节点变动导致的目录分布错误。
示例 :
gluster volume rebalance vol1 fix-layout start
start [force]作用 : 启动数据迁移，根据新的拓扑结构将文件重新分布到所有
Brick上。force参数 : 强制启动均衡，忽略潜在风险 ( 如节点状态不一致 ) 。
典型场景 :
扩容后需要将数据均匀分布到新旧 Brick。
缩容后需将待移除 Brick 的数据迁移到保留的 Brick。
示例 :
# 正常启动 gluster volume rebalance vol1 start # 强制启动 ( 跳过警告 ) gluster volume rebalance vol1 start force
stop作用 : 终止正在运行的均衡任务，已迁移的数据会保留，未完成的迁移将在下次任务中继续。
典型场景 :
均衡任务占用资源过高，需临时暂停。
发现配置错误需调整后重新启动。
示例 :
gluster volume rebalance vol1 stop
status作用 : 显示当前均衡任务的进度、迁移文件数、耗时等详细信息。
输出字段说明 :
Node: 参与均衡的节点。Rebalanced: 已迁移文件数。Scanned: 已扫描文件数。Status: 任务状态 (in progress/completed/failed) 。
示例 :
gluster volume rebalance vol1 status
操作流程示例
场景 : 扩容后重新平衡数据
添加新
Brick:gluster volume add-brick vol1 gluster-node4:/data/brick4启动均衡任务 :
gluster volume rebalance vol1 start监控进度 :
gluster volume rebalance vol1 status完成后的验证 :
gluster volume status vol1 # 检查所有 Brick 状态 df -h /data/brick* # 查看各 Brick 空间使用 [root@glusterfs01 /]# gluster volume status gv1 Status of volume: gv1 Gluster process TCP Port RDMA Port Online Pid ------------------------------------------------------------------------------ Brick glusterfs01:/data/dis_volume 50440 0 Y 557691 Brick glusterfs02:/data/dis_volume 51265 0 Y 568290 Brick glusterfs03:/data/dis_volume 56503 0 Y 560060 Task Status of Volume gv1 ------------------------------------------------------------------------------ There are no active volume tasks #这里显示的是三个节点的/dev/vdb的磁盘容量 [root@glusterfs01 /]# df -h /data/dis* Filesystem Size Used Avail Use% Mounted on /dev/vdb 100G 747M 100G 1% /data /dev/vdb 100G 747M 100G 1% /data /dev/vdb 100G 747M 100G 1% /data
注意事项与优化
性能影响
I/O 压力 : 均衡期间会大量读写磁盘和网络，建议在业务低峰期执行。
资源预留 : 确保
CPU、内存和网络带宽充足，避免影响生产服务。
参数调优
调整线程数 ( 默认
1线程 ) :gluster volume set vol1 rebalance-threads 4限制带宽 ( 默认无限制 ) :
gluster volume set vol1 network.bandwidth 100MB # 限制为 100MB/s
故障处理
任务中断 : 若均衡意外终止，重新执行
start会继续迁移。日志分析 : 查看
/var/log/glusterfs/rebalance.log定位错误。
1. 分布式卷扩容缩容
这里需要先搭建一个三节点的glusterfs的分布式卷环境用以作为这次分布式卷扩容缩容的实验环境。
[root@glusterfs01 /]# gluster volume create gv1 glusterfs01:/data/dis_volume glusterfs02:/data/dis_volume glusterfs03:/data/dis_volume
volume create: gv1: success: please start the volume to access data
[root@glusterfs01 /]# gluster volume start gv1
volume start: gv1: success
[root@glusterfs01 /]# gluster volume info gv1
Volume Name: gv1
Type: Distribute
Volume ID: d6f3b868-e7d1-4c12-8e80-e59409a8566e
Status: Started
Snapshot Count: 0
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: glusterfs01:/data/dis_volume
Brick2: glusterfs02:/data/dis_volume
Brick3: glusterfs03:/data/dis_volume
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
创建完成之后使用客户端挂载, 同时创建20个文件
[root@client ~]# mount -t glusterfs glusterfs01:gv1 /mnt/distributed
[root@client ~]# touch /mnt/distributed/{1..20}.txt之后在三个节点中查看文件分布
[root@glusterfs01 /]# ls /data/dis_volume/
10.txt 11.txt 12.txt 14.txt 15.txt 16.txt 18.txt 1.txt 20.txt 2.txt 3.txt 6.txt
[root@glusterfs02 ~]# ls /data/dis_volume
13.txt 17.txt 19.txt 4.txt 8.txt
[root@glusterfs03 ~]# ls /data/dis_volume/
5.txt 7.txt 9.txt1.1 分布式卷扩容
对集群进行扩容。这里需要对新节点安装
glusterfs软件包。并且需要使用以下命令 , 将新节点添加到集群中gluster peer probe glusterfs04增加一个
brick[root@glusterfs01 /]# gluster volume add-brick gv1 glusterfs04:/data/dis_volume volume add-brick: success [root@glusterfs01 /]# gluster volume info gv1 Volume Name: gv1 Type: Distribute Volume ID: d6f3b868-e7d1-4c12-8e80-e59409a8566e Status: Started Snapshot Count: 0 Number of Bricks: 4 Transport-type: tcp Bricks: Brick1: glusterfs01:/data/dis_volume Brick2: glusterfs02:/data/dis_volume Brick3: glusterfs03:/data/dis_volume Brick4: glusterfs04:/data/dis_volume Options Reconfigured: storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on再创建30个文件, 并查看文件分布
[root@client ~]# touch /mnt/distributed/{101..130}.txt [root@glusterfs01 /]# ls /data/dis_volume/ 103.txt 108.txt 113.txt 119.txt 120.txt 126.txt 14.txt 16.txt 1.txt 2.txt 6.txt 104.txt 10.txt 118.txt 11.txt 123.txt 12.txt 15.txt 18.txt 20.txt 3.txt [root@glusterfs02 ~]# ls /data/dis_volume 101.txt 105.txt 107.txt 112.txt 114.txt 117.txt 122.txt 13.txt 17.txt 19.txt 4.txt 8.txt [root@glusterfs03 ~]# ls /data/dis_volume/ 102.txt 109.txt 111.txt 116.txt 124.txt 127.txt 129.txt 5.txt 9.txt 106.txt 110.txt 115.txt 121.txt 125.txt 128.txt 130.txt 7.txt [root@glusterfs04 ~]# ls /data/dis_volume/这里可以看到
glusterfs04中完全没有文件
1.2 分布式卷rebalance
在创建文件之后 , 尝试使用rebalance命令重新分布文件
[root@glusterfs01 /]# gluster volume rebalance gv1 start
volume rebalance: gv1: success: Rebalance on gv1 has been started successfully. Use rebalance status command to check status of the rebalance process.
ID: a55101fc-cd47-487e-8846-1382b8ee4c81之后再检查gluserfs04节点中的文件
[root@glusterfs04 ~]# ll /data/dis_volume/
total 0
-rw-r--r-- 2 root root 0 Mar 5 01:34 105.txt
-rw-r--r-- 2 root root 0 Mar 5 01:34 115.txt
-rw-r--r-- 2 root root 0 Mar 5 01:34 116.txt
-rw-r--r-- 2 root root 0 Mar 5 01:34 124.txt
-rw-r--r-- 2 root root 0 Mar 5 01:34 125.txt
-rw-r--r-- 2 root root 0 Mar 5 01:34 127.txt
-rw-r--r-- 2 root root 0 Mar 5 01:34 128.txt
-rw-r--r-- 2 root root 0 Mar 5 01:34 129.txt
-rw-r--r-- 2 root root 0 Mar 5 01:23 19.txt
-rw-r--r-- 2 root root 0 Mar 5 01:23 9.txt可以发现其他节点的文件已经被移动到glusterfs04节点中了。
由于上面扩容步骤中, 创建文件时glusterfs04节点完全没有文件, 之后尝试再创建30个文件, 以保证glusterfs04加入集群正常。
[root@client ~]# touch /mnt/distributed/{201..230}.txt
[root@glusterfs04 ~]# ls /data/dis_volume/
105.txt 116.txt 125.txt 128.txt 19.txt 214.txt 218.txt 220.txt 229.txt
115.txt 124.txt 127.txt 129.txt 202.txt 217.txt 219.txt 224.txt 9.txt可以看到glusterfs04节点中有新建的文件。说明集群扩容成功。
这里有一个需要注意的地方 , 当数据量太大的时候 , 对数据进行rebalance必须要考虑的一个问题就是性能 , 不能因为数据rebalance而影响我们的存储的正常使用。Glusterfs也考虑到了这个问题 , 在进行数据rebalance时 , 根据实际场景不同设计了三种不同的“级别”
lazy; 每次仅可以迁移一个文件normal; 默认设置 , 每次迁移2个文件或者是(CPU逻辑个数-4)/2,哪个大 , 选哪个aggressive; 每次迁移4个文件或者是(CPU逻辑个数-4)/2
通过以下命令进行配置:
gluster volume set VOLUME-NAME cluster.rebal-throttle [lazy|normal|aggressive]可以使用以下命令查看对应卷的参数配置
[root@glusterfs04 ~]# gluster volume get gv1 cluster.rebal-throttle
Option Value
------ -----
cluster.rebal-throttle normal (DEFAULT) 如将volume repvol设置为lazy, 运行以下命令即可
gluster volume set gv1 cluster.rebal-throttle lazy1.3 分布式卷缩容
这里缩容分布式卷主要用到的命令:gluster volume remove-brick <VOLNAME> [replica <COUNT>] <BRICK> ... <start|stop|status|commit|force>
这里主要移除glusterfs04节点, 具体的步骤如下:
迁移
glusterfs04节点中gv1卷存储的文件到其他节点上面[root@glusterfs01 /]# gluster volume remove-brick gv1 glusterfs04:/data/dis_volume start It is recommended that remove-brick be run with cluster.force-migration option disabled to prevent possible data corruption. Doing so will ensure that files that receive writes during migration will not be migrated and will need to be manually copied after the remove-brick commit operation. Please check the value of the option and update accordingly. Do you want to continue with your current cluster.force-migration settings? (y/n) y volume remove-brick start: success ID: be30e1af-8e79-4f19-b408-d8cde4f7f6f8这里提示的内容: 建议在运行
remove-brick命令时禁用cluster.force-migration选项，以防止潜在的数据损坏。禁用此选项可确保在迁移过程中被写入的文件不会被自动迁移，需要在提交移除操作后手动复制这些文件。请检查该选项的当前值并根据需要调整。是否继续使用当前的cluster.force-migration设置？系统默认把这个参数关闭了,不用管
[root@glusterfs01 /]# gluster volume get gv1 cluster.force-migration Option Value ------ ----- cluster.force-migration off查看移除节点的状态
[root@glusterfs01 /]# gluster volume remove-brick gv1 glusterfs04:/data/dis_volume status Node Rebalanced-files size scanned failures skipped status run time in h:m:s --------- ----------- ----------- ----------- ----------- ----------- ------------ -------------- glusterfs04 18 0Bytes 18 0 0 completed 0:00:00这里可以看到
glusterfs04节点中的文件已经被迁移到其他节点上面了[root@glusterfs04 ~]# ll /data/dis_volume/ total 0提交删除
glusterfs04中的/data/dis_volume的brick的操作[root@glusterfs01 /]# gluster volume remove-brick gv1 glusterfs04:/data/dis_volume commit volume remove-brick commit: success Check the removed bricks to ensure all files are migrated. If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick.此时可以看到
gv1卷中的brick已经被删掉了[root@glusterfs01 /]# gluster volume info gv1 Volume Name: gv1 Type: Distribute Volume ID: d6f3b868-e7d1-4c12-8e80-e59409a8566e Status: Started Snapshot Count: 0 Number of Bricks: 3 Transport-type: tcp Bricks: Brick1: glusterfs01:/data/dis_volume Brick2: glusterfs02:/data/dis_volume Brick3: glusterfs03:/data/dis_volume Options Reconfigured: performance.client-io-threads: on storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on
2.复制卷扩容缩容
这里需要创建一个新的复制卷
[root@glusterfs01 /]# gluster volume create vg2 replica 2 glusterfs01:/data/repl_volume glusterfs02:/data/repl_volume
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator-Guide/Split-brain-and-ways-to-deal-with-it/.
Do you still want to continue?
(y/n) y
volume create: vg2: success: please start the volume to access data
[root@glusterfs01 /]# gluster volume start vg2
volume start: vg2: success
[root@glusterfs01 /]# gluster volume info vg2
Volume Name: vg2
Type: Replicate
Volume ID: 428a548b-6c69-41fa-8266-015ff0c16d33
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: glusterfs01:/data/repl_volume
Brick2: glusterfs02:/data/repl_volume
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off然后在客户端挂载并创建20个文件
[root@client ~]# mount -t glusterfs glusterfs01:vg2 /mnt/replica/
[root@client ~]# touch /mnt/replica/{1..20}.txt
#glusterfs01节点查看文件
[root@glusterfs01 /]# ls /data/repl_volume/
10.txt 12.txt 14.txt 16.txt 18.txt 1.txt 2.txt 4.txt 6.txt 8.txt
11.txt 13.txt 15.txt 17.txt 19.txt 20.txt 3.txt 5.txt 7.txt 9.txt
#glusterfs02节点查看文件
[root@glusterfs02 ~]# ls /data/repl_volume/
10.txt 12.txt 14.txt 16.txt 18.txt 1.txt 2.txt 4.txt 6.txt 8.txt
11.txt 13.txt 15.txt 17.txt 19.txt 20.txt 3.txt 5.txt 7.txt 9.txt
2.1 复制卷扩容
这里需要注意, 复制卷和纠删码卷的扩容缩容都是按照distribute的逻辑实现的。扩容之后, 对应的逻辑卷会变成distribute-replica卷。因此复制卷扩容的节点数量是原本复制卷节点的倍数。
由于这里原本集群只有两个节点, 因此扩容的话，需要再添加两个节点。
在集群中添加新节点的
brick[root@glusterfs01 /]# gluster volume add-brick vg2 replica 2 glusterfs03:/data/repl_volume glusterfs04:/data/repl_volume Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator-Guide/Split-brain-and-ways-to-deal-with-it/. Do you still want to continue? (y/n) y volume add-brick: success这里可以看到新的
brick已经添加到集群中, 同时vg2卷的类型变为Distributed-Replicate， 同时现在也变成了2*2的模式。[root@glusterfs01 /]# gluster volume info vg2 Volume Name: vg2 Type: Distributed-Replicate Volume ID: 428a548b-6c69-41fa-8266-015ff0c16d33 Status: Started Snapshot Count: 0 Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: glusterfs01:/data/repl_volume Brick2: glusterfs02:/data/repl_volume Brick3: glusterfs03:/data/repl_volume Brick4: glusterfs04:/data/repl_volume Options Reconfigured: cluster.granular-entry-heal: on storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on performance.client-io-threads: off重新平衡各个节点中卷的文件, 这里会将旧文件平衡到新的
brick中[root@glusterfs01 /]# gluster volume rebalance vg2 start volume rebalance: vg2: success: Rebalance on vg2 has been started successfully. Use rebalance status command to check status of the rebalance process. ID: 3ceb0ca2-ae2c-4ad7-b818-ca773679c5d7平衡之后可以看到新的
brick中已经有了文件[root@glusterfs03 ~]# ls /data/repl_volume/ 10.txt 11.txt 12.txt 14.txt 15.txt 16.txt 18.txt 1.txt 20.txt 2.txt 3.txt 5.txt 6.txt 7.txt [root@glusterfs04 ~]# ls /data/repl_volume 10.txt 11.txt 12.txt 14.txt 15.txt 16.txt 18.txt 1.txt 20.txt 2.txt 3.txt 5.txt 6.txt 7.txt创建新的文件, 查看是否可以分配到新的
brick中。这里可以看到集群已经扩容完成[root@client ~]# touch /mnt/replica/{101..120}.txt [root@glusterfs01 /]# ls /data/repl_volume/ 101.txt 105.txt 107.txt 112.txt 114.txt 115.txt 116.txt 117.txt 13.txt 17.txt 19.txt 4.txt 8.txt 9.txt [root@glusterfs02 ~]# ls /data/repl_volume/ 101.txt 105.txt 107.txt 112.txt 114.txt 115.txt 116.txt 117.txt 13.txt 17.txt 19.txt 4.txt 8.txt 9.txt [root@glusterfs03 ~]# ls /data/repl_volume/ 102.txt 104.txt 108.txt 10.txt 111.txt 118.txt 11.txt 12.txt 15.txt 18.txt 20.txt 3.txt 6.txt 103.txt 106.txt 109.txt 110.txt 113.txt 119.txt 120.txt 14.txt 16.txt 1.txt 2.txt 5.txt 7.txt [root@glusterfs04 ~]# ls /data/repl_volume 102.txt 104.txt 108.txt 10.txt 111.txt 118.txt 11.txt 12.txt 15.txt 18.txt 20.txt 3.txt 6.txt 103.txt 106.txt 109.txt 110.txt 113.txt 119.txt 120.txt 14.txt 16.txt 1.txt 2.txt 5.txt 7.txt
2.2 复制卷缩容
节点的缩容，这里是分布式复制，所以缩容也是成对节点的一起缩容, 操作如下:
迁移删除节点中的文件
[root@glusterfs01 /]# gluster volume remove-brick vg2 replica 2 glusterfs03:/data/repl_volume glusterfs04:/data/repl_volume start Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator-Guide/Split-brain-and-ways-to-deal-with-it/. Do you still want to continue? (y/n) y It is recommended that remove-brick be run with cluster.force-migration option disabled to prevent possible data corruption. Doing so will ensure that files that receive writes during migration will not be migrated and will need to be manually copied after the remove-brick commit operation. Please check the value of the option and update accordingly. Do you want to continue with your current cluster.force-migration settings? (y/n) y volume remove-brick start: success ID: 9236752d-0995-411d-860e-86ae275861af提交删除
brick的操作[root@glusterfs01 /]# gluster volume remove-brick vg2 replica 2 glusterfs03:/data/repl_volume glusterfs04:/data/repl_volume commit Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator-Guide/Split-brain-and-ways-to-deal-with-it/. Do you still want to continue? (y/n) y volume remove-brick commit: success Check the removed bricks to ensure all files are migrated. If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick.查看卷的信息状态。此时可以看到
vg2卷已经恢复成replicate类型了[root@glusterfs01 /]# gluster volume info vg2 Volume Name: vg2 Type: Replicate Volume ID: 428a548b-6c69-41fa-8266-015ff0c16d33 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: glusterfs01:/data/repl_volume Brick2: glusterfs02:/data/repl_volume Options Reconfigured: cluster.granular-entry-heal: on storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on performance.client-io-threads: off
3. 纠删卷扩容缩容
这里首先需要创建一个纠删卷, 这里的纠删码的卷中默认自动会有一个冗余卷。
[root@glusterfs01 /]# gluster volume create gv3 disperse 3 glusterfs01:/data/disperse_volume glusterfs02:/data/disperse_volume glusterfs03:/data/disperse_volume
volume create: gv3: success: please start the volume to access data
[root@glusterfs01 /]# gluster volume start gv3
volume start: gv3: success
[root@glusterfs01 /]# gluster volume info gv3
Volume Name: gv3
Type: Disperse
Volume ID: 285e2cfd-b72f-49ad-8b5a-913a191cc0e6
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: glusterfs01:/data/disperse_volume
Brick2: glusterfs02:/data/disperse_volume
Brick3: glusterfs03:/data/disperse_volume
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on然后在客户端挂载并创建20个文件
[root@client ~]# mount -t glusterfs glusterfs01:gv3 /mnt/disperse/
[root@client ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
devtmpfs devtmpfs 4.0M 0 4.0M 0% /dev
tmpfs tmpfs 1.8G 0 1.8G 0% /dev/shm
tmpfs tmpfs 732M 16M 717M 3% /run
/dev/mapper/rl_host--192--168--71--7-root xfs 195G 3.0G 192G 2% /
/dev/vda1 xfs 5.0G 289M 4.7G 6% /boot
tmpfs tmpfs 162M 0 162M 0% /run/user/0
glusterfs01:gv1 fuse.glusterfs 300G 5.2G 295G 2% /mnt/distributed
glusterfs01:vg2 fuse.glusterfs 100G 1.8G 99G 2% /mnt/replica
glusterfs01:gv3 fuse.glusterfs 200G 3.5G 197G 2% /mnt/disperse
[root@client ~]# touch /mnt/disperse/{1..20}.txt
[root@client ~]#
#glusterfs01节点查看文件
[root@glusterfs01 /]# ls /data/disperse_volume/
10.txt 12.txt 14.txt 16.txt 18.txt 1.txt 2.txt 4.txt 6.txt 8.txt
11.txt 13.txt 15.txt 17.txt 19.txt 20.txt 3.txt 5.txt 7.txt 9.txt
#glusterfs02节点查看文件
[root@glusterfs02 ~]# ls /data/disperse_volume/
10.txt 12.txt 14.txt 16.txt 18.txt 1.txt 2.txt 4.txt 6.txt 8.txt
11.txt 13.txt 15.txt 17.txt 19.txt 20.txt 3.txt 5.txt 7.txt 9.txt
#glusterfs03节点查看文件
[root@glusterfs03 ~]# ls /data/disperse_volume/
10.txt 12.txt 14.txt 16.txt 18.txt 1.txt 2.txt 4.txt 6.txt 8.txt
11.txt 13.txt 15.txt 17.txt 19.txt 20.txt 3.txt 5.txt 7.txt 9.txt3.1 纠删卷扩容
由于原本纠删卷有三个
brick, 因此扩容需要三个节点，将逻辑卷类型由原本的Disperse调整为Distributed-Disperse类型在集群中添加新节点的
brick[root@glusterfs01 /]# gluster volume add-brick gv3 glusterfs04:/data/disperse_volume glusterfs05:/data/disperse_volume glusterfs06:/data/disperse_volume volume add-brick: success这里同样可以看到新的
brick已经添加到集群中了, 这里的brick数量也变成了2*(2+1),这里面括号外面的2表示分布式卷的数量为2，纠删卷为3个，其中有一个为冗余卷。[root@glusterfs01 /]# gluster volume info gv3 Volume Name: gv3 Type: Distributed-Disperse Volume ID: 285e2cfd-b72f-49ad-8b5a-913a191cc0e6 Status: Started Snapshot Count: 0 Number of Bricks: 2 x (2 + 1) = 6 Transport-type: tcp Bricks: Brick1: glusterfs01:/data/disperse_volume Brick2: glusterfs02:/data/disperse_volume Brick3: glusterfs03:/data/disperse_volume Brick4: glusterfs04:/data/disperse_volume Brick5: glusterfs05:/data/disperse_volume Brick6: glusterfs06:/data/disperse_volume Options Reconfigured: storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on将原本卷中的文件，调整到新的
brick中[root@glusterfs01 /]# gluster volume rebalance gv3 start volume rebalance: gv3: success: Rebalance on gv3 has been started successfully. Use rebalance status command to check status of the rebalance process. ID: 0772c006-e324-4663-9a16-b7cd6aeaab51平衡后可以看到新的
brick中已经有了文件。[root@glusterfs04 ~]# ls /data/disperse_volume 10.txt 11.txt 12.txt 14.txt 15.txt 16.txt 18.txt 1.txt 20.txt 2.txt 3.txt 5.txt 6.txt 7.txt [root@glusterfs05 ~]# ls /data/disperse_volume 10.txt 11.txt 12.txt 14.txt 15.txt 16.txt 18.txt 1.txt 20.txt 2.txt 3.txt 5.txt 6.txt 7.txt [root@glusterfs06 ~]# ls /data/disperse_volume 10.txt 11.txt 12.txt 14.txt 15.txt 16.txt 18.txt 1.txt 20.txt 2.txt 3.txt 5.txt 6.txt 7.txt
3.2 纠删卷缩容
之后尝试缩容。这里缩容按照纠删卷中原本的brick数量的倍数来调整缩容的brick数量
迁移删除节点中的文件
[root@glusterfs01 /]# gluster volume remove-brick gv3 glusterfs04:/data/disperse_volume glusterfs05:/data/disperse_volume glusterfs06:/data/disperse_volume start It is recommended that remove-brick be run with cluster.force-migration option disabled to prevent possible data corruption. Doing so will ensure that files that receive writes during migration will not be migrated and will need to be manually copied after the remove-brick commit operation. Please check the value of the option and update accordingly. Do you want to continue with your current cluster.force-migration settings? (y/n) y volume remove-brick start: success ID: bbb0aeec-80a5-4028-ab10-84b306b00cbc文件迁移完成之后, 向集群提交删除
brick的操作[root@glusterfs01 /]# gluster volume remove-brick gv3 glusterfs04:/data/disperse_volume glusterfs05:/data/disperse_volume glusterfs06:/data/disperse_volume commit volume remove-brick commit: success Check the removed bricks to ensure all files are migrated. If files with data are found on the brick path, copy them via a gluster mount point before re-purposing the removed brick.此时查看集群卷的状态, 这里gv3的格式已经变成
disperse类型了。[root@glusterfs01 /]# gluster volume info gv3 Volume Name: gv3 Type: Disperse Volume ID: 285e2cfd-b72f-49ad-8b5a-913a191cc0e6 Status: Started Snapshot Count: 0 Number of Bricks: 1 x (2 + 1) = 3 Transport-type: tcp Bricks: Brick1: glusterfs01:/data/disperse_volume Brick2: glusterfs02:/data/disperse_volume Brick3: glusterfs03:/data/disperse_volume Options Reconfigured: performance.client-io-threads: on storage.fips-mode-rchecksum: on transport.address-family: inet nfs.disable: on
参考链接