1. 限制brick的预留空间
由于glusterfs是通过客户端计算来决定去操作远程哪一个brick的数据，这个哈希计算可能会导致多个相同规格的磁盘的使用情况不一致，比如有10块盘，其中有1块盘使用量达到90%。剩余其他的磁盘使用才80%，如果这样导致使用达到99%，最后直到glusterfsd的进程crash(glusterfsd定期会写一个日期的字符串来验证glusterfsd进程对应磁盘是否健康，一旦写入发现磁盘剩余空间无法写入的时候，glusterfsd就自杀了)
因此在glusterfsd运行时候可以设定storage.reserve和storage.reserve-size，前者是设定百分比，后者是设定大小。但是2个参数设定后，glusterfsd会运行一个thread每5s来操作检查一下，free(剩余空间)和storage.reserve或者storage.reserve-size设定大小比较，如果free<=storage.reserve或者free<=storage.reserve-size都会写入失败，报出“No space left on device”错误
glusterfs 设定磁盘空间保留
// 按照百分比对brick进行设定
gluster volume set test storage.reserve percentage
Option: storage.reserve
Description: Percentage of disk space to be reserved. Set to 0 to disable
Option: storage.reserve-size
Description: If set, priority will be given to storage.reserve-size over storage.reserve2. cluster.read-hash-mode工作原模式分析
read-hash-mode的意义在于能否提供IO的负载均衡，尤其是在多副本的情况，能减少IO的争用，提供更好的IOPS能力，read-hash-mode取值范围在0~5，每个值的设定会影响客户端挂载的IOPS能力。接下来分别read-hash-mode参数说明
0代表用户的IO请求存储端的生命中内选择第一个提供文件服务;1代表根据用户的文件的唯一标识gfid(内部是一个uuid的字符串)，粗暴的进行计算哈希然后针对副本卷的brick数进行取模，选择一个后端brick提供服务;2是代表挂载节点的fuse进程计算哈希然后针对副本卷的brick数取模，选择一个后端brick提供服务;3选择选择后端brick的读事务最小的提供服务;4是针对brick服务的网络延迟评估最小的延迟提供服务。5是结合网络延迟和读事务综合选择brick提供服务(内测)；
read-hash-mode在数据分散很均匀的情况(一致性哈希保证)，设定为1的效果比较好，能达到IO请求比较均衡效果；0和2据测试会选择比较固定的brick。3和4属于glusterfs功能和试验阶段的参数值(内部统计不是很精确)。在我们大规模使用情况下尽量选择1的模式
read-hash-mode这个算法也存在一定的缺陷，就是1和2，如果数据集中到某一个brick,那么提供服务的brick也会固定，一份数据的剩余副本不能提供服务，请求集中到某一个brick很容易把磁盘的IO打满。已经给官方提供issue,建议使用一致性哈希算法，能保证每个brick的处理请求数量均衡
read-hash-mode参数说明
[root@CentOS1 ~]$ gluster volume set help |grep cluster.read-hash-mode -A7
Option: cluster.read-hash-mode
Default Value: 1
Description: inode-read fops happen only on one of the bricks in replicate. AFR will prefer the one computed using the method specified using this option.
0 = first readable child of AFR, starting from 1st child.
1 = hash by GFID of file (all clients use same subvolume).
2 = hash by GFID of file and client PID.
3 = brick having the least outstanding read requests.
4 = brick having the least network ping latency.3. glusterfs替换掉brick
分布式卷替换方法
#找一块和需要替换的磁盘空间大小一致的新磁盘
gluster volume info test_vol
Volume Name: test_vol
Type: Distribute
Volume ID: 638ed2b9-928e-4d7c-982d-be7123a823f1
Status: Started
Snapshot Count: 0
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: 192.168.78.12:/test_vol/data1/brick
Brick2: 192.168.78.12:/test_vol/data2/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
#新增一个brick
gluster volume add-brick test_vol 192.168.78.12:/test_vol/data3/brick
#删除就的brick
gluster volume remove-brick test_vol 192.168.78.12:/test_vol/data1/brick start
#查看状态
gluster volume remove-brick test_vol 192.168.78.12:/test_vol/data1/brick status
#提交
gluster volume remove-brick test_vol 192.168.78.12:/test_vol/data1/brick commit多副本卷故障brick替换
gluster volume info test_rep_vol
Volume Name: test_rep_vol
Type: Distributed-Replicate
Volume ID: 50b9747d-591e-4c85-9f22-c94aa851adb8
Status: Started
Snapshot Count: 0
Number of Bricks: 22 x 3 = 66
Transport-type: tcp
Bricks:
Brick1: 192.168.66.21:/test_rep_vol/data1/brick
Brick2: 192.168.66.22:/test_rep_vol/data1/brick
Brick3: 192.168.66.23:/test_rep_vol/data1/brick
Brick4: 192.168.66.21:/test_rep_vol/data2/brick
Brick5: 192.168.66.22:/test_rep_vol/data2/brick
Brick6: 192.168.66.23:/test_rep_vol/data2/brick
Brick7: 192.168.66.21:/test_rep_vol/data3/brick
Brick8: 192.168.66.22:/test_rep_vol/data3/brick
Brick9: 192.168.66.23:/test_rep_vol/data3/brick
Brick10: 192.168.66.21:/test_rep_vol/data4/brick
Brick11: 192.168.66.22:/test_rep_vol/data4/brick
Brick12: 192.168.66.23:/test_rep_vol/data4/brick
Brick13: 192.168.66.21:/test_rep_vol/data5/brick
Brick14: 192.168.66.22:/test_rep_vol/data5/brick
Brick15: 192.168.66.23:/test_rep_vol/data5/brick
Brick16: 192.168.66.21:/test_rep_vol/data6/brick
Brick17: 192.168.66.22:/test_rep_vol/data6/brick
Brick18: 192.168.66.23:/test_rep_vol/data6/brick
Brick19: 192.168.66.21:/test_rep_vol/data7/brick
Brick20: 192.168.66.22:/test_rep_vol/data7/brick
Brick21: 192.168.66.23:/test_rep_vol/data7/brick
Brick22: 192.168.66.21:/test_rep_vol/data8/brick
Brick23: 192.168.66.22:/test_rep_vol/data8/brick
Brick24: 192.168.66.23:/test_rep_vol/data8/brick
Brick25: 192.168.66.21:/test_rep_vol/data9/brick
Brick26: 192.168.66.22:/test_rep_vol/data9/brick
Brick27: 192.168.66.23:/test_rep_vol/data9/brick
Brick28: 192.168.66.21:/test_rep_vol/data11/brick
Brick29: 192.168.66.22:/test_rep_vol/data11/brick
Brick30: 192.168.66.23:/test_rep_vol/data11/brick
Brick31: 192.168.69.44:/test_rep_vol/data1/brick
Brick32: 192.168.72.1:/test_rep_vol/data1/brick
Brick33: 192.168.72.4:/test_rep_vol/data1/brick
Brick34: 192.168.69.44:/test_rep_vol/data2/brick
Brick35: 192.168.72.1:/test_rep_vol/data2/brick
Brick36: 192.168.72.4:/test_rep_vol/data2/brick
Brick37: 192.168.69.44:/test_rep_vol/data3/brick
Brick38: 192.168.72.1:/test_rep_vol/data3/brick
Brick39: 192.168.72.4:/test_rep_vol/data3/brick
Brick40: 192.168.69.44:/test_rep_vol/data4/brick
Brick41: 192.168.72.1:/test_rep_vol/data4/brick
Brick42: 192.168.72.4:/test_rep_vol/data4/brick
Brick43: 192.168.69.44:/test_rep_vol/data5/brick
Brick44: 192.168.72.1:/test_rep_vol/data5/brick
Brick45: 192.168.72.4:/test_rep_vol/data5/brick
Brick46: 192.168.69.44:/test_rep_vol/data6/brick
Brick47: 192.168.72.1:/test_rep_vol/data6/brick
Brick48: 192.168.72.4:/test_rep_vol/data6/brick
Brick49: 192.168.69.44:/test_rep_vol/data7/brick
Brick50: 192.168.72.1:/test_rep_vol/data7/brick
Brick51: 192.168.72.4:/test_rep_vol/data7/brick
Brick52: 192.168.69.44:/test_rep_vol/data8/brick
Brick53: 192.168.72.1:/test_rep_vol/data8/brick
Brick54: 192.168.72.4:/test_rep_vol/data8/brick
Brick55: 192.168.69.44:/test_rep_vol/data9/brick
Brick56: 192.168.72.1:/test_rep_vol/data9/brick
Brick57: 192.168.72.4:/test_rep_vol/data9/brick
Brick58: 192.168.69.44:/test_rep_vol/data11/brick
Brick59: 192.168.72.1:/test_rep_vol/data11/brick
Brick60: 192.168.72.4:/test_rep_vol/data11/brick
Brick61: 192.168.69.44:/test_rep_vol/data10/brick
Brick62: 192.168.72.1:/test_rep_vol/data10/brick
Brick63: 192.168.72.4:/test_rep_vol/data10/brick
Brick64: 192.168.66.21:/test_rep_vol/data10/brick
Brick65: 192.168.66.22:/test_rep_vol/data10/brick
Brick66: 192.168.66.23:/test_rep_vol/data10/brick
Options Reconfigured:
transport.address-family: inet
storage.fips-mode-rchecksum: on
nfs.disable: on
performance.client-io-threads: off
#整个卷的192.168.69.44机器故障，在此机器上的磁盘都要替换掉
#找一台和192.168.69.44规格一抹一样的机器，安装好glusterfs,准备好磁盘
#把192.168.51.147添加到集群
gluster peer probe 192.168.51.147
#替换故障brick
gluster volume replace-brick test_rep_vol 192.168.69.44:/test_rep_vol/data1/brick 192.168.51.147:/test_rep_vol/data1/brick commit force
#继续替换
gluster volume replace-brick test_rep_vol 192.168.69.44:/test_rep_vol/data2/brick 192.168.51.147:/test_rep_vol/data2/brick commit force
#通过在192.168.51.147 上执行df -h 可以看到磁盘数据正在同步
#查看执行完成,通过heal 没有不一致了就表示同步完成
gluster volume heal test_rep_vol info
#查看替换结果
gluster volume info test_rep_vol参考链接