cbear
large picture of the cover

4-GlusterFS分布式存储客户端使用

可以使用Gluster Native Client方法在GNU / Linux客户端中实现高并发性 , 性能和透明故障转移。可以使用NFS v3访问gluster卷。已经对GNU / Linux客户端和其他操作系统中的NFS实现进行了广泛的测试 , 例如FreeBSD , Mac OS X , 以及Windows 7(Professional和Up) 和Windows Server 2003 。其他NFS客户端实现可以与gluster一起使用NFS服务器。使用Microsoft Windows以及SAMBA客户端时 , 可以使用CIFS访问卷。对于此访问方法 , Samba包需要存在于客户端。

总结 : GlusterFS支持三种客户端类型。Gluster Native Client、NFS和CIFS。Gluster Native Client是在用户空间中运行的基于FUSE的客户端 , 官方推荐使用Native Client , 可以使用GlusterFS的全部功能。

1. 使用Gluster Native Client挂载

Gluster Native Client是基于FUSE的，所以需要保证客户端安装了FUSE。这个是官方推荐的客户端，支持高并发和高效的写性能。

在开始安装Gluster Native Client之前，您需要验证客户端上是否已加载FUSE模块，并且可以访问所需的模块，如下所示 :

[root@client ~]# modprobe fuse
[root@client ~]# dmesg | grep -i fuse
[   10.919547] systemd[1]: Starting Load Kernel Module fuse...
[   11.167353] fuse: init (API version 7.36)

安装Gluseter Native Client

[root@client ~]# dnf install glusterfs glusterfs-fuse
[root@client ~]# mount.glusterfs glusterfs01:disperse-volume /mnt/disperse
[root@client ~]# df -hT
Filesystem                                Type            Size  Used Avail Use% Mounted on
devtmpfs                                  devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs                                     tmpfs           1.8G     0  1.8G   0% /dev/shm
tmpfs                                     tmpfs           732M   16M  717M   3% /run
/dev/mapper/rl_host--192--168--71--7-root xfs             195G  3.0G  192G   2% /
/dev/vda1                                 xfs             5.0G  289M  4.7G   6% /boot
tmpfs                                     tmpfs           162M     0  162M   0% /run/user/0
glusterfs01:disperse-volume               fuse.glusterfs  400G  7.0G  393G   2% /mnt/disperse
​
[root@client ~]# findmnt /mnt/disperse 
TARGET        SOURCE                      FSTYPE         OPTIONS
/mnt/disperse glusterfs01:disperse-volume fuse.glusterfs rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072
​

手动挂载选项

这里可以通过命令man mount.glusterfs查看所有相关挂载选项。

  • backup-volfile-servers=SERVERLIST : 提供备份卷文件服务器列表，格式如下 [默认：无]。如果在服务器列表（SERVERLIST）中使用 IPv6 地址，则需要用 “[” 和 “]” 将 IPv6 地址括起来。

    $ mount -t glusterfs -obackup-volfile-servers=<server2>:<server3>:...:<serverN>   <server1>:/<volname> <mount_point>       
​
$mount -t glusterfs -obackup-volfile-servers=<[ipv6_addr2]>:<[ipv6_addr3]>:...:<[ipv6_addrN]> <ipv6_addr1>:/<volname> <mount_point>

  • log-file : 指定日志文件位置。(默认为/var/log/glusterfs/glusterfs.log)

  • log-level : 日志级别。参数:TRACE, DEBUG, WARNING, ERROR, CRITICAL INFO and NONE [默认: INFO]

  • acl : 挂载的文件系统支持ACL配置

  • ro : 以只读方式挂载文件系统

  • no_root_squash : 挂载之后，root用户对该挂载系统权限即为root权限

  • root_squash : 挂载之后,root用户对该挂载文件系统的权限为匿名用户的权限

  • use-readdirp=BOOL ： 设置为ON，则强制在fuse内核模块中使用readdirp模式

  • direct-io-mode=[BOOL|auto] : 指定 FUSE(文件系统用户空间)直接输入输出(I/O)策略 [默认：auto]

  • subdir-mount: 指定卷中的子目录挂载

自动挂载卷

除了使用mount挂载, 还可以使用/etc/fstab自动挂载

语法格式 :HOSTNAME-OR-IPADDRESS:/VOLNAME MOUNTDIR glusterfs defaults,_netdev 0 0

2. NFS挂载

在查看volume详细信息时，我们会发现有一个选项“nfs.disable: on”,从字面上就可以理解默认是禁用了nfs的

[root@glusterfs01 ~]# gluster volume info disperse-volume 
 
Volume Name: disperse-volume
Type: Disperse
Volume ID: 71132f2e-3a06-498c-8e9f-c96d767ea043
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 1) = 5
Transport-type: tcp
Bricks:
Brick1: glusterfs01:/data/disperse_volume
Brick2: glusterfs02:/data/disperse_volume
Brick3: glusterfs03:/data/disperse_volume
Brick4: glusterfs04:/data/disperse_volume
Brick5: glusterfs05:/data/disperse_volume
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
​

因此需要在服务器端开启nfs的功能支持

gluster volume set <VOLNAME> nfs.disable off
​
#示例
[root@glusterfs01 ~]# gluster volume set disperse-volume nfs.disable off
Gluster NFS is being deprecated in favor of NFS-Ganesha Enter "yes" to continue using Gluster NFS (y/n) y
volume set: failed: option : nfs.disable does not exist
Did you mean dht.force-readdirp?
[root@glusterfs01 ~]# gluster volume get disperse-volume all | grep dht.force-readdirp
dht.force-readdirp                       on (DEFAULT)                          

GlusterFS 内置的 NFS 服务(称为 "Gluster NFS")已被标记为弃用，未来版本可能完全移除。官方推荐迁移到 NFS-Ganesha(一个独立且功能更强大的 NFS 服务器实现)。理论上这里启动nfs功能之后, 重启volume。然后server上会启动一个nfs的进程。之后客户端便可以直接使用nfs协议挂载

这里复制一份网上做nfs挂载的流程。因为已经被弃用，所以这里只是作为参考即可

#在查看volume详细信息时，我们会发现有一个选项“nfs.disable: on”,从字面上就可以理解默认是禁用了nfs的
[root@node01 /]# gluster volume info repvol
  
Volume Name: repvol
Type: Replicate
Volume ID: 7015b7fe-8039-4d5d-b698-e8321df5289c
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: node01.lab.example.com:/bricks/thinvol1/brick
Brick2: node02.lab.example.com:/bricks/thinvol1/brick
Options Reconfigured:
transport.address-family: inet
performance.readdir-ahead: on
nfs.disable: on
​
​
#手动放开nfs,并重启volume
[root@node01 ~]# gluster volume set repvol nfs.disable off
volume set: success
[root@node01 ~]# gluster volume stop repvol
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: repvol: success
[root@node01 ~]# gluster volume start repvol
volume start: repvol: success
[root@node01 ~]# gluster volume info repvol | grep nfs.disable
nfs.disable: off
[root@node01 ~]#
​
#同时server上会启动这样一个进程做一个nfs共享以便提供nfs挂载
[root@node01 ~]# ps -aux | grep nfs | grep -v grep
root      20064  0.1  6.5 581640 65028 ?        Ssl  04:23   0:00 /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S /var/run/gluster/d8b498939d669fdc978b5b3d96d5aa9d.socket
​
#放开防火墙限制
[root@node01 ~]# firewall-cmd --add-service=nfs --add-service=rpc-bind
success
​
#在node02上使用nfs方式挂载volume repvol
[root@node02 /]# mount -t nfs node01.lab.example.com:repvol /mnt
[root@node02 /]# df -h | grep mnt
node01.lab.example.com:repvol   2.0G   33M  2.0G   2% /mnt

2.1 yum 安装配置nfs-ganesha

这里直接安装ganesha, 由于这里使用的是Rocky9 系统。需要开启devel的安装源, 之后下载nfs-ganesha5的安装源

[root@glusterfs01 nfs-ganesha-7-dev.7]# dnf -y install centos-release-nfs-ganesha4 centos-release-nfs-ganesha5

之后可以看到，多了两个nfs-ganesha 的 .repo包

[root@glusterfs01 yum.repos.d]# ls
CentOS-Gluster-11.repo     CentOS-Storage-common.repo  epel-testing.repo      rocky-devel.repo      rocky-extras.repo.bak
CentOS-NFS-Ganesha-4.repo  epel-cisco-openh264.repo    rocky-addons.repo      rocky-devel.repo.bak  rocky.repo
CentOS-NFS-Ganesha-5.repo  epel.repo                   rocky-addons.repo.bak  rocky-extras.repo
​

之后直接安装需要的rpm包即可。需要说明的一点, 这里安装nfs-ganesha软件可以单独找一个节点配置，以此提供nfs功能

[root@glusterfs01 ~]# dnf install -y nfs-ganesha nfs-ganesha-gluster

安装完成之后, 修改配置文件/etc/ganesha/ganesha.conf 配置以下内容

NFS_CORE_PARAM {
        mount_path_pseudo = true;
        Protocols = 3,4;
}

EXPORT_DEFAULTS {
        Access_Type = RW;
}

EXPORT {
    Export_Id = 101 ;                 # 唯一导出ID
    Path = "/gluster_vol1";            # 客户端挂载的伪路径（无需实际存在）

    FSAL {
        name = GLUSTER;                # 指定使用 GlusterFS 后端
        hostname = "glusterfs01";      # GlusterFS 节点IP或主机名
        volume = "disperse-volume";    # GlusterFS 卷名称
    }
    GLUSTER { PNFS_MDS = true; }  # 如果支持pNFS的话，需要配置GLUSTER的选项

    Access_type = RW;            # 读写权限
    Squash = No_root_squash;     # 允许root访问
    Disable_ACL = TRUE;  
    Pseudo = "/gluster_vol1";     # 客户端看到的路径（与Path一致）
    Protocols = "3","4" ;       # 启用 NFSv4
    Transports = "UDP","TCP" ;  #使用协议
    SecType = "sys";    
}

修改配置文件之后, 启动nfs-ganesha服务，这里需要注意，这个服务对应的日志:/var/log/ganesha/ganesha.log。

启动之后，使用命令showmount -e localhost 检查是否配置完成

[root@glusterfs01 /]# showmount -e localhost 
Export list for localhost:
/gluster_vol1 (everyone)

后面直接在客户端挂载即可

[root@client ~]# mount -t nfs glusterfs01:/gluster_vol1 /mnt/disperse/
[root@client ~]# df -hT
Filesystem                                Type      Size  Used Avail Use% Mounted on
devtmpfs                                  devtmpfs  4.0M     0  4.0M   0% /dev
tmpfs                                     tmpfs     1.8G     0  1.8G   0% /dev/shm
tmpfs                                     tmpfs     732M   16M  717M   3% /run
/dev/mapper/rl_host--192--168--71--7-root xfs       195G  3.0G  192G   2% /
/dev/vda1                                 xfs       5.0G  289M  4.7G   6% /boot
tmpfs                                     tmpfs     162M     0  162M   0% /run/user/0
glusterfs01:/gluster_vol1                 nfs4      400G  7.0G  393G   2% /mnt/disperse

2.2 编译安装nfs-ganesha

这里直接复制github的文章，作为一个参考查看

  1. 下载安装包

    $ git clone  https://hub.fastgit.org/nfs-ganesha/nfs-ganesha.git
$ cd nfs-ganesha && git submodule update --init
$ git branch -r
  origin/1.5.x
  origin/HEAD -> origin/next
  origin/V2.1-stable
  origin/V2.2-stable
  origin/V2.3-stable
  origin/V2.4-stable
  origin/V2.5-stable
  origin/V2.6-stable
  origin/V2.7-stable
  origin/V2.8-stable
  origin/V3-stable
  origin/fixv3
  origin/gh-pages
  origin/next
  origin/test
$ git checkout V3-stable

  2. 安装依赖包

    // 在centos上安装
 $ yum install  git cmake autoconf libtool bison flex doxygen openssl-devel  krb5-libs krb5-devel libuuid-devel nfs-utils -y
 
 or 
 
 // 在debian/ubuntu上安装
 $ apt install uuid-dev  nfs-kernel-server git cmake autoconf libtool bison flex doxygen openssl-dev  libkrb5-3 krb5-dev
 
 // 需要安装glusterfs中gfapi
 $ yum install centos-release-gluster7.noarch -y  
 $ yum install glusterfs-fuse.x86_64 glusterfs-api.x86_64 glusterfs-libs.x86_64  glusterfs-api-devel.x86_64   glusterfs-api.x86_64 -y

 //或者源码安装glusterfs,后续需要指定glusterfs的安装路径，nfs-ganesha需要检查安装的系统包

  3. nfs-ganesha源码编译安装

    • 安装nfs-ganesha依赖

      yum install -y centos-release-nfs-ganesha30
yum install -y avahi-libs checkpolicy cups-libs gssproxy keyutils libicu   libverto-libevent libwbclient nfs-utils policycoreutils-python-utils psmisc   python3-audit python3-libsemanage python3-policycoreutils  python3-pyyaml python3-setools quota quota-nls  samba-client-libs  samba-common samba-common-libs

    • glusterfs安装检查

      // nfs-ganesha指定了GLUSTER_PREFIX安装目录，会检查GLUSTER_PREFIX/下面的lib、local/lib、local/lib64、include文件夹下面的关于glusterfs安装信息
// 如果是源码安装glusteerfs，默认是安装在/usr/local下面
IF(USE_FSAL_GLUSTER)
  IF(GLUSTER_PREFIX)
    set(GLUSTER_PREFIX ${GLUSTER_PREFIX} CACHE PATH "Path to Gluster installation")
    LIST(APPEND CMAKE_PREFIX_PATH "${GLUSTER_PREFIX}")
    LIST(APPEND CMAKE_LIBRARY_PATH "${GLUSTER_PREFIX}/lib")
    LIST(APPEND CMAKE_LIBRARY_PATH "${GLUSTER_PREFIX}/local/lib")
    LIST(APPEND CMAKE_LIBRARY_PATH "${GLUSTER_PREFIX}/local/lib64")
    LIST(APPEND CMAKE_REQUIRED_INCLUDES "${GLUSTER_PREFIX}/include")
  ELSE()
    set(GLUSTER_PREFIX "/usr" CACHE PATH "Path to Gluster installation")
  ENDIF()
ENDIF()

    • 编译

      $ git clone  https://github.com/nfs-ganesha/nfs-ganesha.git
$ cd nfs-ganesha && git submodule update --init --recursive

$ cd nfs-ganesha/src && vi CMakeLists.txt 
 // 修改如下内容，添加 类似于"set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O0 -ggdb3")"

 if (LINUX)
    set(PLATFORM "LINUX")
    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O0 -ggdb3 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64")
    set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -O0 -ggdb3 -fno-strict-aliasing")
    set(OS_INCLUDE_DIR "${PROJECT_SOURCE_DIR}/include/os/linux")
    find_library(LIBDL dl)  # module loader
endif(LINUX)


$ mkdir nfs-ganhesha/src/build && cd nfs-ganesha/src/build 
// GLUSTER_PREFIX=/usr/local 指定glusterfs安装目录
# cmake -DUSE_FSAL_GLUSTER=ON -DGLUSTER_PREFIX=/usr/local ../ 
$ make -j32 && make install
$ whereis ganesha.nfsd
ganesha: /usr/bin/ganesha.nfsd /usr/lib64/ganesha /etc/ganesha /usr/local/etc/ganesha /usr/libexec/ganesha


#这里同时打开了Glusterfs的支持，关闭ceph和xfs的支持
 cmake . -DUSE_FSAL_GLUSTER=ON -DUSE_FSAL_CEPH=OFF -DUSE_FSAL_XFS=OFF

  4. nfs-ganesha配置

    EXPORT
{
	# Export Id (mandatory, each EXPORT must have a unique Export_Id)
	Export_Id = 77;

	# Exported path (mandatory)
	Path = "/dht-vol";

	# Pseudo Path (required for NFS v4)
	Pseudo = "/dht-vol";

	# Required for access (default is None)
	# Could use CLIENT blocks instead
	Access_Type = RW;

	# Allow root access
	Squash = No_Root_Squash;

	# Security flavor supported
	SecType = "sys";

	# Exporting FSAL
	FSAL {
		Name = "GLUSTER";
		Hostname = "172.16.84.37";
		Volume = "dht-vol";
		enable_upcall = true;
		Transport = tcp; # tcp or rdma
	}
}

  5. 添加service

    vi /usr/lib/systemd/system/nfs-ganesha.service

[Unit]
Description=NFS-Ganesha file server
Documentation=http://github.com/nfs-ganesha/nfs-ganesha/wiki
After=rpcbind.service nfs-ganesha-lock.service
Wants=rpcbind.service nfs-ganesha-lock.service
Conflicts=nfs.target

After=nfs-ganesha-config.service
Wants=nfs-ganesha-config.service

[Service]
Type=forking
# Let systemd create /run/ganesha, /var/log/ganesha and /var/lib/nfs/ganesha
# directories
RuntimeDirectory=ganesha
LogsDirectory=ganesha
StateDirectory=nfs/ganesha
EnvironmentFile=-/run/sysconfig/ganesha
ExecStart=/bin/bash -c "${NUMACTL} ${NUMAOPTS} /usr/bin/ganesha.nfsd ${OPTIONS} ${EPOCH}"
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/bin/dbus-send --system   --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.shutdown

[Install]
WantedBy=multi-user.target
Also=nfs-ganesha-lock.service


#其他两个service文件
[root@glusterfs01 /]# systemctl cat nfs-ganesha-lock
# /usr/lib/systemd/system/nfs-ganesha-lock.service
# This file is part of nfs-ganesha.
#
# rpc.statd is started by the nfs-lock.service, but that also loads the 'lockd'
# kernel module in 'ExecStartPre'. The 'lockd' kernel module will register
# itself as 'nlockmgr' which conflicts with the nfs-ganesha locking
# implementation.
#
# This unit includes all the nfs-lock.service settings and details, but
# overrides the 'ExecStartPre' and 'ExecStartPost' options.
#
# When this unit is started, the original nfs-lock.service is stopped (due to
# the 'Conflicts' directive). With stopping the nfs-lock.service, 'lockd' gets
# instructed to unregister 'nlockmgr' from the portmapper.
#
# The nfs-ganesha.service depends on this unit.
#

[Unit]
Before=nfs-ganesha.service
Conflicts=nfs-lock.service

[Service]
ExecStartPre=
ExecStopPost=

[Install]
WantedBy=nfs-ganesha.service

# /etc/systemd/system/nfs-ganesha-lock.service.d/rpc-statd.conf
[Unit]
Description=NFS status monitor for NFSv2/3 locking.
DefaultDependencies=no
Conflicts=umount.target
Requires=nss-lookup.target rpcbind.socket
Wants=network-online.target
After=network-online.target nss-lookup.target rpcbind.socket

PartOf=nfs-utils.service

[Service]
Environment=RPC_STATD_NO_NOTIFY=1
Type=forking
PIDFile=/run/rpc.statd.pid
ExecStart=/usr/sbin/rpc.statd

  6. nfs-ganehsa服务启动

    $ systemctl daemon-reload
$ systemctl start  nfs-ganesha 
$ systemctl stop nfs-ganesha 


// 或者二进制启动
/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT

  7. 客户端挂载

    yum install –y nfs-utils

//支持挂载子目录，这里10.168.18.141是nfs-ganehsa节点的IP(这个独立于glusgerfs后端服务)
mount -t nfs4  10.168.18.141:/dht-vol  /mnt/nfs

// 如果配置了pfs,则按照如下方式进行pnfs挂载
mount -t nfs  -o   port=2049,vers=4.1,proto=tcp     10.168.18.141:/dht-vol /mnt/speech_v5_rep_vol

3. samba挂载

  1. 单独找一个节点安装samba。这里需要注意安装samba节点的同时需要安装samba-vfs-glusterfs模块。由于系统安装的rockylinux9系统，默认的安装包里面没有这个模块的软件包。因此这里直接使用centos-stream9的yum源。

    配置的yum源如下

    [root@glusterfs02 ~]# cat /etc/yum.repos.d/Samba.repo 
[samba]
name=samba repo
baseurl=https://buildlogs.centos.org/centos/9-stream/storage/x86_64/samba-420/
enable=1
gpgcheck=0

    之后运行命令安装即可

    [root@glusterfs02 ~]# dnf install samba samba-vfs-glusterfs

  2. samba配置文件。这里所有的选项可以参考samba的模块官方链接:vfs_glusterfs

    [gluster-repvol]
comment = For samba share of volume repvol
vfs objects = glusterfs
glusterfs:volume = disperse-volume
glusterfs:logfile = /var/log/samba/glusterfs-repvol.%M.log
glusterfs:loglevel = 7
glusterfs:volfile_server = glusterfs01
path = /
read only = no
guest ok = yes

  3. 启动samba服务，创建samba客户端挂载用户

    [root@glusterfs02 ~]# systemctl start smb
[root@glusterfs02 ~]# adduser -s /sbin/nologin smbuser
[root@glusterfs02 ~]# smbpasswd -a smbuser
New SMB password:
Retype new SMB password:
Added user smbuser.

  4. 找一个客户端安装samba客户端工具

    root@client ~]# yum install -y samba-client cifs-utils

  5. 客户端测试

    [root@client ~]# smbclient -L glusterfs02 -U%

	Sharename       Type      Comment
	---------       ----      -------
	print$          Disk      Printer Drivers
	IPC$            IPC       IPC Service (Samba 4.20.2)
SMB1 disabled -- no workgroup available
[root@client ~]# smbclient -L glusterfs02 -U%

	Sharename       Type      Comment
	---------       ----      -------
	print$          Disk      Printer Drivers
	gluster-repvol  Disk      For samba share of volume repvol
	IPC$            IPC       IPC Service (Samba 4.20.2)
SMB1 disabled -- no workgroup available

  6. 之后直接通过客户端挂载即可

    [root@client ~]# mount -t cifs -o username=smbuser,password=Bocft01! //glusterfs02/gluster-repvol /opt
[root@client ~]# df -hT
Filesystem                                Type      Size  Used Avail Use% Mounted on
devtmpfs                                  devtmpfs  4.0M     0  4.0M   0% /dev
tmpfs                                     tmpfs     1.8G     0  1.8G   0% /dev/shm
tmpfs                                     tmpfs     732M   16M  717M   3% /run
/dev/mapper/rl_host--192--168--71--7-root xfs       195G  3.0G  192G   2% /
/dev/vda1                                 xfs       5.0G  289M  4.7G   6% /boot
tmpfs                                     tmpfs     162M     0  162M   0% /run/user/0
glusterfs01:/gluster_vol1                 nfs4      400G  7.0G  393G   2% /mnt/disperse
//glusterfs02/gluster-repvol              cifs      400G  7.0G  393G   2% /opt

参考链接

GlusterFS学习之路（三）客户端挂载和管理GlusterFS卷 - 烟雨浮华 - 博客园

Glusterfs volume 的三种挂载方式 - Smile杰丶 - 博客园

glusterfs-note/document/md/nfs-ganesha/基于glusterfs的nfs-ganesha方案.md at master · perrynzhou/glusterfs-note · GitHub

glusterfs-note/document/md/nfs-ganesha/nfs-ganesha源码安装.md at master · perrynzhou/glusterfs-note · GitHub


熊熊

site logo

Hi,Friend

© 2026 cbear