Gluster FS is a scalable, distributed file system that aggregates disk storage resources from multiple servers into a single global namespace.Advantages
Gluster FS Architecture:
Setup GlusterFS Cluster in Oracle Cloud Infrastructure
Setting up OCI Virtual Machine Instances
- Create two virtual machine instances in OCI. You can create virtual machine instances in OCI using the console or REST apis or any automation tool of your choice. The instructions in this blog work on centos 7 operating system for both VM instances.
- You need to make sure that the two instances that you have created are able to talk to each other. If the two instances are in the same subnet, they should be able to talk to each other automatically. if they are in separate subnets edit the security rule and open the security list to enable the instances in the two subnets to access each other.
- In the interest of this blog, I have named my instances instance01 and instance02.
- Make an entry in the /etc/hosts file with the IP and hostname of both VM instances as shown below. The IP's should be resolvable using the hostnames. Change the IP in the following to your server IP's.
- Create two storage volumes in oci (using oci console) and attach one to each OCI VM instance.
Instance Port Configuration
Open the below ports using security rules in both subnet security lists as well the server firewall if it is enabled
111 - 24007 GlusterFS Daemon
24008 - GlusterFS Management
38465 to 38467 - GlusterFS NFS service
49152 to 60000 - Depends on number of bricks
If you are using Oracle Linux, then follow below additional steps highlighted in purple else skip this one
- Edit file /etc/yum.conf and add a line assumeyes=1 at the last
- Create a file under /etc/yum.repos.d/ by the name of Gluster.repo and add below content in it
- On all servers - Login as root user and execute below commands- yum update -y
yum install xfsprogs attr -y
yum install centos-release-gluster -y
yum install glusterfs-server glusterfs-client glusterfs-common -y
vi /etc/sysconfig/selinux and disabled it
systemctl enable glusterd
service glusterd start
firewall-cmd --zone=public --add-port=24007-24009/tcp --add-port=49152-49251/tcp --permanent
Create Mount points for OCI Block storage volumes
sudo mkfs -t xfs /dev/sdb
You need to do the following in both the OCI virtual machine instances. Format the volume to xfs.sdb is the name of the oci block storage volumes. You can list the available devices using lsblk command.Create a mount directory named /gshare and mount the formatted volume
sudo mkdir /gshare sudo mount /share /dev/sdbAdd the mount to /etc/fstab
/dev/sdb /gshare xfs defaults,nofail 0
GlusterFS installation on Virtual machine instances
You need to perform the steps below on both virtual machine instances
- Create a GlusterFs repo.
- Install the glusterFS server on the VM instances
- Start the glusterd service
- Create a GlusterFS HA shared volume
[opc@instance01 ~]$ sudo gluster volume create gdata replica 2 instance01:/gshare/data instance02:/gshare/data
- Start the gdata volume
[opc@instance01 ~]$ sudo gluster volume start gdata
- Create a third virtual machine instance in OCI. You can create virtual machine instances in OCI using the console or REST apis or any automation tool of your choice. The virtual machines instance is also based on Centos 7 operating system. Once the instance is up and running, open the oci console and navigate to this instance and look at the public ip of the newly created instance. The initial three digits of the ip address gives you the CIDR range that needs to be enabled in this step. (e.g. 152* as in this example)
Set the volume permissions for gdata volume for client access. Here I am using 152.* CIDR. You need to replace it based on your network range of your GlusterFS client instance subnet
[opc@instance01 ~]$ sudo gluster volume set gdata auth.allow "152.*"
- To get all the info about the volume, execute the following command
- Enable fuse kernel module
[opc@instanceglusterclient ~]$ sudo modprobe fuse
- Install all the glusterFS client dependencies
[opc@instanceglusterclient ~]$ sudo yum install fuse fuse-libs openib libibverbs -y
- Install the GlusterFS client
- Create a client mount directory
[opc@instanceglusterclient ~]$ sudo mkdir /gfdata
- Mount gfdata directory to the glusterFS replicated volume
sudo vi /etc/yum.repos.d/Gluster.repo
Add and enable the following repo
sudo yum install glusterfs-server -y
[opc@instance01 ~]$ sudo systemctl start glusterd
[opc@instance01 ~]$ sudo systemctl status glusterd
Output: glusterd.service - GlusterFS, a clustered file-system serverLoaded: loaded (/usr/lib/systemd/system/glusterd.service;
vendor preset: enabled)Active: active (running) since Sat 2020-02-29 03:04:09 GMT;
8s agoDocs: man:glusterd(8)Process: 6556 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS) Main PID: 6557 (glusterd)
CGroup: /system.slice/glusterd.service 6557 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INF...Feb 29 03:04:09 instance01 systemd: Starting GlusterFS, a clustered file.....Feb 29 03:04:09 instance01 systemd: Started GlusterFS, a clustered file-...r
Configure the GlusterFS server just installed on both VM instances
From instance01 execute the following command to create a trusted storage pool with instance02.
[opc@instance01 ~]$ sudo gluster peer probe instance02
peer probe: success.
2. Check the peer status using the following command
[opc@instance01 ~]$ sudo gluster peer status
Number of Peers: 1Hostname: instance02Uuid: ec83924b-877e-4fe8-b58a-b7541285999fState: Peer in Cluster (Connected)
3. Create a data directory on gshare mount on both virtual machine instances
[opc@instance01 ~]$ sudo mkdir /gshare/data
[opc@instance01 ~]$ sudo gluster volume info gdata
Volume Name: gdataType: ReplicateVolume ID: 699ec5a2-55c1-43a2-94c1-2a8779f32d4cStatus: StartedSnapshot Count: 0Number of Bricks: 1 x 2 = 2Transport-type: tcpBricks:Brick1: instance01:/gshare/dataBrick2: instance02:/gshare/dataOptions Reconfigured:auth.allow: 152.*transport.address-family: inetstorage.fips-mode-rchecksum: onnfs.disable: onperformance.client-io-threads: off Start the GlusterFS Client Setup on third instance created in step7 above
[opc@instanceglusterclient ~]$ sudo yum install glusterfs-client -y
GlusterFS Client Configuration
The data will be replicated if you are writing from the GlusterFS client. You can mount the GlusterFS volume to any number of clients. A client machine is not part of the glusterFS cluster. It is the machine in which you want to mount the replicated volume.
[opc@instanceglusterclient ~]$ sudo mount -t glusterfs instance01:/gdata /gfdata
You are all done !!!! . Your two node GlusterFS is working now !!!!
About Vast Edge - Simplified Cloud Solutions
Founded in the year 2004, Vast Edge is a leading IT Consulting Company and Global Service provider of Business Intelligence, big data analytics, cloud ERP, IoT platform, enterprise backup, and disaster recovery, Blockchain, Cassandra, AI/ML, and Integration solutions. We perform Application modernization, follow continuous change management, and implement advanced practices while working on the key areas of enhancing methodologies of project development.
Get a Free Assessment from our cloud experts.