DownloadStorage Server Installation and Configuration
From GlusterDocumentation
Contents
|
Installing GlusterFS on the Server
Technical Requirements
Hardware Requirements
GlusterFS is supported on most industry standard (x86-64) hardware with direct-attached-storage, RAID and FC/Infiniband/iSCSI SAN disk backends with SATA/SAS/FC disks.
Networking Requirements
GlusterFS supports Gigabit Ethernet, 10 Gigabit Ethernet, and InfiniBand.
Operating System Requirements
| OS | Architecture | Modes |
|---|---|---|
| CentOS 5.x | x86_64 | Client & Server |
| RedHat EL 5.x | x86_64 | Client & Server |
| Solaris 10 | x86_64 | Server only |
| RedHat EL 5.x | x86_64 | Client & Server |
| Ubuntu 8.04.x LTS | x86_64 | Client & Server |
| Ubuntu 10.04 LTS | x86_64 | Client & Server |
| Fedora 11 | x86_64 | Client & Server |
Note: While the 3.x versions of Gluster will compile on 32bit systems we do not QA or test on 32-bit systems. We strongly suggest you do NOT run Gluster in a 32-bit environment.
File Systems
We recommend Ext3 or Ext4. Any other POSIX compliant disk file system such as ZFS, ReiserFS, btrfs, or JFS may also work, but have not been tested widely. XFS has known performance problems with their extended attributes implementation, if you use XFS your Gluster performance will be cut by at least 60%.
Preparing to Install GlusterFS on the Server
This section describes how to locate and install the latest version of GlusterFS for your operating system.
Downloading GlusterFS
To download GlusterFS, go to:
Select the package or tarball for your operating system.
Installing on GNU/Linux
Depending on your Linux distribution, use one of the following methods.
Red Hat Package Manager (RPM) based Distributions
To install the GlusterFS package on Fedora, OpenSUSE, Red Hat, or CentOS Linux distributions, run the following commands:
bash# rpm -ivh glusterfs-*-[VERSION].rpm bash# glusterfs --version
The build version information displays, for example:
glusterfs 2.0.3 built on Jul 7 2009 12:22:40 Repository revision: 9ec22fd9fc28684b7e2093b4c67d449c7f021623 Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
You can also build a binary RPM from source using the following command on Red Hat Linux:
bash# rpmbuild -ta glusterfs-[VERSION].tar.gz
Debian-based Distributions
You can get GlusterFS from www.debian.org in the testing and unstable repositories. Currently, there are few contributors maintaining the GlusterFS Debian package. (You can get the latest package by installing from source.)
bash# apt-get install glusterfs-* bash# glusterfs --version
The build information is displayed, for example:
glusterfs 2.0.3 built on Jul 7 2009 12:22:40 Repository revision: 9ec22fd9fc28684b7e2093b4c67d449c7f021623 Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
Installing from Source
(As of November 2009, the following additional packages are required in order to build from source. Others may be needed as well, depending on your system: a Make utility, a C compiler such as gcc, ld, Flex or lex, and bison)
You can get the source tarball from the ftp repository of the project. To install the tar file from source, enter the following commands:
bash# tar -xzf glusterfs-version.tar.gz bash# cd glusterfs-version bash# ./configure --prefix= > /dev/null GlusterFS configure summary =========================== FUSE client : yes Infiniband verbs : yes epoll IO multiplex : yes Berkeley-DB : yes libglusterfsclient : yes argp-standalone : no bash# make && make install bash# ldconfig bash# glusterfs --version
The build information is displayed, for example:
glusterfs 2.0.3 built on Jul 7 2009 12:22:40 Repository revision: 9ec22fd9fc28684b7e2093b4c67d449c7f021623 Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
Installing on Mac OS X
Mac OS X users can build from the source tarball without problems, or install the .dmg images available from the GlusterFS ftp site. After downloading, click on the .dmg image to get a GlusterFS package.
- If you are installing the package locally, click to install.
- If you are installing on a remote machine from a terminal, enter the following commands:
bash# hdiutil attach glusterfs-version.dmg bash# installer -pkg /Volumes/glusterfs-version/glusterfs-version.pkg -target / bash# hdiutil detach /Volumes/glusterfs-version/ bash# glusterfs --version
The build information is displayed, for example:
glusterfs 2.0.3 built on Jul 7 2009 12:22:40 Repository revision: 9ec22fd9fc28684b7e2093b4c67d449c7f021623 Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
Note: For complete installation steps for your version of Mac OS X, read the README.MacOS file available with the .dmg image.
Installing on Solaris
To install GlusterFS for Solaris, perform the following steps:
Solaris Dependencies: Following software packages should be installed before compiling GlusterFS source.
- SUNWarc - system SUNWarc Lint Libraries (usr)
- SUNWhea - system SUNWhea SunOS Header Files
- SUNWgcc gcc - The GNU C compiler
- SUNWbinutils binutils - GNU binutils
- pkg-get - /opt/csw/bin
- GNU automake - /opt/csw/bin -- Automake 1.9 works.
- GNU autoconf - /opt/csw/bin
- GNU M4 - /opt/csw/bin
- GNU Bison 2.3 - /opt/csw/bin (GlusterFS works with this Bison)
- GNU Flex 2.5.4 - /opt/csw/bin (GlusterFS works with this Flex)
Notes:: Flex is provided by SUNWflexlex package, installed /usr/sfw/bin path. GlusterFS is incompatbile with SUNWflexlex. Use GNU Flex instead. To install GNU Flex:
# pkg-get install flex
Note: Similarly use GNU make utility and not Solaris make.
Build Instructions:
# bash bash-3.00# export PATH=/opt/csw/bin:/usr/sfw/bin:$PATH bash-3.00# gunzip -d glusterfs-VERSION.tar.gz bash-3.00# tar -xf glusterfs-VERSION.tar bash-3.00# cd glusterfs-VERSION bash-3.00# LDFLAGS="-L/opt/csw/lib" ./configure --disable-fuse-client --disable-ibverbs bash-3.00# make CFLAGS="-g -O0" LDFLAGS="-L/opt/csw/lib" bash-3.00# make install bash-3.00# /usr/local/sbin/glusterfs --version
Working Build Example with Solaris 10 10/09 s10x_u8wos_08a X86:
- Secure copy sol-10-u8-ga-x86-dvd.iso to system if you don't have a USB or DVD to attach.
lofiadm -a /zpool1/foobar1/sol-10-u8-ga-x86-dvd.iso /dev/lofi/1 
mount -F hsfs -o ro /dev/lofi/1 /vcdrom
cd /mnt/vdcrom/Solaris_10/Product
cp -r SUNWarc /var/spool/pkg
cp -r SUNWhea /var/spool/pkg
cp -r SUNWgcc /var/spool/pkg
cp -r SUNWbinutils /var/spool/pkg
pkgadd
- ar is also needed, and won't be in your path
PATH=/opt/csw/bin:/usr/sfw/bin:/usr/ccs/bin$PATH export PATH
pkgadd -d http://www.opencsw.org/pkg_get.pkg
/opt/csw/pkg-get -i automake
* automake-1.9.6 is NEEDED
pkg-get install autoconf
pkg-get install gm4
pkg-get install bison
pkg-get install flex
pkg-get install wget
cd /install
wget http://ftp.gluster.com/pub/gluster/glusterfs/3.0/LATEST/glusterfs-3.0.4.tar.gz
gunzip -d glusterfs-3.0.4.tar.gz
tar -xv fglusterfs-3.0.4.tar.gz
cd /install/glusterfs-3.0.4
LDFLAGS="-L/opt/csw/lib" ./configure --disable-fuse-client --disable-ibverbs
- Source has make hardcoded, just sym link it.
ln -s /usr/sfw/bin/gmake /usr/sfw/bin/make
- automake 1.11 is odd, and alternatives in Solaris is clunky.
- rm automake, and sym link 1.9.
rm /opt/csw/bin/automake ln -s /opt/csw/bin/automake-1.9 /opt/csw/bin/automake
make CFLAGS="-g -O0" LDFLAGS="-L/opt/csw/lib"
make install
The build information is displayed, for example:
glusterfs 2.0.3 built on Jul 7 2009 12:22:40 Repository revision: 9ec22fd9fc28684b7e2093b4c67d449c7f021623 Copyright (c) 2006-2009 Gluster Inc. <http://www.gluster.com>
Configuring GlusterFS
This section describes how to configure volumes using the glusterfs-volgen command on your server. These volume files are used by the client and the server.
Before configuring volumes, you need to know about the hardware and business requirements of your storage cluster. For example:
- How many servers can you dedicate to storage?
- Do you have critical data that should be replicated?
If you are not sure of the answers, discuss these issues with your team before proceeding.
For the reference page of the glusterfs-volgen command, see glusterfs-volgen.
NOTE: This documentation holds good only for "glusterfs-volgen" from 2.0.9 and 3.0.x releases
Configuring Volumes
Configuring Distributed Volumes
To create a simple distributed volume on multiple servers, follow the --name argument with the name of the store and the name or IP address of each host. (There are four servers in the following example.)
$ glusterfs-volgen --name store1 hostname1:/export/sdb1 hostname2:/export/sdb1 hostname3:/export/sdb1 hostname4:/export/sdb1
This will generate five files, four of which are for the servers and the other being the client volume file.
hostname1-store1-export.vol hostname2-store1-export.vol hostname3-store1-export.vol hostname4-store1-export.vol store1-tcp.vol
Here the hostname*.vol files are the server volume files and store1-tcp.vol is the client volume file.
If you want your clients to automatically fetch the volume file from the server, you need to copy the store1-tcp.vol to one of the servers (for example, hostname1) under the name /etc/glusterfs/glusterfs.vol. This will allow your clients to mount by just doing:
# glusterfs --volfile-server=hostname1 /mnt/glusterfs
You can also mount it by doing:
# mount -t glusterfs hostname1:6996 /mnt/glusterfs
If you specified a different port (using -p <port>) to glusterfs-volgen, then replace 6996 with that number.
Configuring Replicated Volumes
To create distributed replicated volume of multiple servers:
- Follow the --name argument with the name of the store and the name or IP address of each host. (There are four servers in the following example.)
- Enter raid 1 for replication.
$ glusterfs-volgen --name repstore1 --raid 1 hostname1:/export/sdb1 hostname2:/export/sdb1 hostname3:/export/sdb1 hostname4:/export/sdb1
Configuring Volumes over InfiniBand
To create a distributed volume over InfiniBand with 2GB client caching:
$ glusterfs-volgen --name ibstore1 --transport ib-verbs hostname1:/export/sdb1 hostname2:/export/sdb1 hostname3:/export/sdb1 hostname4:/export/sdb1
To create a striped volume over InfiniBand:
$ glusterfs-volgen --name ibstripe1 --transport ib-verbs --raid 0 hostname1:/export/sdb1 hostname2:/export/sdb1 hostname3:/export/sdb1 hostname4:/export/sdb1
Starting GlusterFS Service
To start or stop GlusterFS filesystem service manually,
# /etc/init.d/glusterfsd [start|stop]
NOTE: Now after generating volume files it would created as <hostname>-<volume-name>-export.vol rename this to glusterfsd.vol if it needs to be started with init scripts.
To automatically start GlusterFS service every time when system boots follow the procedure below.
Red Hat based
From the command line, type
# chkconfig --add glusterfsd # chkconfig --levels 2345 glusterfsd on
To interactively configure run level service, you can also use a curses based ntsysv tool.
Debian GNU/Linux, Ubuntu based
Use update-rc.d or curses based rcconf interactive tool
Generic Approach
Make an entry in your /etc/rc.local file.
# echo "glusterfsd" >> /etc/rc.local
Optional Configuration
The following sections on unfs and CIFS are optional, depending on your implementation.
Add NFSv3 Protocol
This procedure adds NFSv3 protocol on top of GlusterFS using an enhanced version of UNFSv3 server. It has bug fixes and performance enhancements.
Download
Red Hat Enterprise Linux 5.x (x86_64):
http://ftp.gluster.com/pub/gluster/glusterfs/misc/unfs3/unfs3-0.9.23gluster1-1.el5.rpm
Source:
http://ftp.gluster.com/pub/gluster/glusterfs/misc/unfs3/unfs3-0.9.23gluster1.tar.gz
Source RPM:
http://ftp.gluster.com/pub/gluster/glusterfs/misc/unfs3/unfs3-0.9.23gluster1-1.el5.src.rpm
Install
Install the RPM on Red Hat Enterprise Linux:
$ rpm -ivh unfs3-0.9.23gluster1-1.el5.rpm
Install using the source tarball:
$ tar -xzf unfs3-0.9.23gluster1.tar.gz $ cd /unfs3-0.9.23gluster1 $ ./bootstrap $ ./configure $ make;make install
Create an exports file to use with the unfs3 server. The syntax for the unfs3 file is similar to the Linux NFS server /etc/exports file, but with fewer options. (See unfsd(8) for a list of all supported options.)
In the exports file, each line specifies an exported directory along with the options that control this export. The following are some common sample unfs3 configuration lines:
#Only one client and allow both read-write. /exports 192.168.1.201(rw) #Do not re-map root UID to anonymous user. /data/maildirs 192.168.1.202(rw,no_root_squash) #Allow clients from "trusted" to mount read-write without root UID squashing. #For all others, allow only read-only. /home trusted(rw,no_root_squash) (ro) #Allow machine "weirdo" to mount export directory using default options. "/with spaces" weirdo #Allow one specific client read-write access and allow machines from #a subnet read-only access while squashing all UIDs to anonymous user. /usr 1.2.3.4(rw) 192.168.2.0/24(ro,all_squash) #Another way of specifying netmask. /home/foo bar(rw) 10.0.0.0/255.0.0.0(root_squash) #Allow explicit specification of the anonymous UID. /home/joe joes_pc(anonuid=1100,anongid=1100,rw,all_squash)
Starting the unfs3 Server
Create an exports file in /etc/unfs3exports on the NFS server.
/testpath 192.168.0.0/16(rw,no_root_squash)
Start the unfs3 server
$ unfsd -e /etc/unfs3exports
Add CIFS protocol
CIFS support for GlusterFS can be configured with Samba. Add sections to the smb.conf file so that the path points to directories inside the GlusterFS mount point. In the following example, /mnt/gluster is the mount point:
[test2] comment = Sample export via SAMBA valid users = user3 printable = no writable = yes public = no path = /mnt/gluster/user3_data posix locking = no
Next Steps
Now, you are ready to install the client, see Client Installation and Configuration.
