Installation
From Gfswiki
How to install and run clvm and gfs (imported from Usage.txt (http://sources.redhat.com/cgi-bin/cvsweb.cgi/%7Echeckout%7E/cluster/doc/usage.txt?rev=1.1&content-type=text/plain&cvsroot=cluster)).
Refer to the cluster project page for the latest information. http://sources.redhat.com/cluster/
| Table of contents |
Get source
- download the source tarballs
latest linux kernel - ftp://ftp.kernel.org/pub/linux/ device-mapper - ftp://sources.redhat.com/pub/dm/ lvm2 - ftp://sources.redhat.com/pub/lvm2/ iddev - ftp://sources.redhat.com/pub/cluster/ ccs - ftp://sources.redhat.com/pub/cluster/ fence - ftp://sources.redhat.com/pub/cluster/ cman - ftp://sources.redhat.com/pub/cluster/ cman-kernel - ftp://sources.redhat.com/pub/cluster/ dlm - ftp://sources.redhat.com/pub/cluster/ dlm-kernel - ftp://sources.redhat.com/pub/cluster/ gfs - ftp://sources.redhat.com/pub/cluster/ gfs-kernel - ftp://sources.redhat.com/pub/cluster/
- or to download source from cvs see
http://sources.redhat.com/dm/ http://sources.redhat.com/lvm2/ http://sources.redhat.com/cluster/
summary: after cvs login, cvs -d :pserver:cvs@sources.redhat.com:/cvs/dm checkout device-mapper cvs -d :pserver:cvs@sources.redhat.com:/cvs/lvm2 checkout LVM2 cvs -d :pserver:cvs@sources.redhat.com:/cvs/cluster checkout cluster
Build and install
- Configure and compile your kernel
- build and install userland programs and libraries (order is important)
cd device-mapper ./configure make; make install
device-mapper build note: on Debian make sure you have either none of libselinux1 and libselinux1-dev installed, or both of them
# magma.h is needed by other parts, so install it first cd cluster/magma ./configure --kernel_src=/path/to/patched/kernel make && make install # now build everything in cluster cd ../ ./configure --kernel_src=/path/to/patched/kernel make; make install lvm2 ./configure --with-clvmd --with-cluster=shared --with-kernel-source=/path/to/patched/kernel make; make install chmod 755 scripts/clvmd_fix_conf.sh scripts/clvmd_fix_conf.sh /lib device-mapper ./configure --with-kernel-dir=/path/to/patched/kernel make; make install
Load kernel modules
depmod -a modprobe dm-mod device-mapper/scripts/devmap_mknod.sh modprobe gfs modprobe lock_dlm
Modules that should be loaded: lock_dlm, dlm, cman, gfs, lock_harness and dm-mod if device-mapper was built as a module
Startup procedure
Run these commands on each cluster node:
> ccsd # Starts the CCS daemon
> cman_tool join # Joins the cluster
> fence_tool join # Joins the fence domain (starts fenced, must
# start before any gfs stuff is used.)
> clvmd # Starts the CLVM daemon
> vgchange -aly # Activates LVM volumes (locally)
> mount -t gfs /dev/vg/lvol /mnt # Mounts a GFS file system
Shutdown procedure
Run these commands on each cluster node:
> umount /mnt # Unmounts a GFS file system > vgchange -aln # Deactivates LVM volumes (locally) > killall clvmd # Stops the CLVM daemon > fence_tool leave # Leaves the fence domain (stops fenced) > cman_tool leave # Leaves the cluster > killall ccsd # Stops the CCS daemon
Creating CCS config
There is no GUI or command line program to create the config file yet. The cluster config file, cluster.conf, must therefore be created manually. Once created, cluster.conf should be placed in the /etc/cluster/ directory on one cluster node. CCS daemon (ccsd) will take care of transferring it to other nodes where it's needed. (FIXME: updating cluster.conf in a running cluster is supported but not documented.)
A minimal cluster.conf example is shown below.
Creating CLVM logical volumes
Use standard LVM commands (see LVM documentation on using pvcreate, vgcreate, lvcreate.) A node must be running the CLVM system to use the LVM commands. Running the CLVM system means successfully running the commands above up through starting clvmd.
Creating GFS file systems
> gfs_mkfs -p lock_dlm -t <ClusterName>:<FSName> -j <Journals> <Device>
<ClusterName> must match the cluster name used in CCS config
<FSName> is a unique name chosen now to distinguish this fs from others
<Journals> the number of journals in the fs, one for each node to mount
<Device> a block device, usually an LVM logical volume
Creating a GFS file system means writing to a CLVM volume which means the CLVM system must be running (see previous section.)
Cluster startup/shutdown notes
Fencing: In the start-up steps above, "fence_tool join" is the equivalent of simply starting fenced. fence_tool is useful because additional options can be specified to delay the actual starting of fenced. Delaying can be useful to avoid unnecessarily fencing nodes that haven't joined the cluster yet. The only option fence_tool now provides to address this is "-t <seconds>" to wait the given number of seconds before starting fenced. Shutdown: There is also a practical timing issue with respect to the shutdown steps being run on all nodes when shutting down an entire cluster. When shutting down the entire cluster (or shutting down a node for an extended period) use "cman_tool leave remove". This automatically reduces the number of expected votes as each node leaves and prevents the loss of quorum which could keep the last nodes from cleanly completing shutdown. Using the "remove" leave option should not be used in general since it introduces potential split-brain risks.
If the "remove" leave option is not used, quorum will be lost after enough nodes have left the cluster. Once the cluster is inquorate, remaining members that have not yet completed "fence_tool leave" in the steps above will be stuck. Operations such as umounting gfs or leaving the fence domain ("fence_tool leave") will block while the cluster is inquorate. They can continue and complete only once quorum is regained.
If this happens, one option is to join the cluster ("cman_tool join") on some of the nodes that have left so that the cluster regains quorum and the stuck nodes can complete their shutdown. Another option is to forcibly reduce the number of expected votes for the cluster which allows the cluster to become quorate again ("cman_tool expected <votes>"). This later method is the equivalent to using the "remove" option when leaving.
Config file
This example primarily illustrates the variety of fencing configurations. The first node uses "cascade fencing"; if the first method fails (power cycling with an APC Masterswitch), the second is tried (port disable on a Brocade FC switch). In this example, the node has dual paths to the storage so the port on both paths must be disabled (the same idea applies to nodes with dual power supplies.) There is only one method of fencing the second node (via an APC Masterswitch) so no cascade fencing is possible. If no hardware is available for fencing, manual fencing can be used as shown for the third node. If a node with manual fencing fails, a human must take notice (a message appears in the system log) and run fence_ack_manual after resetting the failed node. (The node that actually carries out fencing operations is the node with the lowest ID in the fence domain.)
<?xml version="1.0"?>
<cluster name="alpha" config_version="1">
<cman>
</cman>
<clusternodes>
<clusternode name="nd01" votes="1">
<fence>
<method name="cascade1">
<device name="apc1" port="1"/>
</method>
<method name="cascade2">
<device name="brocade1" port="1"/>
<device name="brocade2" port="1"/>
</method>
</fence>
</clusternode>
<clusternode name="nd02" votes="1">
<fence>
<method name="single">
<device name="apc1" port="2"/>
</method>
</fence>
</clusternode>
<clusternode name="nd03" votes="1">
<fence>
<method name="single">
<device name="human" ipaddr="nd03"/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice name="apc1" agent="fence_apc2" ipaddr="10.1.1.1" login="apc" passwd="apc"/>
<fencedevice name="brocade1" agent="fence_brocade" ipaddr="10.1.1.2" login="user" passwd="pw"/>
<fencedevice name="brocade2" agent="fence_brocade" ipaddr="10.1.1.3" login="user" passwd="pw"/>
<fencedevice name="human" agent="fence_manual"/>
</fencedevices>
</cluster>
Multiple clusters
When multiple clusters are used, it can be useful to specify the cluster name on the cman_tool command line. This forces CCS to select a cluster.conf with the same cluster name. The node then joins this cluster.
> cman_tool join -c <ClusterName>
[Note: If the -c option is not used, ccsd will first check the local copy of cluster.conf to extract the cluster name and will
only grab a remote copy of cluster.conf if it has the same cluster name and a greater version number. If a local copy of
cluster.conf does not exist, ccsd may grab a cluster.conf for a different cluster than intended -- cman_tool would then report an
error that the node is not listed in the file.
So, if you don't currently have a local copy of cluster.conf (and there are other clusters running) or you wish to join a different cluster with a different cluster.conf from what exists locally, you must specify the -c option.]
Two node clusters
Ordinarily the loss of quorum after one node fails out of two will prevent the remaining node from continuing (if both nodes have one vote.) Some special configuration options can be set to allow the one remaining node to continue operating if the other fails. To do this only two nodes with one vote each can be defined in cluster.conf. The two_node and expected_votes values must then be set to 1 in the cman config section as follows.
<cman two_node="1" expected_votes="1"> </cman>
Advanced Network Configuration
Multihome
CMAN can be configured to use multiple network interfaces. If one fails it should be able to continue running with the one remaining. A node's name in cluster.conf is always associated with the IP address on one network interface; "nd1" in the following:
<node name="nd1" votes="1"> </node>
To use a second network interface, the node must have a second hostname associated with the IP address on that interface; "nd1-e1" in the following. The second hostname is specfied in an "altname" section.
<node name="nd1" votes="1">
<altname name="nd1-e1"/>
</node>
Multicast
CMAN can be configured to use multicast instead of broadcast (broadcast is used by default if no multicast parameters are given.) To configure multicast when one network interface is used add one line under the <cman> section and another under the <node> section:
<cman>
<multicast addr="224.0.0.1"/>
</cman>
<node name="nd1" votes="1">
<multicast addr="224.0.0.1" interface="eth0"/>
</node>
The multicast addresses must match and the address must be usable on the
interface name given for the node.
When two interfaces are used, multicast is configured as follows:
<cman>
<multicast addr="224.0.0.1"/>
<multicast addr="224.0.0.9"/>
</cman>
<node name="nd1" votes="1">
<altname name="nd1-e1"/>
<multicast addr="224.0.0.1" interface="eth0"/>
<multicast addr="224.0.0.9" interface="eth1"/>
</node>
IPv6
- When using multiple interfaces, all must use the same address family.
