Veritas Cluster I/O Fencing, Part 2

To follow up from part 1, this post will go into a bit more detail how to set up your I/O fencing for your cluster. This example is using Solaris 10 and VCS 5.0.

Before you continue, you need to have your three coordinator disks and all your data disks visible to all the cluster nodes. All these disks should have the SCSI-3 Persistent Reservation bit set. Put your three coordinator disks into their own disk group. I call mine dgvxfencoord.

1. Testing your shared disks
VCS includes a tool to test weather your disks have the SCSI-3 PR bit set, called vxfentsthdw. You only need to run this on one node in the cluster, and using ssh/rsh, the tool will perform the tests for all the nodes. You’ll need to set up ssh keys (or .rhosts) so root on your first cluster node can log into the other cluster nodes with no password – this is just temporary so our testing will work. If you use rsh, just use the -n flag with vxfentsthdw.The easiest way to use this tool is to specify disk groups to test. For example:

# /opt/VRTSvcs/vxfen/bin/vxfentsthdw -g dgvxfencoord

This will test every disk in the dgvxfencoord disk group, reading/writing keys to the disks, locking them for exclusive use by each member of the cluster. You should see “PASSED” after each test.

While testing your data disks, you may want to use the -r flag, to use non-destructive testing, if you have data on your disks already:

/opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -g datadg

Once your tests indicate that SCSI-3 Persistent Reservations are working, you’re ready to move on. Your diskgroup with the coordinator disks never needs to be imported at all, since there are no file systems or other data on them.

vxdg deport dgvxfencoord
vxdg -t import dgvxfencoord #(turns off automatic importing when system starts)
vxdg -g dgvxfencoord set coordinator=on
vxdg deport dgvxfencoord

2. Perform these steps on each node of the cluster to set up I/O Fencing
I’m a big fan of copy and paste from online documentation, so here you go. This will tell the vx fencing kernel driver to use a diskgroup of “dgvxfencoord” for the coordinator disks, tell it to use SCSI-3 with dynamic multipathing, and will then restart the vx fencing service.

echo “dgvxfencoord” > /etc/vxfendg
cp /etc/vxfen.d/vxfenmode_scsi3_dmp /etc/vxfenmode
/etc/init.d/vxfen stop
/etc/init.d/vxfen start

3. VCS Configuration
So once you have the vx fencing driver set up, you have to tell your cluster to use it. First, stop your cluster and resources:

haconf -dump -makero
hastop -all

Then hand-edit the main.cf file in /opt/VRTSvcs/conf/config. Insert one line within the cluster definition block. Here’s an example:

cluster BIG-CLUSTER4
UserNames = { admin = “cERpdxPmHpzS.” }
Administrators = { admin }
ClusterAddress = “192.168.65.144″
UseFence = SCSI3
)

Once you insert your line, it’s a good idea to check the syntax of main.cf:

hacf -verify /etc/VRTSvcs/conf/config

Then, copy the updated main.cf file from this node to the other nodes using your preferred method – rcp, scp, ftp, whatever.

Then on each node, “hastart”.

You can verify the fencing configuration with this:

#/sbin/vxfenadm -d

I/O Fencing Cluster Information:
================================

Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:

* 0 (server1)
1 (server2)

RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)

4. Testing your fencing setup
You’ll want to test this before going into production. I have used a few methods to test this, but these are the easiest.

1. If you have physical access to your server
Unplug the two heartbeat links and the public network link. Fencing should kick in and the nodes will all race for those cooridator disks. The winner will take control, and the other cluster nodes will panic and reboot. Have a console connection on the nodes to verify.

2. If you have switch access, or access to someone who does
An easy thing to do is to disable the network ports corresponding to the heartbeats and public network link. Almost the same as #1.

3. If you want to perform the test by yourself and have no physical access
Set up scripts to change the speed/duplex on the NICs running your heartbeats and public network. Do this from the serial console, so you don’t lose access obviously (I’ve done similar things – quite a few times). Once your switch is still auto/auto and your NIC is forced to 10-half with no auto-negotiation, communication will be impossible and effectively you’ve severed your links.

Happy clustering!