For some reason, configuring link-based IPMP in VCS is somewhat tricky, and the documentation doesn’t seem to help much. It seems all the default values for VCS are for probe-based IPMP only.

To achieve link-based IPMP, here’s how I’ve configured my MultiNICB resource:

Link-based IPMP MultiNICB properties

UseMpathd: 1
Tells VCS to use mpathd for network link status
MpathCommand: /usr/lib/inet/in.mpathd -a
The default, /usr/sbin/in.mpathd is just incorrect – it doesn’t live there.
ConfigCheck: 0
If you leave this at 1, it will overwrite your /etc/hostname.xxx files with probe-based IPMP configuration
Device: (your IPMP interfaces here)
The “interface alias” for each device is not needed, leave them blank.
IgnoreStatus: 0
You want VCS to NOT ignore link status, since this is how link-based IPMP works.
GroupName:
Do not use your IPMP group name here, it’s not needed. VCS is not monitoring the group, mpathd is.

Here’s how it looks in main.cf:

MultiNICB csgmultinic (
UseMpathd = 1
MpathdCommand = “/usr/lib/inet/in.mpathd -a”
ConfigCheck = 0
Device = { ce0 = “”, ce4 = “” }
IgnoreLinkStatus = 0
)

Dec 11 16:39:20 server llt: [ID 525299 kern.notice] LLT WARNING V-14-1-10497 crossed links? link 0 and link 1 of node 0 on the same network

Before you continue, you need to have your three coordinator disks and all your data disks visible to all the cluster nodes. All these disks should have the SCSI-3 Persistent Reservation bit set. Put your three coordinator disks into their own disk group. I call mine dgvxfencoord.

1. Testing your shared disks
VCS includes a tool to test weather your disks have the SCSI-3 PR bit set, called vxfentsthdw. You only need to run this on one node in the cluster, and using ssh/rsh, the tool will perform the tests for all the nodes. You’ll need to set up ssh keys (or .rhosts) so root on your first cluster node can log into the other cluster nodes with no password – this is just temporary so our testing will work. If you use rsh, just use the -n flag with vxfentsthdw.The easiest way to use this tool is to specify disk groups to test. For example:

# /opt/VRTSvcs/vxfen/bin/vxfentsthdw -g dgvxfencoord

This will test every disk in the dgvxfencoord disk group, reading/writing keys to the disks, locking them for exclusive use by each member of the cluster. You should see “PASSED” after each test.

While testing your data disks, you may want to use the -r flag, to use non-destructive testing, if you have data on your disks already:

/opt/VRTSvcs/vxfen/bin/vxfentsthdw -r -g datadg

Once your tests indicate that SCSI-3 Persistent Reservations are working, you’re ready to move on. Your diskgroup with the coordinator disks never needs to be imported at all, since there are no file systems or other data on them.

vxdg deport dgvxfencoord
vxdg -t import dgvxfencoord #(turns off automatic importing when system starts)
vxdg -g dgvxfencoord set coordinator=on
vxdg deport dgvxfencoord

2. Perform these steps on each node of the cluster to set up I/O Fencing
I’m a big fan of copy and paste from online documentation, so here you go. This will tell the vx fencing kernel driver to use a diskgroup of “dgvxfencoord” for the coordinator disks, tell it to use SCSI-3 with dynamic multipathing, and will then restart the vx fencing service.

echo “dgvxfencoord” > /etc/vxfendg
cp /etc/vxfen.d/vxfenmode_scsi3_dmp /etc/vxfenmode
/etc/init.d/vxfen stop
/etc/init.d/vxfen start

3. VCS Configuration
So once you have the vx fencing driver set up, you have to tell your cluster to use it. First, stop your cluster and resources:

haconf -dump -makero
hastop -all

Then hand-edit the main.cf file in /opt/VRTSvcs/conf/config. Insert one line within the cluster definition block. Here’s an example:

cluster BIG-CLUSTER4
UserNames = { admin = “cERpdxPmHpzS.” }
Administrators = { admin }
ClusterAddress = “192.168.65.144″
UseFence = SCSI3
)

Once you insert your line, it’s a good idea to check the syntax of main.cf:

hacf -verify /etc/VRTSvcs/conf/config

Then, copy the updated main.cf file from this node to the other nodes using your preferred method – rcp, scp, ftp, whatever.

Then on each node, “hastart”.

You can verify the fencing configuration with this:

#/sbin/vxfenadm -d

I/O Fencing Cluster Information:
================================

Fencing Protocol Version: 201
Fencing Mode: SCSI3
Fencing SCSI3 Disk Policy: dmp
Cluster Members:

* 0 (server1)
1 (server2)

RFSM State Information:
node 0 in state 8 (running)
node 1 in state 8 (running)

4. Testing your fencing setup
You’ll want to test this before going into production. I have used a few methods to test this, but these are the easiest.

1. If you have physical access to your server
Unplug the two heartbeat links and the public network link. Fencing should kick in and the nodes will all race for those cooridator disks. The winner will take control, and the other cluster nodes will panic and reboot. Have a console connection on the nodes to verify.

2. If you have switch access, or access to someone who does
An easy thing to do is to disable the network ports corresponding to the heartbeats and public network link. Almost the same as #1.

3. If you want to perform the test by yourself and have no physical access
Set up scripts to change the speed/duplex on the NICs running your heartbeats and public network. Do this from the serial console, so you don’t lose access obviously (I’ve done similar things – quite a few times). Once your switch is still auto/auto and your NIC is forced to 10-half with no auto-negotiation, communication will be impossible and effectively you’ve severed your links.

Happy clustering!

Here’s a scenario: There is a partial power outage, IOS upgrade failure, spanning tree loop, or anything else that can cause multiple network switch failures in your data center. Because of this, your cluster nodes can no longer communicate using ANY of their heartbeat links, or public network. Without I/O fencing enabled, each node would believe all the other nodes were “down”, and try to perform a failover and run all the defined service groups in the cluster. Multiple nodes trying to read/write to the same storage may result in data corruption, and this is your worst case scenario. I/O fencing will help here.

There is a SCSI-3 feature called “SCSI-3 Persistent Reservation”, which allows cluster nodes to write “keys” to shared disks, effectively locking the disk for exclusive use by a node.

Ask your storage administrator to enable SCSI-3 Persistent Reservation on each LUN you are assigned. On some arrays, this is the default behavior, but others require you to turn on the feature per LUN.

All the cluster nodes must be assigned three small “coordinator disks”, which serve as a locking mechanism for the shared storage. Just three shared disks per cluster, and all nodes must have access to them.

In the same scenario above, when the heartbeats go down and nodes are thought to be “offline”, each surviving node will race for control of the coordinator disks, ejecting any other nodes’ keys and writing their own key to the disk, locking other nodes out.

If there is more than one surviving node in the cluster, the “loser” of the race will actually panic and reboot. That’s not a typo – the node will kernel panic and reboot. This is the only sure way to ensure the node will not proceed and potentially corrupt data on the shared storage.

Consult your VCS documentation for the setup steps to enable I/O Fencing. I will be posting a “part 2″ also, with my abbreviated version to get it up and running.

We use SRDF replication here, and the fd-srdf script provided with the SRDF agent only copies over a small percentage of the resources in our service groups (no zones, no IPs, no Oracle resources, etc). I modified it to grab all of these things and copy them over. The special things about this script is that it changes all the disk groups and mounts to diskgroup_fd resources for the names and mount points for a firedrill service group. Helps if you have >50 mount points like me.