Of course you could have prevented this by breaking your root mirror before installing the patches, or by using LiveUpgrade. But I know sometimes we just don’t do these things, for various reasons.

One solution to this is to boot from the network into single-user mode, mount your root disk, and back out the patch or patches on disk, hopefully repairing the damage and returning you to a bootable state. Of course this is assuming you have a jumpstart server on your network as well. NOTE: I tested this with a Solaris 10 06/06 boot image on the jumpstart server – I haven’t tested earlier versions)

Where I am, all the root disks are mirrored with SVM. An procedure I’d used in the past was to boot from the network, run patchrm on the first disk in the mirror to back out the patch, and then disable the mirror, so the second disk would not be used when rebooted. Then re-mirror later. This was rather tedious and error-prone, especially with multiple metadevices, soft partitions, etc. I found a new way to accomplish this task: keep the mirror intact and back out the patches while booted from the network. Our systems also use zones, and their zonepaths are on soft partitions. This procedure will also back out the patches from the zones. Here is the exact procedure:

1. Boot from network into single-user mode
ok> boot net -s
2. Mount root file system READ ONLY from the first disk in the mirror:
# mount -o ro /dev/dsk/c1t0d0s0 /mnt
3. Copy the SVM configuration to the running OS:
# cp /mnt/kernel/drv/md.conf /kernel/drv/md.conf
4. Unmount the root disk
# umount /mnt
5. Update the SVM driver to load the new configuration (ignore error messages)
# update_drv -f md
6. Set up metadevices in configuration
# metainit -r
7. Run metasync on root mirror metadevice
# metasync d10
8. Mount root metadevice on /mnt
# mount /dev/md/dsk/d10 /mnt
9. If the system has zones, run metasync on the metadevice containing the soft partitions, and mount all zone root file systems
# metasync d40
# mount /dev/md/dsk/d53 /mnt/zones/zonepath1
# mount /dev/md/dsk/d56 /mnt/zones/zonepath2
10. Rollback the failed patch.
# patchrm -R /mnt $patch 2>&1 | tee -a /mnt/backout.log
11. umount /mnt and reboot server


Sun makes available several patch bundles that many customers use for patching. The most popular bundle is the Recommended patch cluster for your Solaris version. This patch cluster provides the latest revisions of any patch that addresses a Sun Alert, which is any fix for security, data corruption or system availability. This is well-tested combination of patches, known to be stable and compatible.

A second patch cluster is also available from Sun – the Sun Alert patch cluster. This cluster is similar to the recommended cluster, but with one difference. It contains the minimum revision of any patch that addresses a Sun Alert. Applying this cluster will fix all Sun Alert issues, while introducing the least amount of change to your systems.

Once you have more that 50 servers, it can take months to completely patch them, depending on maintenance windows and uptime requirements. Sometimes it’s difficult to know if a particular server is patched yet or not, or when the last patching took place. One trick I learned a while back was to create a nearly-empty Solaris package, with only a text file, containing the patch level installed. This package is installed or updated with each patch cycle so you know exactly which set of patches is installed at any time.

Sun has come up with several patch management tools over the years, some better than others (PatchDiag, PatchCheck, PatchPro, smpatch). The most recent uses the “Patch Update Manager” in Solaris 10 or the xVM Ops Center product. If you’re looking for something low cost (free), then it might be worth looking at Patch Check Advanced. This tool analyzes your system, automatically downloads and installs patches fairly easily. It’s a great tool.

DISM provides dynamically resizable shared memory. Any process that uses a DISM segment can lock and unlock parts of a memory segment, and by doing so, the application can dynamically adjust to the addition (or removal) of physical memory from a server.

In the initial releases of Solaris 10, DISM was unavailable within Solaris Zones, because the ability for processes to lock memory segments was not available. If you do try to run Oracle with DISM in Zones before 11/06, you’ll see completely awful database performance (I’ve seen it). The fix for this was to disable DISM in Oracle, by setting the Oracle parameters sga_max_size and sga_target to the same value, so the SGA would not resize at all.

Solaris 10 update 3 (11/06) introduced a new zone privilege: proc_lock_memory, which gives processes within the zone the ability to lock memory segments. So DISM will now work if this privilege is enabled. To enable it, just turn it on in the zone config and reboot the zone:

# zonecfg -z oraclezone
zonecfg:oraclezone> set limitpriv=default,proc_lock_memory
zonecfg:oraclezone> commit
zonecfg:oraclezone> exit
# zoneadm -z oraclezone reboot

If you see an error after the “set limitpriv” line when you try, make sure you have Solaris 11/06 or later (or the patched equivilant).

I’d say the best strategy is to increase the size of /var. If you’re using the standard UFS file system with no volume management, this means backing up, re-creating the partition, and restoring the data. If you do have some sort of volume management, sometimes the answer is a simple metattach/growfs or vxresize command.

If you want another option, just to get your by until you have the time to increase /var, there is another easy method. When patchadd adds any patch to the system, the files being replaced get saved off in case you need to remove the patch later, restoring these files. These files are compressed and stored in /var/sadm/pkg/ /save/ and in /var/sadm/pkg/ /save/pspool/ /save/. The files are called undo.Z.

Note: It is completely safe to delete these .Z files, as long as you are sure you will never need to back out its associated patch! Doing this can free up significant space.

I’ve even done things like this in a pinch: (the shotgun approach)

#find /var -name undo.Z -exec rm {} \;

For some reason, configuring link-based IPMP in VCS is somewhat tricky, and the documentation doesn’t seem to help much. It seems all the default values for VCS are for probe-based IPMP only.

To achieve link-based IPMP, here’s how I’ve configured my MultiNICB resource:

Link-based IPMP MultiNICB properties

UseMpathd: 1
Tells VCS to use mpathd for network link status
MpathCommand: /usr/lib/inet/in.mpathd -a
The default, /usr/sbin/in.mpathd is just incorrect – it doesn’t live there.
ConfigCheck: 0
If you leave this at 1, it will overwrite your /etc/hostname.xxx files with probe-based IPMP configuration
Device: (your IPMP interfaces here)
The “interface alias” for each device is not needed, leave them blank.
IgnoreStatus: 0
You want VCS to NOT ignore link status, since this is how link-based IPMP works.
GroupName:
Do not use your IPMP group name here, it’s not needed. VCS is not monitoring the group, mpathd is.

Here’s how it looks in main.cf:

MultiNICB csgmultinic (
UseMpathd = 1
MpathdCommand = “/usr/lib/inet/in.mpathd -a”
ConfigCheck = 0
Device = { ce0 = “”, ce4 = “” }
IgnoreLinkStatus = 0
)

The psncollector service runs a ‘prtfru -x’ command, which can take several minutes to complete on a large server like an E2900. With the 120 second timeout, the start method fails:

# svcs -x
svc:/application/psncollector:default (Product Serial Number Collector)
State: maintenance since Sun 25 Jan 2009 10:01:34 AM PST
Reason: Start method failed repeatedly, last died on Killed (9).
See: http://sun.com/msg/SMF-8000-KS
See: /var/svc/log/application-psncollector:default.log
Impact: This service is not running.

# tail /var/svc/log/application-psncollector:default.log
[ Jan 25 08:59:51 Executing start method ("/lib/svc/method/svc-psncollector") ]
Using /var/run
[ Jan 25 09:02:01 Method or service exit timed out. Killing contract 48 ]
[ Jan 25 09:02:05 Method or service exit timed out. Killing contract 48 ]
[ Jan 25 09:02:18 Method "start" failed due to signal KILL ]

The easy fix was to increase the service start timeout value:

# svccfg -s psncollector setprop start/timeout_seconds=480
# svccfg -s psncollector setprop restart/timeout_seconds=480
# svcadm refresh psncollector
# svcadm clear psncollector

Once cleared, the service started up, taking its usual 3+ minutes.

Some of the misconceptions come from other UNIX and Linux OS’s, which measure the value differently. So an incorrect definition doesn’t necessarily demonstrate a lack of knowledge, but some amount ignorance to the way Solaris does it. Linux for example, also includes in its calculation the threads waiting for I/O, not just threads waiting for CPU.

In previous versions of Solaris (2.3-2.9), load average was a simple calculation. It was the average number of runnable and running threads. In other words, it was the number of threads running on the CPUs, plus the number of threads in the run queue, waiting for CPUs, averaged over time.

In Solaris 10, load average is calculated slightly differently than in previous versions.

The calculation is made by summing high-resolution user time, system time, and thread wait time, then processing this total to generate averages with exponential decay.

This calculation is slightly more comprehensive (and complex), because it takes into account CPU latency – the time taken to move a thread from the run queue onto a CPU. However, the older way of calculating this will yield almost identical results, so either definition I’d call “correct”. I still use the older definition because it is just easier to understand.

So what is a “high” number for load average? Well, first it depends on how many CPUs you have on your system, since the calculations do not take that into account. If you have one CPU, then a load average of 1.0 would mean you are, on average, consuming exactly 100% of that one CPU over the measurement period. If your number climbs above 1.0, then you have threads in the run queue at some point, waiting for CPU time. Solaris actually handles CPU saturation very well, so this may not mean your performance will degrade; it just means your CPU is well-used.

On the other hand, if you have 8 CPUs and a load average of 32, you may be seeing a performance degradation, as your system is somewhat CPU-bound. Each CPU is, on average, 100% utilized by running threads, and there are, on average, 24 more threads in the run queue. Depending on the application, this may be acceptable – it just depends on the expected response-time or expected processing time for your application.

# chmod -R 777 /usr
^C^C^C

Oops – You just changed some files in /usr to 777 before you were able to cancel. You don’t know how many or which files were affected. pkgchk can save you here.

In Solaris, there is a software “registry” file, /var/sadm/install/contents, which gives us information on every file installed on the system, or at least every file associated with a Solaris package. This file includes information about file permissions, owner and group information, file size and checksum. Here’s an excerpt from the contents file:

/etc/opt/SUNWexplo/t3files.txt f none 0444 root bin 123 11760 1208943439 SUNWexplu
/etc/opt/SUNWexplo/t3input.txt e build 0400 root bin 590 45160 1208943439 SUNWexplu
/etc/opt/SUNWexplo/tapeinput.txt e build 0400 root bin 885 6403 1208943439 SUNWexplu
/etc/opt/SUNWexplo/xscfinput.txt e build 0400 root bin 758 60717 1208943439 SUNWexplu
/etc/pam.conf e pamconf 0644 root sys 3103 17166 1219679093 SUNWcsr
/etc/passwd e passwd 0644 root sys 672 56039 1219679093 SUNWcsr
/etc/patch d none 0755 root sys SUNWppror
/etc/patch/patch.conf v preserve 0644 root sys 365 31670 1186005379 SUNWppror
/etc/patch/secret.conf v preserve 0600 root sys 207 17050 1186005379 SUNWppror
/etc/path_to_inst v preserve 0444 root root 26 2566 1106347450 SUNWcsd
/etc/power.conf e powerconf 0644 root sys 488 40965 1106350205 SUNWpmr
/etc/printers.conf e preserve 0644 root sys 162 13902 1106350198 SUNWpcr
/etc/profile e etcprofile 0644 root sys 712 51625 1219679093 SUNWcsr

The pkgchck command can be used to fix the file attributes (owner and group) of any or all package-installed files.

# pkgchk -f

This command would check every file listed in /var/sadm/install/contents, and if needed, change the owner and permissions of the files on the system to match the registry. Yes, this is sort of a shotgun approach, and you may not want to invoke changes this widely across the system.

In my example, only /usr was affected, so you can narrow down the criteria with a find command:

# find /usr -perm 777 -exec pkgchk -f -p {} \;

Or if you only changed one file, and now want it to change it back to whatever it was, just tell pkgchk to only work on the file you specify:

# pkgchk -f -p /etc/crypto/kcf.conf

pkgadd takes a -n argument, that tells it to operate in non-interactive mode. However, this alone will not let you install much of anything, because if the pkgadd command needs any input from the user, it will just exit and your package will not be installed. To give pkgadd the information to act on its own and install your package, you have to provide the -a option and specify an “installation administration file”.

This “admin” file contains all the parameters pkgadd will need to operate. The default file exists in /var/sadm/install/admin/default. Copy it to your home directory and take a look at it.

mail=
instance=unique
partial=ask
runlevel=ask
idepend=ask
rdepend=ask
space=ask
setuid=ask
conflict=ask
action=ask
networktimeout=60
networkretries=3
authentication=quit
keystore=/var/sadm/security
proxy=
basedir=default

You can get information on all of the parameters in the file with:

# man -s 4 admin

What I usually do, to forcefully install the packages without asking anything, is just replace all the occurences of “ask” to “nocheck”. This will take the default file, and create a new one, changing ask to nocheck.

# sed 's/ask/nocheck/' < /var/sadm/install/admin/default > /home/user/admin.file

Now you can do your pkginstall without any questions being asked:

# pkgadd -n -a admin.file SUNWblah

Another handy parameter in the admin file, especially when you are installing packages across multiple hosts, is the “mail” parameter. When you set this with your email, you will be notified when the package installs on each system.

When any type of network connection is made to your servers, it’s important to know the source of the connection – where that connection originated. Yes, many hackers will use a proxy or bounce host or hosts to hide their true IP, but at least this information can give you a place to start if you needed to track them. This becomes even more useful in company-internal incidents where users are less able to hide.

TCP Wrappers has been around for ages. It’s a mechanism to allow or deny access to any inetd service, based upon the connecting IP address or host name. It used to be a more difficult to use – one had to download source, compile, install, configure, etc. But these days it’s built into many inetd variants, including Solaris.

Just for connection logging, we don’t necessarily need to set up TCP Wrappers to deny/allow hosts to connect based on IP or host name, so we’ll skip that part. If you want to go the extra mile and set this up, you configure the hosts.allow and hosts.deny files in /etc. Google around, it’s easy to find a howto.

With the SMF-based inetd in Solaris 10, it’s easy to turn on TCP wrappers for just one service or all services at once. If you just wanted to enable wrappers/logging for the FTP service, you’d change the properties of the FTP inetd service with inetadm:

# inetadm -m ftp tcp_wrappers=true

Or, to change the default value for ALL inetd services, you’d use the -M option:

# inetadm -M tcp_wrappers=true

When this change is made, a log entry will be made, usually in /var/log/syslog, unless you’ve changed your syslog configuration:

Jan 13 08:56:52 waters vnetd[26111]: [ID 927837 daemon.info] connect from rocky
Jan 13 09:00:26 waters in.rshd[28426]: [ID 927837 daemon.info] connect from hungryhippo
Jan 13 09:17:25 waters in.rshd[8174]: [ID 927837 daemon.info] connect from penta
Jan 13 09:24:19 waters in.telnetd[12414]: [ID 927837 daemon.info] connect from 192.168.151.95
Jan 13 09:35:17 waters in.ftpd[23954]: [ID 927837 daemon.info] connect from mercury

In Solaris 8, you’d accomplish this same goal by altering your inetd start script, /etc/init.d/inetsvc:
just add the -t option to the last line:

/usr/sbin/inetd -s -t &