The Complete Guide to System Maintenance
Taking care of your servers is crucial. It’s important to stay as current as possible to ensure you have the best security and performance. Additionally, the further back you are the more likely that the response will be to upgrade when you call in with a problem. The key items to upgrade include LPARs, VIO servers, HMCs and server and I/O firmware.
First, planning is critical. Many find it challenging to get a maintenance window at all so the plan is often to cram as much into it as possible. But that’s almost a guarantee that something will get broken and, worst of all, it’ll be difficult to figure out the cause. It’s important to separate times for network, storage and system updates. For example, if you’re using LPM (Live partition mobility) to move LPARs to prepare for maintenance and someone starts performing maintenance on the network, then you’ll have a problem. It happens all the time. The same applies to active directory server maintenance, if that’s how you authenticate for your LPARs, and to things like citrix servers, if that’s how you access them from outside. I’ve lost count of the times I’ve had an upgrade go south because someone rebooted the citrix server I was using. So planning should include the timing of the upgrades and those of anything you’re depending on for your upgrade to work.
Before any update, you must download the files and read all of the readme files. This ensures that you install any patches in the right order. Typically, you upgrade the HMC,then the firmware and then the operating system, however the readme will make clear what the prerequisites are.
IBM also requires that you have a valid hardware maintenance agreement (HWMA) to download and install firmware and a valid software maintenance agreement (SWMA) to download and upgrade software. You can check entitlements on the IBM entitled software site under “my entitled software”. You’ll need the server model and serial number, i.e. 8205-e6d serial 06abcd1. For some products you’ll need all seven characters.
Determine the Levels
To figure out what levels you should install, determine current installed levels by doing the following as root:
The above will show you the oslevel as something like:
The above is AIX 7.1 tl01 sp4
For a VIO you run the above after entering oem_setup_env but you also run the following as padmin
For a VIO, oslevel will normally show something like: 6100-09-05-1524 with ioslevel showing something like 188.8.131.52 You can check the VIOS to NIM mapping table to find out how these should match
The above will provide you with the firmware level for the server and also for any I/O that is attached to it i.e. fibre cards, network cards, disks, etc. These adapters will need updating regularly as IBM brings out mandatory updates from time to time. This applies especially to the 10Gb network and 8/16Gb fibre cards. For example the 5735 2 port 8Gb fibre card has a mandatory update as of January 2016 – the level needs to be at 203305.
Now that you have your starting points, you can use FLRT (fix level recommendation tool) or FLRT Lite to figure out what updates need to be applied, download the operating system ones and then look at the readme documents. In the readme you will either find a change history with a list of all the changes incorporated in these fixes or a link to the firmware history which lists the changes in every firmware release from your installed one to the one you’re going to. Once you have the list of potential levels then you must determine what combination to install. Typically, most people want to wait two to three months after a level is released before installing it unless it’s mandatory.
Step 1: always run errpt if this is an operating system or to check error logs prior to starting any update. No point in trying to update a system that has problems. Step 2: take a backup–for AIX or the VIO this would be a mksysb to an external resource, for the HMC this would involve running a save upgrade data to the hard drive and then backing up the management console data. This second backup can be done to the USB key if you have one or to a remote FTP server. Use of the DVD is no longer supported. Once the backup is complete you are ready to go.
Prior to starting any firmware updates you may want to use LPM to move any affected LPARs to a new server, especially if you are going to have to power cycle the server for a deferred or disruptive update.
HMC (Hardware Management Console) Updates
I try to keep my HMC at the absolute latest level that will support my servers and that’s supported on the HMC. Right now the latest version of the HMC software is v184.108.40.206 (MH01616) with ifix MH01649 applied. This level requires that the HMC must be at least a CR8 deskside or a CR5 rack mount. Earlier HMCs are not supported on v8. Additionally, POWER5 servers are not supported on v8. There are also prerequisites prior to upgrading to v8.8.5 that are further explained in the readme. For older HMCs or for POWER5 server support the most current version is v220.127.116.11M3 (MH01546) with iFix MH01635.
There are three kinds of updates for the HMC and they are handled a little differently. Two of them (updates and fixes) are handled as general updates where the HMC is updated via the GUI. You need to know if the update is an update or a fix⎯this is detailed in the readme and determines the location you point to in order to obtain the update. When I select Update HMC on the HMC I use the FTP option, point to ftp.software.ibm.com and login as anonymous. If this is an update I point to /software/server/hmc/updates. If it’s a fix, the directory is usually /software/server/hmc/fixes. The HMC prompts you with a list of patches with names like MH01567.iso and you pick the correct one.
The other option is a more major upgrade. Typically I do these by using SSH to the HMC. If I was going to upgrade to v18.104.22.168 I would open two SSH sessions to the HMC. On one I would run the following script to monitor the upgrade:
while true ; do
ls -la /hmcdump
On the other I would be downloading the software. Once it’s downloaded you won’t see the files in /hmcdump which is why the monitor script above is helpful in knowing when it has ended. On the actual download session, you would do the following:
chhmcfs -o f -d 0
The above clears out all temp files
getupgfiles -h public.dhe.ibm.com -u anonymous --passwd anonymous -d /software/server/hmc/network/v8830
The above downloads the actual upgrade files. Once that’s done you can exit the first session and type in the following two commands to tell the HMC to perform the upgrade:
chhmc -c altdiskboot -s enable --mode upgrade
The above tells it to set up to upgrade on boot
hmcshutdown -r -t now
It takes about 20 to 30 minutes to actually perform the upgrade. Once it’s done you must install any required upgrades or fixes. Each upgrade or fix will take at least 20 minutes as a reboot of the HMC is required for each. If you have redundant HMCs you should do them both the same day as problems can occur when they are out of sync.
Like what you just read? To receive technical tips and articles directly in your inbox twice per month, sign up for the EXTRA e-newsletter here.
comments powered by