Tuning a Perfect Note
A look at performance tuning and new AIX 5.3 commands.
As more mission-critical work finds its way to distributed systems, performance management for those systems is becoming more important. The goal for systems management is not only to maximize system throughput, but also to reduce response time. To do this, it's necessary to not only monitor and tune the system resources, but also to work on profiling and tuning applications. In previous articles, we looked at specific areas of performance tuning, primarily those related to I/O tuning. In this article, I'll look at tuning in general and some of the new commands in AIX* 5.3, and I'll provide a starter set of tunables for the reader to test.
In AIX, several major resource types must be monitored and tuned, including CPU, memory, I/O time, network time, and perhaps some serial or modem time and application programs. From the user's perspective, the only time they see is total execution time, so we'll start by looking at that.
Total execution time, from a user's perspective, consists of wall-clock time. The components of total running time include user-state CPU, system- or kernel-state CPU, I/O time and network time, and time when the processor is running a different application and your application is waiting (run Q time).
When examining performance problems, or even just taking a baseline measurement, it's important to take a phased approach to reviewing the problem. I normally run a script to gather all of the data I need (see "Helpful Links" sidebar) and I also run nmon in logging mode so I have some additional data. I then review the data. First, I look at processor-related issues, then I/O, then paging, and finally I check to see if anything looks untoward. I review all of the tunables settings and figure out what needs changing. I also check all firmware levels on the servers and cards, and I check the maintenance level on the software. Finally, if I still can't see the problem, I resort to running traces.
With the advent of AIX 5.3, not only were new commands added to the performance tools suite, but some of the default recommendations changed. Previously, the list of commands included the following: topas, sar, vmstat, iostat, ps, svmon, netstat, entstat, nfsstat, filemon, fileplace, netpmon, lspv, lsvg, lsattr, lsdev, lslv and lsps. In later maintenance levels of AIX 5.2 and/or AIX 5.3 these commands were supplemented with some new commands and were also made fractional LPAR aware.
There are a couple of applications that you can download from the IBM* Web sites that are extremely useful (again, see the "Helpful Links" sidebar). On Alphaworks, you can get the Graphical LPAR Monitor for System p5* platforms. This allows you to graphically monitor resources being used by one or more LPARs at the same time. Additionally, you can download nmon from the IBM Wiki and use that to monitor the system in real-time or log performance over a period of time. You can then download the nmon analyzer, which runs on Windows* and uses Microsoft* Excel, to crunch your nmon data.
With the advent of AIX 5.3, not only were new commands added to the performance tools suite, but some of the default recommendations changed.