AIX > Tips & Techniques > Systems Management

DSH Simplifies the Task of Managing Multiple Systems

DSH enables users to execute commands across multiple AIX systems


If you've ever wanted to run a command across several AIX systems, or to check those systems to see if a service is active, look no further. The Distributed Shell (dsh) command can be used to execute commands on multiple systems in parallel. For example, if you need to check the date and time in your environment, you can use dsh to run the date command on all your AIX systems in one hit.

 

This handy tool is included in the base AIX operating system. I'll offer a few pointers on using it.

Yes, by writing your own scripts and interesting "for loops," you can accomplish the things that dsh does, but dsh lets you avoid the roll-your-own process and use a standard method in any AIX environment.

Installation

DSH is installed with AIX as part of the Cluster Systems Management (CSM) client file sets. These file sets are necessary to support dynamic LPAR (DLPAR) operations in a partitioned AIX environment. The actual dsh tool is installed as a symbolic link from /usr/bin/dsh to /opt/csm/bin/dsh, via the csm.dsh fileset (see Figure 1).

Put some thought into where you run dsh—e.g., from a central server you can use as the admin host from which you'll distribute all your commands. Given that most AIX sites use a NIM server as a central place from which to administer their systems (i.e. for installing and updating AIX), I feel it is the most logical place from which to run dsh commands.

To use dsh you must configure your user environment appropriately. This involves two steps. First, configure your SSH keys between the admin host and all of your remote hosts. While either rsh or ssh can be used to execute commands on remote nodes, from a security perspective, I recommend using ssh only. Many online tutorials explain the different ssh configuration options that allow you to execute commands on another host without being prompted for a password. See the References sidebar for these links.

Once you can execute a command on all your hosts without being prompted for a password (see below), you're ready to configure dsh.

$ ssh aix01 date
Tue Sep 16 16:44:49 EET 2008

Configuration

DSH requires that these environment variables are set within your shell:

$ env | egrep 'DSH|WCOLL'
DSH_NODE_RSH=/usr/bin/ssh
WCOLL=/home/gibsonc/wcolls/aix

Place these entries in your .kshrc file:

export DSH_NODE_RSH=/usr/bin/ssh
export WCOLL=/home/gibsonc/wcolls/aix
export DSH_NODE_OPTS="-q"

The DSH_NODE_RSH variable indicates which remote command executable to use when dsh is called. Set this to /usr/bin/ssh so that dsh will use the ssh command when connecting to remote hosts.

The next variable is WCOLL. Known as the working collective, it points to a file that contains a list of all the hosts that you want to execute a command on:

$ cat $WCOLL
aix01
aix02
aix03
aix04
aix05
aix06
aix07
aix08

Though it's long been replaced by the DSH_NODE_LIST variable, I still use WCOLL. It takes me back to the time of the RS/6000 SP, when commands like dsh were part of IBM's PSSP (Parallel System Support Programs) package. However, either will work.

The final variable, DSH_NODE_OPTS, is optional. It specifies which options are passed to the remote command. I set this to "-q" so ssh runs as "ssh -q". I do this to suppress the ssh banner message. (It's another personal preference: I'd rather not see a banner displayed from every host I run a command on. With a large number of hosts, I find it impossible to see the command output amongst the banner messages!)

I often set up several node list files for different sets or groups of AIX systems. For example, I always have a global file, named aix, that contains all of my AIX hosts. However, if I have a HACMP cluster somewhere, then I set up a separate file containing only the nodes in that cluster.

Another step I always take is to set up aliases that change where my WCOLL variable points. I include these aliases in my .kshrc file. See Figure 2 for an example.

So, by default, my WCOLL points to my aix hosts file. I have several aliases, all pointing to different files, that contain specific groups of AIX hosts. For example, the nim file/alias contains a list of all of my NIM servers. If I want to run commands on one of my HACMP clusters, I simply enter the appropriate alias name for that cluster. For example:

$ hac5
/home/gibsonc/wcolls/hac5

$ echo $WCOLL
/home/gibsonc/wcolls/hac5

$ cat ~wcolls/hac5
aix01
aix02
aix03

$ dsh uptime
aix01: Wed Sep 17 16:11:29 EET 2008
aix02: Wed Sep 17 16:11:26 EET 2008
aix03: Wed Sep 17 16:11:25 EET 2008

Using dsh and Related Tools

From time to time you may need to execute commands on only some of your host systems. The "-w" flag allows you to list the specific nodes on which to execute a command:

$ dsh -waix07,aix08 date
aix07: Tue Sep 16 16:40:31 EET 2008
aix08: Tue Sep 16 16:40:30 EET 2008

Other tools are included with dsh. Some you may find useful include dcp, dping and dshbak. The dcp command copies a file (or files) to multiple nodes in parallel. You will need to include the following variables in your .kshrc file first, so that dcp will use scp as the file transfer method.

export DCP_NODE_RCP=/usr/bin/scp
export DCP_NODE_OPTS="-q"

So to copy a file to all of my hosts, I’d perform the following (see Figure 3).

The dping utility allows you to ping several hosts at once. It's useful for performing a quick "up or down" health check on several systems:

gibsonc@hxnim2 /home/gibsonc $ dping -n aix01,aix02,aix03
aix01: ping (alive)
aix02: ping (alive)
aix03: ping (alive)

The dshbak utility formats dsh command output, showing the hostname and output from the command underneath:

$ dsh -waix07,aix08 date | dshbak
HOST: aix07
----------------
Tue Sep 16 16:40:43 EET 2008

HOST: aix08
----------------
Tue Sep 16 16:40:43 EET 2008

I also find dsh useful for running commands on my virtual I/O servers (VIOS). Again, this requires ssh keys be configured between your admin host and the padmin account on the VIOS (see Figure 4).

I often need to check if a particular subsystem or service is active on my AIX hosts. DSH makes this easy, as I don't need to login to each host to query the status of a process. In the following example I execute the lssrc command on several hosts to determine if the NTP daemon is active.

$ dsh lssrc -s xntpd | dshbak
HOST: aix01
----------------
Subsystem         Group    &nbs p;       PID        & nbsp; Status
 xntpd            tcpip &n bsp;          294994    &n bsp;  active

HOST: aix02
----------------
Subsystem         Group    &nbs p;       PID        & nbsp; Status
 xntpd            tcpip &n bsp;          516120    &n bsp;  active

HOST: aix03
----------------
Subsystem         Group    &nbs p;       PID        & nbsp; Status
 xntpd            tcpip &n bsp;          274454    &n bsp;  active

To view the current settings for your dsh environment, execute dsh with the "-q" flag:

$ dsh -q
DSH:DCP_DEVICE_OPTS=
DSH:DCP_DEVICE_RCP=
DSH:DCP_NODE_OPTS=-q
DSH:DCP_NODE_RCP=/usr/bin/scp
DSH:DSH_CONTEXT=
DSH:DSH_DEVICE_LIST=
DSH:DSH_DEVICE_OPTS=
DSH:DSH_DEVICE_RCP=
DSH:DSH_DEVICE_RSH=
DSH:DSH_ENVIRONMENT=
DSH:DSH_FANOUT=
DSH:DSH_LOG=
DSH:DSH_NODEGROUP_PATH=
DSH:DSH_NODE_LIST=/home/gibsonc/wcolls/hac5
DSH:DSH_NODE_OPTS=-q
DSH:DSH_NODE_RCP=
DSH:DSH_NODE_RSH=/usr/bin/ssh
DSH:DSH_OUTPUT=
DSH:DSH_PATH=
DSH:DSH_REPORT=
DSH:DSH_SYNTAX=
DSH:DSH_TIMEOUT=
DSH:RSYNC_RSH=

A Useful Tool

DSH is not new but its benefits are often overlooked. I hope you’ve found this brief insight into dsh a useful one. Maybe it will help you do your job more efficiently—or even impress your fellow AIX admin colleagues!

References

 

 

Chris Gibson is an AIX and PowerVM specialist located in Melbourne, Australia. He is an IBM Champion for Power Systems, IBM CATE (Power Systems and AIX), and a co-author of several IBM Redbooks publications.


comments powered by Disqus
Buyers Guide

Advertisement

IBM Systems Magazine Subscribe Box Read Now Link Subscribe Now Link iPad App

Advertisement