Wmii Python script to monitor remote machines

I like to monitor our web servers by ssh'ing into the remote machine and watching "top", tailing log files, etc. Normally, I open a terminal, ssh into the remote machine, run the monitoring command (e.g. "top"), then repeat for the rest of the remote machines. Then I adjust the window sizes so I can see everything at once.

My window manager, wmii, is great for tiling a bunch of windows at once. It is also scriptable with Python, so I wrote a Python script to create my web server monitoring view. Below is my script. I also put a video on YouTube.

#!/usr/bin/env python

import os
import time

NGINX_MONITOR_CMD = "tail --follow=name /var/log/nginx/cache.log | grep --color -E '(HIT|MISS|EXPIRED|STALE|UPDATING|\*\*\*)'"
APACHE_MONITOR_CMD = "top"
MYSQL_MONITOR_CMD = "mysqladmin extended -i10 -r | grep -i 'questions\|aborted_clients\|opened_tables\|slow_queries\|threads_created' "

CMDS_COL1 = ['urxvt -title "Nginx 1" -e ssh -t us-ng1 "%s" &' % NGINX_MONITOR_CMD,
             'urxvt -title "Nginx 2" -e ssh -t us-ng2 "%s" &' % NGINX_MONITOR_CMD,
             ]
CMDS_COL2 = ['urxvt -title "Apache 1" -e ssh -t us-med1 "%s" &' % APACHE_MONITOR_CMD,
             'urxvt -title "Apache 2" -e ssh -t us-med2 "%s" &' % APACHE_MONITOR_CMD,
             'urxvt -title "Apache 3" -e ssh -t us-med3 "%s" &' % APACHE_MONITOR_CMD,
             ]
CMDS_COL3 = ['urxvt -title "MySQL 1" -e ssh -t us-my1 "%s" &' % MYSQL_MONITOR_CMD,
             'urxvt -title "MySQL 2" -e ssh -t us-my2 "%s" &' % MYSQL_MONITOR_CMD,
             ]
COLUMNS = [CMDS_COL1, CMDS_COL2, CMDS_COL3]

def create_windows():
    for i, col in enumerate(COLUMNS):
        cindex = str(i+1)
        for cmd in col:
            os.system(cmd)
            time.sleep(1)
            os.system('wmiir xwrite /tag/sel/ctl send sel %s' % cindex)
        os.system('wmiir xwrite /tag/sel/ctl colmode %s default-max' % cindex)
    os.system('wmii.py 45.5 31.5 23')

if __name__ == '__main__':
    create_windows()

Note 1: The script above uses another script I wrote previously, wmii.py, to set the column widths.

Note 2: The remote server addresses are specified by the nicknames us-ng1, us-ng2, us-med1, etc. configured in my ~/.ssh/config file as described here.

Note 3 (on using ssh and top): I first tried doing ssh host top, but this gave me a TERM environment variable not set. error. I then tried ssh host "export TERM=rxvt-unicode; top", but this gave me a top: failed tty get error. The solution that worked for me was to use the -t option with ssh. E.g. ssh -t host top. This is what I used in the script above.

Note 4 (added 2010-03-05): I used "tail --follow=name" instead of "tail -f" so that tail will follow the log file even after it has been rotated. For more information, see the man page for tail.

Note 5 (added 2010-03-05): To prevent your ssh session from timing out, add the following 2 lines to your ~/.ssh/config file (via):

Host *
  ServerAliveInterval 60