Blog <-

Archive for the ‘sysadmin’ Category

RSS   RSS feed for this category

MobaXterm – (Free) All-in-one Xserver/SSH/Linux environment for Windows

I recently stumbled on MobaXterm. It's a complete unix enviroment including X Server/SSH/Telnet/SCP/FTP client all in one. The list of features is impressive to say the least. This is an excellent replacement for Putty.

A small selection of the most useful features:

  • Free. What more is there to say?
  • Tabs and Horizontal / Vertical split panels finally bring the full power of native Unix/Linux terminal emulators to Windows
  • Integrated X server. MobaXterm comes with an integrated X Server. Everything is set up correctly out-of-the-box. X11 forwarding means you can simply SSH to a remote machine and start X11 programs. It supports displaying remote X11 windows as native windows or you can run the X Server ina separate tab/window.
  • Session Management makes it easy to quickly connect to the machine you want.
  • Integrated SFTP when SSHing to a remote machine means you don't have to start a separate SFTP/SCP session. Just browse, upload and download remote files from the left side of the SSH session.
  • Many supported services, such as SSH, Telnet, local Linux/Cygwin terminal, local Windows command prompt, RSH, XDMCP, RDP, VNC, FTP, etc.
  • Session multiplexing provides a quick method of running commands on multiple machines at the same time.
  • SSH bouncing through a gateway SSH server means no more SSHing from machine to machine.
  • Cygwin environment so you can actually get some work done natively on Windows. Batteries, bells, whistles and kitchen sinks (as well as games) included: full unix environment with tools like grep, find, vim, etc, etc, etc.

There are countless more features. This is the terminal emulator app I always hoped Putty would become. Of all the different shells around Putty, separate SSH connection managers and terminals I've tried, this is by far the best one.

Setting I/O priorities on Linux

All us system admins know about nice, which lets you set the CPU priorities of a process. Every now and then I need to run an I/O-heavy process. This inevitably makes the system intermittently unresponsive or slow, which can be a real annoyance.

If your system is slow to respond, you can check to see if I/O is the problem (which it usually will be) using a program called iotop, which is similar to the normal top program except it doesn't show CPU/Memory but disk reads/writes. You may need to install it first:

# aptitude install iotop

The output looks like this:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                       
12404 be/4 fboender  124.52 K/s  124.52 K/s  0.00 % 99.99 % cp /home/fboender
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]

As you can see, the copy process with PID 12404 is taking up 99.99% of my I/O, leaving little for the rest of the system.

In recent Linux kernels (2.6.13 with the CFQ io scheduler), there's an option to renice the I/O of a process. The ionice tool allows you to renice the processes from userland. It comes pre-installed on Debian/Ubuntu machines in the util-linux package. To use it, you must specify a priority scheduling class using the -c option.

  • -c0 is an old deprecated value of "None", which is now the same as Best-Effort (-c2)
  • -c1 is Real Time priority, which will give the process the highest I/O priority
  • -c2X is Best-Effort priority puts the process in a round-robin queue where it will get a slice of I/O every so often. How much it gets can be specified using the -n option which takes a value from 0 to 7
  • -c3 is Idle, which means the process will only get I/O when no other process requires it.

For example, I want a certain process (PID 12404) to only use I/O when no other process requires it, because the task is I/O-heavy, but is not high priority:

# ionice -c3 -p 12404

The effects are noticeable immediately. My system responds faster, there is less jitter on the desktop and the commandline.


Persistent undo history in Vim

Once you quit Vim, the undo history for that file is gone. This sometimes gives me problems if I accidentally made a change in a file without knowing it. This usually happens due to a bad Vim command which, for instance, capitalized a single letter.

There's an option which allows you to make the undo history persistent between Vim sessions. That means you can still undo changes you made in a file, even if you've quit Vim in the meantime.

You can add the following to your .vimrc to enable it:

set undofile   # Maintain undo history between sessions

This will create undo files all over the place, which look like this:

-rw-r--r-- 1 fboender fboender 320 2012-07-26 10:23 bad_gateway.txt
-rw-r--r-- 1 fboender fboender 523 2012-07-24 14:51 .bad_gateway.txt.un~

You can remedy this by including the following option in your configuration:

set undodir=~/.vim/undodir

Make sure to create the undodir:

$ mkdir ~/.vim/undodir

The undo files will now be saved in the undodir:

$ ls -la .vim/undodir/
total 12
drwxr-xr-x  2 fboender fboender 4096 2012-07-26 10:32 .
drwxr-xr-x 12 fboender fboender 4096 2012-07-26 10:24 ..
-rw-r--r--  1 fboender fboender  519 2012-07-26 10:32 %home%fboender%bad_gateway.txt

Conque: Terminal emulators in Vim buffers

For the longest time, I've searched for a way to run terminal emulators in Vim buffers.

As a kind of work-around, I created Bexec, which allows you to run the current contents of a buffer through an external program. It then captures the output and inserts/appends it to another buffer.

Although Bexec works reasonable, and still has it's uses, it's not a true terminal emulator in Vim. Today I finally found a Vim plugin that let's you actually run interactive commands / terminals in Vim buffers: Conque.

It requires Vim with Python support built in. Installation is straight-forward if you've got the requirements.

Download the .vmb file, edit it in vim, and issue:

:so %

It will then be installed. Quit vim, restart it, and you can now run start using it:

:ConqueTerm bash

Very awesome.

Re-use existing SSH agent (cygwin et al)

(Please note that this post is not specific to Windows nor Cygwin; it'll work on a remote unix machine just as well)

On my netbook, I use Windows XP in combination with Cygwin (A unix environment for Windows) and Mintty for my Unixy needs. From there, I usually SSH to some unix-like machine somewhere, so I can do systems administration or development.

Unfortunately, the default use of an SSH agent under Cygwin is difficult, since there's no parent process that can run it and put the required information (SSH_AUTH_SOCK) in the environment. On most Linux distribution, the SSH agent is started after you log in to an X11 session, so that every child process (terminals you open, etc) inherits the SSH_AUTH_SOCK environment setting and SSH can contact the ssh-agent to get your keys. Result? You have to start a new SSH agent, load your key and enter your password for each Mintty terminal you open. Quite annoying.

The upside is, it's not very hard to configure your system properly so that you need only one SSH agent running on your system, and thus only have to enter your password once.

The key lies in how ssh-agent creates the environment. When we start ssh-agent in the traditional manner, we do:

$ eval `ssh-agent`
Agent pid 1784

The command starts the SSH agent and sets a bunch of environment variables:

$ set | grep SSH_

The SSH_AUTH_SOCK is how the ssh command knows how to contact the agent. As you can see, the socket filename is generated randomly. That means you can't reuse the socket, since you can't guess the socket filename.

Good thing ssh-agent allows us to specify the socket filename, so we can easily re-use it.

Put the following in your ~/.bashrc:

# If no SSH agent is already running, start one now. Re-use sockets so we never
# have to start more than one session.

export SSH_AUTH_SOCK=/home/fboender/.ssh-socket

ssh-add -l >/dev/null 2>&1
if [ $? = 2 ]; then
   # No ssh-agent running
   rm -rf $SSH_AUTH_SOCK
   # >| allows output redirection to over-write files if no clobber is set
   ssh-agent -a $SSH_AUTH_SOCK >| /tmp/.ssh-script
   source /tmp/.ssh-script
   echo $SSH_AGENT_PID >| ~/.ssh-agent-pid
   rm /tmp/.ssh-script

What the script above does is, it sets the socket filename manually to /home/yourusername/.ssh-socket. It then runs ssh-add, which will attempt to connect to the ssh-agent through the socket. If it fails, it means no ssh-agent is running, so we do some cleanup and start one.

Now, all you have to do is start a single terminal, and load your keys once:

$ ssh-add ~/.ssh/fboender\@electricmonk.rsa
Enter passphrase for .ssh/fboender@electricmonk.rsa: [PASSWORD]
Identity added: .ssh/fboender@electricmonk.rsa (.ssh/fboender@electricmonk.rsa)

Now you can start as many new terminals as you'd like, and they'll all use the same ssh-agent, never requiring you to enter your password for that key more than once per boot.


I've updated the script with suggestions from Anthony Geoghegan. It now also works if noclobber is set.

monit alerts not emailed: Resource temporarily unavailable

While setting up Monit, a tool for easy monitoring of hosts and services, I ran into a problem. I had configured Monit to email alerts to my email address, using my personal mail server (IP/email addresses obfuscated to protect the innocence of my email inbox):

set mailserver
set alert
set httpd port 2812

check file test path /tmp/monittest
  if failed permission 644 then alert

After starting it with monit -c ./monitrc, I could reach the webserver at port 2812. I also saw that the check was failing:

# monit -c ./monitrc status
File 'test'
  status                            Permission failed
  monitoring status                 monitored
  permission                        600

However, it was not sending me emails with startup, and status error reports. My server's mail log showed no incoming mail, making it seem like Monit wasn't even trying to send email. Turning on Monit's logging feature, I noticed:

# monit -c ./monitrc quit
# monit -c ./monitrc -l ./monit.log
# tail monit.log
error : 'test' permission test failed for /tmp/monittest -- current permission is 0600
error : Sendmail: error receiving data from the mailserver '' -- Resource temporarily unavailable

I tried a manual connection to the mail server from the host where Monit was running, and it worked just fine.

The problem turned out to be a connection timeout to the mail server. Most mail servers nowadays wait a certain number of seconds before accepting connections. This reduces the rate at which spam can be delivered. Monit wasn't waiting long enough before determining that the mail server wasn't working, and bailed out of reporting errors with a 'Resource temporarily unavailable'.

The solution is easy. The set mailserver configuration allows you to specify a timeout:

set mailserver with timeout 30 seconds

I'm happy to report that Monit is now sending email alerts just fine.

The user isn't always wrong

Some time ago, my mother bought a new laptop. It came preinstalled with Windows Vista, which proved to be quite the disaster. The laptop wasn't nowhere near fast enough to run it, so I installed Ubuntu on it. This allowed my mom to do everything she needed to do with the laptop, while at the same time making it easy for me to administer the beast.

One day my mom phoned me, and explained about a problem she was having:

"Whenever I move the laptop into the kitchen, it stops working!"

Now my mom is no computer expert, but she picked up Ubuntu quickly and has never needed much hand-holding when it comes to using the laptop. This one, however, sounded to me like one of those situations where the user couldn't possibly be correct. We went through the basic telephone support routine, but she persisted in her observation that somehow the kitchen was responsible for her laptop misery.

Eventually, after deciding the problem couldn't be fixed over the phone, I agreed to come over to my parents house the next evening to take a look at it. With my general moody "a family member's PC needs fixing" attitude and a healthy dose of skepticism ("this is going to be one of those typical the-cable-isn't-plugged-in problems"), I arrived at my parents.

"Okay, let's see if we can't fix this problem", I said, as I powered up the laptop upstairs. Everything worked fine. Picking up the laptop, I moved it downstairs into the living room. No problems whatsoever. Next, the kitchen. And lo and behold:

The laptop crashed almost immediately.

"Coincidence", I thought, and tried it again. And again, as soon as I entered the kitchen, the laptop crashed. I… was… Stunned! I had never encountered a problem like this before. What could possibly make it behave like that?

After pondering this strange problem for a while, I thought "what's the only location-dependent thing in a laptop?", and it dawned on me that it might just be related to the WiFi. I powered up the laptop once again in the living room, completely turned off the WiFi by rmmod-ing the relevant kernel modules, and entered the kitchen. No crash. It kept on working perfectly. Until I turned on the WiFi again.

With the aid of some log files (which I should have checked in the first place, I admit), I quickly found the culprit. The very last thing I saw in the log files just before the computer crashed… an attempt to discover the neighbors WiFi! A wonky WiFi router in combination with buggy drivers cause the laptop to crash, but only when it came in range of said WiFi router. And that happened only in the kitchen!

In the end I disabled automatic WiFi discovery on the laptop, since my mom didn't really take it out of the house anyway, and the problems disappeared. I never encountered a problem like that again, but I did learn one thing though:

No matter how impossible the problem may seem… The user isn't always wrong.

Read the POSIX standard

Stop reading your local manual pages when programming/scripting stuff, and use the POSIX standard instead:

Online POSIX 2008 (EEE Std 1003.1-2008) standard

There are four main parts:

Some Do's and Dont's:

Finally, read Bash Pitfalls to learn why your shell scripting sucks.

VirtualBox: List guest IPs

I often clone VirtualBox machines as a quick way to get a fresh box to test some stuff on. The problem is, I don't know which IP the new clone gets assigned. Fortunately, if you've got VirtualBox Guest Additions installed on the guest, you can use the guestproperty to get the IP.

Here's a quick shell script for listing the v4 IPs of all the running guest virtual machines.


VBoxManage list runningvms | cut -d "{" -f1 | sed "s/\"//g" | while read VBOXNAME; do
    IP=$(VBoxManage guestproperty get "$VBOXNAME" /VirtualBox/GuestInfo/Net/0/V4/IP | cut -d":" -f2)
    echo "$VBOXNAME: $IP"

P.S. If you're using Debian guests: delete the /etc/udev/rules.d/*persistent-net* files. See here for why.

Tmux scrolling with shift-pageup/down

Put this in your .tmux.conf to enable scrolling using the Shift-PageUp/Down keys:

set -g terminal-overrides 'xterm*:smcup@:rmcup@'