Blog

Some time ago, my mother bought a new laptop. It came preinstalled with Windows Vista, which proved to be quite the disaster. The laptop wasn't nowhere near fast enough to run it, so I installed Ubuntu on it. This allowed my mom to do everything she needed to do with the laptop, while at the same time making it easy for me to administer the beast.

One day my mom phoned me, and explained about a problem she was having:

"Whenever I move the laptop into the kitchen, it stops working!"

Now my mom is no computer expert, but she picked up Ubuntu quickly and has never needed much hand-holding when it comes to using the laptop. This one, however, sounded to me like one of those situations where the user couldn't possibly be correct. We went through the basic telephone support routine, but she persisted in her observation that somehow the kitchen was responsible for her laptop misery.

Eventually, after deciding the problem couldn't be fixed over the phone, I agreed to come over to my parents house the next evening to take a look at it. With my general moody "a family member's PC needs fixing" attitude and a healthy dose of skepticism ("this is going to be one of those typical the-cable-isn't-plugged-in problems"), I arrived at my parents.

"Okay, let's see if we can't fix this problem", I said, as I powered up the laptop upstairs. Everything worked fine. Picking up the laptop, I moved it downstairs into the living room. No problems whatsoever. Next, the kitchen. And lo and behold:

The laptop crashed almost immediately.

"Coincidence", I thought, and tried it again. And again, as soon as I entered the kitchen, the laptop crashed. I… was… Stunned! I had never encountered a problem like this before. What could possibly make it behave like that?

After pondering this strange problem for a while, I thought "what's the only location-dependent thing in a laptop?", and it dawned on me that it might just be related to the WiFi. I powered up the laptop once again in the living room, completely turned off the WiFi by rmmod-ing the relevant kernel modules, and entered the kitchen. No crash. It kept on working perfectly. Until I turned on the WiFi again.

With the aid of some log files (which I should have checked in the first place, I admit), I quickly found the culprit. The very last thing I saw in the log files just before the computer crashed… an attempt to discover the neighbors WiFi! A wonky WiFi router in combination with buggy drivers cause the laptop to crash, but only when it came in range of said WiFi router. And that happened only in the kitchen!

In the end I disabled automatic WiFi discovery on the laptop, since my mom didn't really take it out of the house anyway, and the problems disappeared. I never encountered a problem like that again, but I did learn one thing though:

No matter how impossible the problem may seem… The user isn't always wrong.

Stop reading your local manual pages when programming/scripting stuff, and use the POSIX standard instead:

Online POSIX 2008 (EEE Std 1003.1-2008) standard

There are four main parts:

• The Base Definitions (XBD): The basics of a POSIX-compliant system.
• The System Interfaces (XSH): What POSIX-compliant systems offer to programs. I.e. programming in C (stdlib)
• The Shell & Utilities (XCU): How a POSIX-compliant shell and system utilities behave.
• The Rationale (XRAT): Why they decided to do things the way they did.

Some Do's and Dont's:

• Read the Shell Command Language section so you know how the shell actually works. Surprisingly many shell scripters apparently have no idea how the shell really works, and get results through trial and error (which inevitably leads to buggy shell scripts.)
• Write POSIX Basic Regular Expressions
• Write POSIX Extended Regular Expressions if you really must
• Don't use or overwrite POSIX Environment reserved words when using variables.
• Use Shell & Utilities → Utilities as a reference guide on what will work on just about every Unix system.

Finally, read Bash Pitfalls to learn why your shell scripting sucks.

I often clone VirtualBox machines as a quick way to get a fresh box to test some stuff on. The problem is, I don't know which IP the new clone gets assigned. Fortunately, if you've got VirtualBox Guest Additions installed on the guest, you can use the guestproperty to get the IP.

Here's a quick shell script for listing the v4 IPs of all the running guest virtual machines.

#!/bin/sh

VBoxManage list runningvms | cut -d "{" -f1 | sed "s/\"//g" | while read VBOXNAME; do
    IP=$(VBoxManage guestproperty get "$VBOXNAME" /VirtualBox/GuestInfo/Net/0/V4/IP | cut -d":" -f2)
    echo "$VBOXNAME: $IP"
done

P.S. If you're using Debian guests: delete the /etc/udev/rules.d/*persistent-net* files. See here for why.

Put this in your .tmux.conf to enable scrolling using the Shift-PageUp/Down keys:

set -g terminal-overrides 'xterm*:smcup@:rmcup@'

As a developer and systems administrator, I use VirtualBox a lot for building binaries, testing upgrades, etc. It always struck me as a waste that I'd have to clone an entire HD image whenever I needed a fresh install of a machine. Why couldn't I just use a single base image for each Virtual Machine, and have VirtualBox perform copy-on-write whenever it made changes? That way, only the changes to the base image would have to be stored separately for each clone, saving lots of disk space. Turns out it is possible to do just that! I had some problems with the steps in that article though, so here's how I did it.

First, I created a new Virtual Machine and installed it like I always do. Once the VM was all set up, I shut it down, and cloned its harddisk:

$ VBoxManage clonehd ~/VirtualBox\ VMs/minimal.deb.local/minimal.deb.local.vdi ~/base.vdi

Next, I created a new Virtual Machine:

$ VBoxManage createvm --name "clone1" --ostype Debian_64 --register
Virtual machine 'clone1' is created and registered.
UUID: 1becc453-f4a9-44a8-a6c8-e43b80baf04d
Settings file: '/home/fboender/VirtualBox VMs/clone1/clone1.vbox'
$ VBoxManage modifyvm "clone1" --nic1 hostonly --hostonlyadapter1 "vboxnet0"
$ VBoxManage storagectl "clone1" --name "sata1" --add sata
$ VBoxManage storageattach "clone1" --storagectl "sata1" --port 0 --device 0 --type hdd --medium ~/base.vdi --mtype multiattach

The trick here lies in the --mtype multiattach option to the storageattach command. It tells VirtualBox that I'm going to attach this harddisk image to multiple different Virtual Machines. VirtualBox will then automatically do Copy-on-Write of all changes to a snapshot instead of to the base image. If I simply set the base.vdi harddisk image to immutable, as instructed by the article on Xaprb, I cannot attach it to multiple VirtualMachines. Using the --mtype multiattach also instructs VirtualBox to make persistant Copy-on-Writes. This means that, unlike the Xaprb article, your snapshot is not reset when starting the VirtualMachine. Thus you will not have to change the snapshots to autoreset=false.

You can start the VM now:

$ VBoxManage startvm "clone1"

If you want to create another VirtualMachine using the same base image, you can repeat the steps above, and replace every occurance of "clone1" with "clone2" or some other name. Then, when you attach the storage, you must not refer to the actual VDI file as it exists on disk, but you must simply refer to its name. So instead of specifying "--medium ~/base.vdi", simply enter: "--medium base.vdi". The full command thus becomes:

$ VBoxManage storageattach "clone2" --storagectl "sata1" --port 0 --device 0 --type hdd --medium base.vdi --mtype multiattach

We cannot refer directly to the image on disk, because it is already registered with VirtualBox. If you try to do this anyway, you will get an error such as:

VBoxManage: error: Cannot register the hard disk '/home/fboender/./base.vdi' {d3c861c1-6861-46c7-94a1-fd7a91987507} because a hard disk '/home/fboender/base.vdi' with UUID {d3c861c1-6861-46c7-94a1-fd7a91987507} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBox, interface IVirtualBox, callee nsISupports
Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, AccessMode_ReadWrite, pMedium.asOutParam())" at line 209 of file VBoxManageDisk.cpp
VBoxManage: error: Invalid UUID or filename "./base.vdi"

If you create new VMs through the GUI, and attach the existing "base.vdi" harddisk during the Wizard, it will automatically attach that image in multiattach mode.

Like I said, all changes to the Virtual Machines are not written to the base.vdi image, but to a snapshot instead. The snapshots are very minimal in size:

$ ls -lh VirtualBox\ VMs/clone1/Snapshots/
total 26M
-rw------- 1 fboender fboender 26M 2011-09-24 10:30 {8f91d6ba-71bb-4618-8c82-5d8bd13fb045}.vdi

Only 26 Mb for a full-blown Debian install. Not bad.

I guess the spammers finally found my blog, cause I've been getting a lot of pignback/trackback spam. I tried some anti-spam plugins, but none really worked, so I disabled pingbacks altogether. Here's how:

First, log into wordpress as an admin. Go to Settings → Discussion, and uncheck the Allow link notifications from other blogs (pingbacks and trackbacks.) box.

That's not all though, cause that just works for new articles. Old ones will still allow ping/trackbacks. As far as I could tell, WordPress doesn't have a feature to disable those for older posts, so we'll have to fiddle around in the database directly.

Connect to your MySQL server using the WordPress username and password. If you no longer remember those, you can find them in the wp-config.php file.

$ mysql -u USERNAME -p -h localhost
Password: PASSWORD
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1684228
Server version: 5.0.51a-24+lenny5 (Debian)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

At the MySQL prompt, we must now switch to our WordPress database. Again, if you don't remember the name, you can find it in the wp-config.php file. In my case, it is em_wordpress

mysql> USE em_wordpress;
Database changed

Finally, we update all posts and disable ping backs on them:

mysql> UPDATE wp_posts SET ping_status = 'closed';
Query OK, 1084 rows affected (0.10 sec)
Rows matched: 1084  Changed: 1084  Warnings: 0

There we go. No more ping- and trackback spam on old posts.

Here's a quick way to create your own Debian package. This is merely a simple package which will not be included in the official Debian repositories, so we can ignore most of the Debian packaging guidelines.

First off, we need to create a directory which shall hold the contents of the package. In this case, a simple Python script called 'myscript'.

mkdir myscript

Next, we need to create the control file. This file contains some meta-data for the Debian package such as the name, description, etc. It lives in a special directory called 'DEBIAN' in the root of the package.

mkdir myscript/DEBIAN

vi myscript/DEBIAN/control

We put the following contents in the control file

Package: myscript
Version: 0.1
Section: utils
Priority: optional
Architecture: all
Essential: no
Depends: python
Maintainer: Your Name 
Description: The short description of the script
 A longer description of the descript, possible
 spanning multiple lines.

Most of these fields are rather self-explanatory. You can find a list of valid Sections on the "List of Sections" page. Make sure to use the last part of the URL, not the actual title of the section. For scripts, the Architecture will almost always be 'all'. The Depends field tells the package installation software which other packages should be installed for this package to work correctly. Multiple packages can be specified by separating them with commas. See Chapter 7 of the Debian Policy Manual for details. The Description field is a single line containing a simple description of the package. The extended description can be placed under it, and may span multiple lines. Prefix each line with a single space.

Now we add the actual contents of the package to the myscript directory. We shall be installing the script in /usr/bin, so we create that directory and place our script in it.

mkdir -p myscript/usr/bin

cp ~/dev/myscript myscript/usr/bin/

Here's the final directory layout of the myscript directory:

myscript
myscript/usr
myscript/usr/bin
myscript/usr/bin/myscript
myscript/DEBIAN
myscript/DEBIAN/control

The final step: actually creating the package!

fakeroot dpkg-deb --build myscript

fakeroot is a special tool that runs another program (in this case dpkg-deb) and fakes all filesystem ownerships as being 'root:root'. This is needed because the files in this Debian package need to be owned by root. The dpkg-deb --build command takes care of creating the package.

The myscript.deb Debian package can now be installed on your system using dpkg:

dpkg -i ./myscript.deb

Users of a GUI can usually right-click .deb files in their file managers and choose to install them from there. There are no special things to consider for de-installation of the package, unless the binary in your package creates files in non-temporary storage or users' homedirs.

Please note again that this is not an official Debian package, and is missing many things required in well-made Debian packages ready for inclusion in the official Debian repositories.

Update::

If you want to include configuration files in the package, just add a etc/ directory in your package:

myscript
myscript/etc
myscript/etc/myscript
myscript/etc/myscript/myscript.conf

Normally when upgrading a package that contains a configuration file, Debian asks:

Configuration file `/etc/myscript/myscript.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** tvtgrab.conf (Y/I/N/O/D/Z) [default=N] ? 

To get the same thing for your package, you'll have to let dpkg know which files are configuration files. You can do so by including a conffiles in the DEBIAN directory:

cat myscript/DEBIAN/conffiles
/etc/myscript/myscript.conf

Debian will now recognize your configuration files.

Here's a nice study on the performance of Linux's network firewalling/packet mangling layer:

Netfilter Performance Testing

Conclusions (mine, based on the study):

• Netfilter/iptables is up to par with other filter solutions when it comes to plain routing.
• Netfilter/iptables is not up to par with other filter solutions when it comes to connection tracking (basically just getting all the traffic through netfilter and keeping track of it) and filtering
• When a chain has many rules, netfilter/iptables filtering performance drops significantly. Chain modifications (adding rules) performance also degrades significantly. This starts at 256 rules, so don't use more.

The problems seem to stem from the way Netfilter stores and processes the rules:

It is well known that netfilter/iptables does not scale well if one wants to use large number of rules in a single chain. The reason of the problem lies in the fact that the rules are processed in netfilter/iptables one after another, linearly.

The Minecraft server was running very slowly and gobbling up a significant amount of memory. Game-play was laggy, chunk loading stuttery and I saw a LOT of the following message in the server.log:

[WARNING] Can't keep up! Did the system time change, or is the server overloaded?

I decided to unleash my google-fu, dug through the Java manuals and found some optimizations that:

• Significantly reduced the CPU load.
• Reduced the memory usage of Minecraft from about 80% to 20 to 30% (of 1Gb). That means you can run Minecraft server on as little as 200Mb.
• Got rid of the "Can't keep up!" messages.
• Reduced lag on the server to almost nothing.
• Reduced chunk loading stuttering.

It does seem there has been a slight increase in chunks not loading properly, but that might be my imagination.

So what did I do? It's really simple:

• Run the Minecraft server and world from a RAM disk. This will greatly enhance performance of loading chunks and at the same time reduce CPU load. Memory usage in total will go up (because the entire world is now in RAM), but since the new Minecraft McRegion storage format (introduced in Minecraft v1.3 Beta) uses a lot less disk space, it's no big deal.
• Provide the -Xincgc option to Java. This enabled the concurrent incremental garbage collector, which basically means that Java won't pause for a couple of seconds to clean up old unused stuff (unloaded chunks). This reduces lag and choppiness in the loading of chunks/movement of mobs and destruction/placement of blocks.

I'll discuss here how to set up a RAM disk and how to make persistent copies of your live Minecraft map (or you'd lose everything on reboot).

Setting up a RAM disk

This section assumes that you're running under GNU/Linux, you've got Minecraft in a directory /home/minecraft/minecraft/minecraft_server/ (the directory where the minecraft_server.jar lives). Commands are prefixed by either a '$' or a '#' prompt, meaning you should either run the command as the minecraft user (or whichever user has read/write access to the minecraft server) or the root user.

Let's get started! First, check how much space you will need for your minecraft world:

$ du -hs /home/minecraft/minecraft/minecraft_server
50M	/home/minecraft/minecraft/minecraft_server

The minecraft directory is currently using 50mb. It's already a fairly large world, so we'll give it double that: 100Mb.

Now, move the minecraft_server directory to a different name, because we need to create an empty RAM disk in its place:

$ mv /home/minecraft/minecraft/minecraft_server /home/minecraft/minecraft/minecraft_server.persistent
$ mkdir /home/minecraft/minecraft/minecraft_server

Next, add an entry for the RAM disk to /etc/fstab. This will make sure it is automatically remounted when your system restarts

$ sudo echo "tmpfs    /home/minecraft/minecraft/minecraft_server     tmpfs    rw,size=100M    0    0" >> /etc/fstab

Mount it:

# mount /home/minecraft/minecraft/minecraft_server

Copy the contents off the backup minecraft_server directory over to the RAM disk:

$ cp -ar /home/minecraft/minecraft/minecraft_server.persistent/* /home/minecraft/minecraft/minecraft_server/

You can now start Minecraft with the following command:

$ tmux new -d -n "minecraft" "minecraft" "java -Xincgc -Xmx1G -jar minecraft_server.jar nogui"

Making persistent copies

It is imperative that you regularly create a persistent copy of the RAM disk! If the power on your server ever fails (or if you reboot it manually), your world is LOST! If you're running the Minecraft server in a `tmux` session (and you've started tmux with: 'tmux -n minecraft -s minecraft'), you can create a shellscript and call it from a CRON job say every hour. You can also get my Minecraft Server run script, which can also do backups and start/stop the Minecraft server without having to attach to the console. But for those who just want the persistent-copy script, here you go:

PATH_MC="/home/minecraft/minecraft/minecraft_server"

# Temporary turn off MC saving so we don't get a corrupt backup
tmux send -t "minecraft" "save-off" C-m
tmux send -t "minecraft" "save-all" C-m
# Wait until the MC server log indicates the save is complete
while true; do
    sleep 0.2
    TMP=`grep "Save complete." $PATH_MC/server.log | wc -l`
    if [ $TMP -gt $SAVE_COMPLETE ]; then
        break
    fi
done
# Create persistent copy from RAM to disk
rm -rf "$PATH_MC.persistent"
cp -ar "$PATH_MC" "$PATH_MC.persistent"
# Turn world saving back on
tmux send -t "minecraft" "save-on" C-m

Save it as a file called /home/minecraft/mc_persistent.sh and make it executable:

chmod 750 /home/minecraft/mc_persistent.sh

Create a now cronjob and call it every hour:

0 * * * * /home/minecraft/mc_persistent.sh

That's it! Happy lag-free mining.

There's an interesting on-going series of articles on file system latency over at Brendan's Blog. Usually when system administrators look into I/O performance, we look at the I/O of the disks. This is usually fine for a rough estimate of raw disk performance, but there's a lot more going on between the actual application and the disk: buffers, cache, the file system, etc. Brendan goes into detail regarding these matters by examining I/O performance of a MySQL database at both the disk and file system level:

• Part 1
• Part 2
• Part 3
• Part 4