Log

Put this in your .tmux.conf to enable scrolling using the Shift-PageUp/Down keys:

set -g terminal-overrides 'xterm*:smcup@:rmcup@'

As a developer and systems administrator, I use VirtualBox a lot for building binaries, testing upgrades, etc. It always struck me as a waste that I'd have to clone an entire HD image whenever I needed a fresh install of a machine. Why couldn't I just use a single base image for each Virtual Machine, and have VirtualBox perform copy-on-write whenever it made changes? That way, only the changes to the base image would have to be stored separately for each clone, saving lots of disk space. Turns out it is possible to do just that! I had some problems with the steps in that article though, so here's how I did it.

First, I created a new Virtual Machine and installed it like I always do. Once the VM was all set up, I shut it down, and cloned its harddisk:

$ VBoxManage clonehd ~/VirtualBox\ VMs/minimal.deb.local/minimal.deb.local.vdi ~/base.vdi

Next, I created a new Virtual Machine:

$ VBoxManage createvm --name "clone1" --ostype Debian_64 --register
Virtual machine 'clone1' is created and registered.
UUID: 1becc453-f4a9-44a8-a6c8-e43b80baf04d
Settings file: '/home/fboender/VirtualBox VMs/clone1/clone1.vbox'
$ VBoxManage modifyvm "clone1" --nic1 hostonly --hostonlyadapter1 "vboxnet0"
$ VBoxManage storagectl "clone1" --name "sata1" --add sata
$ VBoxManage storageattach "clone1" --storagectl "sata1" --port 0 --device 0 --type hdd --medium ~/base.vdi --mtype multiattach

The trick here lies in the --mtype multiattach option to the storageattach command. It tells VirtualBox that I'm going to attach this harddisk image to multiple different Virtual Machines. VirtualBox will then automatically do Copy-on-Write of all changes to a snapshot instead of to the base image. If I simply set the base.vdi harddisk image to immutable, as instructed by the article on Xaprb, I cannot attach it to multiple VirtualMachines. Using the --mtype multiattach also instructs VirtualBox to make persistant Copy-on-Writes. This means that, unlike the Xaprb article, your snapshot is not reset when starting the VirtualMachine. Thus you will not have to change the snapshots to autoreset=false.

You can start the VM now:

$ VBoxManage startvm "clone1"

If you want to create another VirtualMachine using the same base image, you can repeat the steps above, and replace every occurance of "clone1" with "clone2" or some other name. Then, when you attach the storage, you must not refer to the actual VDI file as it exists on disk, but you must simply refer to its name. So instead of specifying "--medium ~/base.vdi", simply enter: "--medium base.vdi". The full command thus becomes:

$ VBoxManage storageattach "clone2" --storagectl "sata1" --port 0 --device 0 --type hdd --medium base.vdi --mtype multiattach

We cannot refer directly to the image on disk, because it is already registered with VirtualBox. If you try to do this anyway, you will get an error such as:

VBoxManage: error: Cannot register the hard disk '/home/fboender/./base.vdi' {d3c861c1-6861-46c7-94a1-fd7a91987507} because a hard disk '/home/fboender/base.vdi' with UUID {d3c861c1-6861-46c7-94a1-fd7a91987507} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBox, interface IVirtualBox, callee nsISupports
Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, AccessMode_ReadWrite, pMedium.asOutParam())" at line 209 of file VBoxManageDisk.cpp
VBoxManage: error: Invalid UUID or filename "./base.vdi"

If you create new VMs through the GUI, and attach the existing "base.vdi" harddisk during the Wizard, it will automatically attach that image in multiattach mode.

Like I said, all changes to the Virtual Machines are not written to the base.vdi image, but to a snapshot instead. The snapshots are very minimal in size:

$ ls -lh VirtualBox\ VMs/clone1/Snapshots/
total 26M
-rw------- 1 fboender fboender 26M 2011-09-24 10:30 {8f91d6ba-71bb-4618-8c82-5d8bd13fb045}.vdi

Only 26 Mb for a full-blown Debian install. Not bad.

I guess the spammers finally found my blog, cause I've been getting a lot of pignback/trackback spam. I tried some anti-spam plugins, but none really worked, so I disabled pingbacks altogether. Here's how:

First, log into wordpress as an admin. Go to Settings → Discussion, and uncheck the Allow link notifications from other blogs (pingbacks and trackbacks.) box.

That's not all though, cause that just works for new articles. Old ones will still allow ping/trackbacks. As far as I could tell, WordPress doesn't have a feature to disable those for older posts, so we'll have to fiddle around in the database directly.

Connect to your MySQL server using the WordPress username and password. If you no longer remember those, you can find them in the wp-config.php file.

$ mysql -u USERNAME -p -h localhost
Password: PASSWORD
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1684228
Server version: 5.0.51a-24+lenny5 (Debian)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

At the MySQL prompt, we must now switch to our WordPress database. Again, if you don't remember the name, you can find it in the wp-config.php file. In my case, it is em_wordpress

mysql> USE em_wordpress;
Database changed

Finally, we update all posts and disable ping backs on them:

mysql> UPDATE wp_posts SET ping_status = 'closed';
Query OK, 1084 rows affected (0.10 sec)
Rows matched: 1084  Changed: 1084  Warnings: 0

There we go. No more ping- and trackback spam on old posts.

Here's a quick way to create your own Debian package. This is merely a simple package which will not be included in the official Debian repositories, so we can ignore most of the Debian packaging guidelines.

First off, we need to create a directory which shall hold the contents of the package. In this case, a simple Python script called 'myscript'.

mkdir myscript

Next, we need to create the control file. This file contains some meta-data for the Debian package such as the name, description, etc. It lives in a special directory called 'DEBIAN' in the root of the package.

mkdir myscript/DEBIAN

vi myscript/DEBIAN/control

We put the following contents in the control file

Package: myscript
Version: 0.1
Section: utils
Priority: optional
Architecture: all
Essential: no
Depends: python
Maintainer: Your Name 
Description: The short description of the script
 A longer description of the descript, possible
 spanning multiple lines.

Most of these fields are rather self-explanatory. You can find a list of valid Sections on the "List of Sections" page. Make sure to use the last part of the URL, not the actual title of the section. For scripts, the Architecture will almost always be 'all'. The Depends field tells the package installation software which other packages should be installed for this package to work correctly. Multiple packages can be specified by separating them with commas. See Chapter 7 of the Debian Policy Manual for details. The Description field is a single line containing a simple description of the package. The extended description can be placed under it, and may span multiple lines. Prefix each line with a single space.

Now we add the actual contents of the package to the myscript directory. We shall be installing the script in /usr/bin, so we create that directory and place our script in it.

mkdir -p myscript/usr/bin

cp ~/dev/myscript myscript/usr/bin/

Here's the final directory layout of the myscript directory:

myscript
myscript/usr
myscript/usr/bin
myscript/usr/bin/myscript
myscript/DEBIAN
myscript/DEBIAN/control

The final step: actually creating the package!

fakeroot dpkg-deb --build myscript

fakeroot is a special tool that runs another program (in this case dpkg-deb) and fakes all filesystem ownerships as being 'root:root'. This is needed because the files in this Debian package need to be owned by root. The dpkg-deb --build command takes care of creating the package.

The myscript.deb Debian package can now be installed on your system using dpkg:

dpkg -i ./myscript.deb

Users of a GUI can usually right-click .deb files in their file managers and choose to install them from there. There are no special things to consider for de-installation of the package, unless the binary in your package creates files in non-temporary storage or users' homedirs.

Please note again that this is not an official Debian package, and is missing many things required in well-made Debian packages ready for inclusion in the official Debian repositories.

Here's a nice study on the performance of Linux's network firewalling/packet mangling layer:

Netfilter Performance Testing

Conclusions (mine, based on the study):

• Netfilter/iptables is up to par with other filter solutions when it comes to plain routing.
• Netfilter/iptables is not up to par with other filter solutions when it comes to connection tracking (basically just getting all the traffic through netfilter and keeping track of it) and filtering
• When a chain has many rules, netfilter/iptables filtering performance drops significantly. Chain modifications (adding rules) performance also degrades significantly. This starts at 256 rules, so don't use more.

The problems seem to stem from the way Netfilter stores and processes the rules:

It is well known that netfilter/iptables does not scale well if one wants to use large number of rules in a single chain. The reason of the problem lies in the fact that the rules are processed in netfilter/iptables one after another, linearly.

The Minecraft server was running very slowly and gobbling up a significant amount of memory. Game-play was laggy, chunk loading stuttery and I saw a LOT of the following message in the server.log:

[WARNING] Can't keep up! Did the system time change, or is the server overloaded?

I decided to unleash my google-fu, dug through the Java manuals and found some optimizations that:

• Significantly reduced the CPU load.
• Reduced the memory usage of Minecraft from about 80% to 20 to 30% (of 1Gb). That means you can run Minecraft server on as little as 200Mb.
• Got rid of the "Can't keep up!" messages.
• Reduced lag on the server to almost nothing.
• Reduced chunk loading stuttering.

It does seem there has been a slight increase in chunks not loading properly, but that might be my imagination.

So what did I do? It's really simple:

• Run the Minecraft server and world from a RAM disk. This will greatly enhance performance of loading chunks and at the same time reduce CPU load. Memory usage in total will go up (because the entire world is now in RAM), but since the new Minecraft McRegion storage format (introduced in Minecraft v1.3 Beta) uses a lot less disk space, it's no big deal.
• Provide the -Xincgc option to Java. This enabled the concurrent incremental garbage collector, which basically means that Java won't pause for a couple of seconds to clean up old unused stuff (unloaded chunks). This reduces lag and choppiness in the loading of chunks/movement of mobs and destruction/placement of blocks.

I'll discuss here how to set up a RAM disk and how to make persistent copies of your live Minecraft map (or you'd lose everything on reboot).

Setting up a RAM disk

This section assumes that you're running under GNU/Linux, you've got Minecraft in a directory /home/minecraft/minecraft/minecraft_server/ (the directory where the minecraft_server.jar lives). Commands are prefixed by either a '$' or a '#' prompt, meaning you should either run the command as the minecraft user (or whichever user has read/write access to the minecraft server) or the root user.

Let's get started! First, check how much space you will need for your minecraft world:

$ du -hs /home/minecraft/minecraft/minecraft_server
50M	/home/minecraft/minecraft/minecraft_server

The minecraft directory is currently using 50mb. It's already a fairly large world, so we'll give it double that: 100Mb.

Now, move the minecraft_server directory to a different name, because we need to create an empty RAM disk in its place:

$ mv /home/minecraft/minecraft/minecraft_server /home/minecraft/minecraft/minecraft_server.persistent
$ mkdir /home/minecraft/minecraft/minecraft_server

Next, add an entry for the RAM disk to /etc/fstab. This will make sure it is automatically remounted when your system restarts

$ sudo echo "tmpfs    /home/minecraft/minecraft/minecraft_server     tmpfs    rw,size=100M    0    0" >> /etc/fstab

Mount it:

# mount /home/minecraft/minecraft/minecraft_server

Copy the contents off the backup minecraft_server directory over to the RAM disk:

$ cp -ar /home/minecraft/minecraft/minecraft_server.persistent/* /home/minecraft/minecraft/minecraft_server/

You can now start Minecraft with the following command:

$ tmux new -d -n "minecraft" "minecraft" "java -Xincgc -Xmx1G -jar minecraft_server.jar nogui"

Making persistent copies

It is imperative that you regularly create a persistent copy of the RAM disk! If the power on your server ever fails (or if you reboot it manually), your world is LOST! If you're running the Minecraft server in a `tmux` session (and you've started tmux with: 'tmux -n minecraft -s minecraft'), you can create a shellscript and call it from a CRON job say every hour. You can also get my Minecraft Server run script, which can also do backups and start/stop the Minecraft server without having to attach to the console. But for those who just want the persistent-copy script, here you go:

PATH_MC="/home/minecraft/minecraft/minecraft_server"

# Temporary turn off MC saving so we don't get a corrupt backup
tmux send -t "minecraft" "save-off" C-m
tmux send -t "minecraft" "save-all" C-m
# Wait until the MC server log indicates the save is complete
while true; do
    sleep 0.2
    TMP=`grep "Save complete." $PATH_MC/server.log | wc -l`
    if [ $TMP -gt $SAVE_COMPLETE ]; then
        break
    fi
done
# Create persistent copy from RAM to disk
rm -rf "$PATH_MC.persistent"
cp -ar "$PATH_MC" "$PATH_MC.persistent"
# Turn world saving back on
tmux send -t "minecraft" "save-on" C-m

Save it as a file called /home/minecraft/mc_persistent.sh and make it executable:

chmod 750 /home/minecraft/mc_persistent.sh

Create a now cronjob and call it every hour:

0 * * * * /home/minecraft/mc_persistent.sh

That's it! Happy lag-free mining.

There's an interesting on-going series of articles on file system latency over at Brendan's Blog. Usually when system administrators look into I/O performance, we look at the I/O of the disks. This is usually fine for a rough estimate of raw disk performance, but there's a lot more going on between the actual application and the disk: buffers, cache, the file system, etc. Brendan goes into detail regarding these matters by examining I/O performance of a MySQL database at both the disk and file system level:

• Part 1
• Part 2
• Part 3
• Part 4

(The lastest version of this article is always available in stand-alone HTML format and in PDF format. The original AsciiDoc source is also available. Please link to the HTML version, not this Blog post!)

SSH is capable of more than you'd think! This article describes some of the lesser known features and configuration options. It covers authentication, authorization, tunnels and proxies, file transfer and more.

Read the rest of this entry »

Interesting article over at Linuxconfig.org:

This article describes a simple way on how to create a home made debian package and include it into a local package repository. Although we could use a existing Debian/Ubuntu package, we will start from scratch by creating our own minimalistic unofficial debian package. Once our package is ready, we will include it into our local package repository. This article illustrates very simplistic approach of creating debian package, however it may serve as a template in many different scenarios.

A while ago I fiddled around with my Vim configuration, and I removed some things I thought weren't necessary. A little while later I noticed that, when I copied things in a terminal Vim by selecting them with the mouse or a normal visual selection, I couldn't paste them into other X11 programs such as the terminal, Firefox and other Vim instances. Of course, I didn't have a backup of my old vimrc.

After much fiddling around with the various guioptions, mouse and clipboard settings, and many a Google search, I found a page which offered some insight:

For this to work, your Vim has to be compiled with the +xterm_clipboard setting. Run

vim --version | grep xterm_clipboard

to see if that option is compiled in

My Vim (Xubuntu 10.10) showed -xterm_clipboard instead of +xterm_clipboard meaning it didn't support directly cutting/copying to the X11 clipboard. It turns out that I had removed the vim-gtk package and installed the normal vim package, since I never use the GTK/Gnome version anyway.

Re-installing the vim-gtk package solved the problem:

aptitude install vim-gtk

If you want to be able to select with the mouse, you will also need to set the following in your ~/.vimrc:

set mouse=a

If you don't have that you can still select with the mouse, but it won't be Vim who does your selection but your terminal emulator. That results in line numbers being included in your selection (if you've set number) as well as lines longer than the terminal being cut off.