Electricmonk.nl weblog » linux

Tmux scrolling with shift-pageup/down

admin — Thu, 05 Jan 2012 06:38:11 +0000

Put this in your .tmux.conf to enable scrolling using the Shift-PageUp/Down keys:

set -g terminal-overrides 'xterm*:smcup@:rmcup@'

Multiple VirtualBox VMs using one base image (copy-on-write)

admin — Sat, 24 Sep 2011 08:34:06 +0000

As a developer and systems administrator, I use VirtualBox a lot for building binaries, testing upgrades, etc. It always struck me as a waste that I'd have to clone an entire HD image whenever I needed a fresh install of a machine. Why couldn't I just use a single base image for each Virtual Machine, and have VirtualBox perform copy-on-write whenever it made changes? That way, only the changes to the base image would have to be stored separately for each clone, saving lots of disk space. Turns out it is possible to do just that! I had some problems with the steps in that article though, so here's how I did it.

First, I created a new Virtual Machine and installed it like I always do. Once the VM was all set up, I shut it down, and cloned its harddisk:

$ VBoxManage clonehd ~/VirtualBox\ VMs/minimal.deb.local/minimal.deb.local.vdi ~/base.vdi

Next, I created a new Virtual Machine:

$ VBoxManage createvm --name "clone1" --ostype Debian_64 --register
Virtual machine 'clone1' is created and registered.
UUID: 1becc453-f4a9-44a8-a6c8-e43b80baf04d
Settings file: '/home/fboender/VirtualBox VMs/clone1/clone1.vbox'
$ VBoxManage modifyvm "clone1" --nic1 hostonly --hostonlyadapter1 "vboxnet0"
$ VBoxManage storagectl "clone1" --name "sata1" --add sata
$ VBoxManage storageattach "clone1" --storagectl "sata1" --port 0 --device 0 --type hdd --medium ~/base.vdi --mtype multiattach

The trick here lies in the --mtype multiattach option to the storageattach command. It tells VirtualBox that I'm going to attach this harddisk image to multiple different Virtual Machines. VirtualBox will then automatically do Copy-on-Write of all changes to a snapshot instead of to the base image. If I simply set the base.vdi harddisk image to immutable, as instructed by the article on Xaprb, I cannot attach it to multiple VirtualMachines. Using the --mtype multiattach also instructs VirtualBox to make persistant Copy-on-Writes. This means that, unlike the Xaprb article, your snapshot is not reset when starting the VirtualMachine. Thus you will not have to change the snapshots to autoreset=false.

You can start the VM now:

$ VBoxManage startvm "clone1"

If you want to create another VirtualMachine using the same base image, you can repeat the steps above, and replace every occurance of "clone1" with "clone2" or some other name. Then, when you attach the storage, you must not refer to the actual VDI file as it exists on disk, but you must simply refer to its name. So instead of specifying "--medium ~/base.vdi", simply enter: "--medium base.vdi". The full command thus becomes:

$ VBoxManage storageattach "clone2" --storagectl "sata1" --port 0 --device 0 --type hdd --medium base.vdi --mtype multiattach

We cannot refer directly to the image on disk, because it is already registered with VirtualBox. If you try to do this anyway, you will get an error such as:

VBoxManage: error: Cannot register the hard disk '/home/fboender/./base.vdi' {d3c861c1-6861-46c7-94a1-fd7a91987507} because a hard disk '/home/fboender/base.vdi' with UUID {d3c861c1-6861-46c7-94a1-fd7a91987507} already exists
VBoxManage: error: Details: code NS_ERROR_INVALID_ARG (0x80070057), component VirtualBox, interface IVirtualBox, callee nsISupports
Context: "OpenMedium(Bstr(pszFilenameOrUuid).raw(), enmDevType, AccessMode_ReadWrite, pMedium.asOutParam())" at line 209 of file VBoxManageDisk.cpp
VBoxManage: error: Invalid UUID or filename "./base.vdi"

If you create new VMs through the GUI, and attach the existing "base.vdi" harddisk during the Wizard, it will automatically attach that image in multiattach mode.

Like I said, all changes to the Virtual Machines are not written to the base.vdi image, but to a snapshot instead. The snapshots are very minimal in size:

$ ls -lh VirtualBox\ VMs/clone1/Snapshots/
total 26M
-rw------- 1 fboender fboender 26M 2011-09-24 10:30 {8f91d6ba-71bb-4618-8c82-5d8bd13fb045}.vdi

Only 26 Mb for a full-blown Debian install. Not bad.

Creating simple Debian packages

admin — Tue, 06 Sep 2011 19:51:26 +0000

Here's a quick way to create your own Debian package. This is merely a simple package which will not be included in the official Debian repositories, so we can ignore most of the Debian packaging guidelines.

First off, we need to create a directory which shall hold the contents of the package. In this case, a simple Python script called 'myscript'.

mkdir myscript

Next, we need to create the control file. This file contains some meta-data for the Debian package such as the name, description, etc. It lives in a special directory called 'DEBIAN' in the root of the package.

mkdir myscript/DEBIAN

vi myscript/DEBIAN/control

We put the following contents in the control file

Package: myscript
Version: 0.1
Section: utils
Priority: optional
Architecture: all
Essential: no
Depends: python
Maintainer: Your Name 
Description: The short description of the script
 A longer description of the descript, possible
 spanning multiple lines.

Most of these fields are rather self-explanatory. You can find a list of valid Sections on the "List of Sections" page. Make sure to use the last part of the URL, not the actual title of the section. For scripts, the Architecture will almost always be 'all'. The Depends field tells the package installation software which other packages should be installed for this package to work correctly. Multiple packages can be specified by separating them with commas. See Chapter 7 of the Debian Policy Manual for details. The Description field is a single line containing a simple description of the package. The extended description can be placed under it, and may span multiple lines. Prefix each line with a single space.

Now we add the actual contents of the package to the myscript directory. We shall be installing the script in /usr/bin, so we create that directory and place our script in it.

mkdir -p myscript/usr/bin

cp ~/dev/myscript myscript/usr/bin/

Here's the final directory layout of the myscript directory:

myscript
myscript/usr
myscript/usr/bin
myscript/usr/bin/myscript
myscript/DEBIAN
myscript/DEBIAN/control

The final step: actually creating the package!

fakeroot dpkg-deb --build myscript

fakeroot is a special tool that runs another program (in this case dpkg-deb) and fakes all filesystem ownerships as being 'root:root'. This is needed because the files in this Debian package need to be owned by root. The dpkg-deb --build command takes care of creating the package.

The myscript.deb Debian package can now be installed on your system using dpkg:

dpkg -i ./myscript.deb

Users of a GUI can usually right-click .deb files in their file managers and choose to install them from there. There are no special things to consider for de-installation of the package, unless the binary in your package creates files in non-temporary storage or users' homedirs.

Please note again that this is not an official Debian package, and is missing many things required in well-made Debian packages ready for inclusion in the official Debian repositories.

Netfilter (iptables) performance tests

admin — Fri, 29 Jul 2011 10:26:51 +0000

Here's a nice study on the performance of Linux's network firewalling/packet mangling layer:

Netfilter Performance Testing

Conclusions (mine, based on the study):

Netfilter/iptables is up to par with other filter solutions when it comes to plain routing.
Netfilter/iptables is not up to par with other filter solutions when it comes to connection tracking (basically just getting all the traffic through netfilter and keeping track of it) and filtering
When a chain has many rules, netfilter/iptables filtering performance drops significantly. Chain modifications (adding rules) performance also degrades significantly. This starts at 256 rules, so don't use more.

The problems seem to stem from the way Netfilter stores and processes the rules:

It is well known that netfilter/iptables does not scale well if one wants to use large number of rules in a single chain. The reason of the problem lies in the fact that the rules are processed in netfilter/iptables one after another, linearly.

Minecraft Server optimization

admin — Fri, 22 Jul 2011 18:40:33 +0000

The Minecraft server was running very slowly and gobbling up a significant amount of memory. Game-play was laggy, chunk loading stuttery and I saw a LOT of the following message in the server.log:

[WARNING] Can't keep up! Did the system time change, or is the server overloaded?

I decided to unleash my google-fu, dug through the Java manuals and found some optimizations that:

Significantly reduced the CPU load.
Reduced the memory usage of Minecraft from about 80% to 20 to 30% (of 1Gb). That means you can run Minecraft server on as little as 200Mb.
Got rid of the "Can't keep up!" messages.
Reduced lag on the server to almost nothing.
Reduced chunk loading stuttering.

It does seem there has been a slight increase in chunks not loading properly, but that might be my imagination.

So what did I do? It's really simple:

Run the Minecraft server and world from a RAM disk. This will greatly enhance performance of loading chunks and at the same time reduce CPU load. Memory usage in total will go up (because the entire world is now in RAM), but since the new Minecraft McRegion storage format (introduced in Minecraft v1.3 Beta) uses a lot less disk space, it's no big deal.
Provide the -Xincgc option to Java. This enabled the concurrent incremental garbage collector, which basically means that Java won't pause for a couple of seconds to clean up old unused stuff (unloaded chunks). This reduces lag and choppiness in the loading of chunks/movement of mobs and destruction/placement of blocks.

I'll discuss here how to set up a RAM disk and how to make persistent copies of your live Minecraft map (or you'd lose everything on reboot).

Setting up a RAM disk

This section assumes that you're running under GNU/Linux, you've got Minecraft in a directory /home/minecraft/minecraft/minecraft_server/ (the directory where the minecraft_server.jar lives). Commands are prefixed by either a '$' or a '#' prompt, meaning you should either run the command as the minecraft user (or whichever user has read/write access to the minecraft server) or the root user.

Let's get started! First, check how much space you will need for your minecraft world:

$ du -hs /home/minecraft/minecraft/minecraft_server
50M	/home/minecraft/minecraft/minecraft_server

The minecraft directory is currently using 50mb. It's already a fairly large world, so we'll give it double that: 100Mb.

Now, move the minecraft_server directory to a different name, because we need to create an empty RAM disk in its place:

$ mv /home/minecraft/minecraft/minecraft_server /home/minecraft/minecraft/minecraft_server.persistent
$ mkdir /home/minecraft/minecraft/minecraft_server

Next, add an entry for the RAM disk to /etc/fstab. This will make sure it is automatically remounted when your system restarts

$ sudo echo "tmpfs    /home/minecraft/minecraft/minecraft_server     tmpfs    rw,size=100M    0    0" >> /etc/fstab

Mount it:

# mount /home/minecraft/minecraft/minecraft_server

Copy the contents off the backup minecraft_server directory over to the RAM disk:

$ cp -ar /home/minecraft/minecraft/minecraft_server.persistent/* /home/minecraft/minecraft/minecraft_server/

You can now start Minecraft with the following command:

$ tmux new -d -n "minecraft" "minecraft" "java -Xincgc -Xmx1G -jar minecraft_server.jar nogui"

Making persistent copies

It is imperative that you regularly create a persistent copy of the RAM disk! If the power on your server ever fails (or if you reboot it manually), your world is LOST! If you're running the Minecraft server in a `tmux` session (and you've started tmux with: 'tmux -n minecraft -s minecraft'), you can create a shellscript and call it from a CRON job say every hour. You can also get my Minecraft Server run script, which can also do backups and start/stop the Minecraft server without having to attach to the console. But for those who just want the persistent-copy script, here you go:

PATH_MC="/home/minecraft/minecraft/minecraft_server"

# Temporary turn off MC saving so we don't get a corrupt backup
tmux send -t "minecraft" "save-off" C-m
tmux send -t "minecraft" "save-all" C-m
# Wait until the MC server log indicates the save is complete
while true; do
    sleep 0.2
    TMP=`grep "Save complete." $PATH_MC/server.log | wc -l`
    if [ $TMP -gt $SAVE_COMPLETE ]; then
        break
    fi
done
# Create persistent copy from RAM to disk
rm -rf "$PATH_MC.persistent"
cp -ar "$PATH_MC" "$PATH_MC.persistent"
# Turn world saving back on
tmux send -t "minecraft" "save-on" C-m

Save it as a file called /home/minecraft/mc_persistent.sh and make it executable:

chmod 750 /home/minecraft/mc_persistent.sh

Create a now cronjob and call it every hour:

0 * * * * /home/minecraft/mc_persistent.sh

That's it! Happy lag-free mining.

SSH Tips and Tricks

admin — Mon, 02 May 2011 14:18:43 +0000

(The lastest version of this article is always available in stand-alone HTML format and in PDF format. The original AsciiDoc source is also available. Please link to the HTML version, not this Blog post!)

SSH is capable of more than you'd think! This article describes some of the lesser known features and configuration options. It covers authentication, authorization, tunnels and proxies, file transfer and more.

Authentication

Passwordless key

One very useful feature of SSH are passwordless keys. They are especially useful when you're writing scripts that need to run commands on a remote hosts. Since those will run unattended, you don't want them to prompt for a password.

Warning

Be careful when using passwordless keys! If the security of the client machine is compromised, the remote server will be just as compromised! Check the Authorization section of this article for tips on how to limit the damage of a compromised client machine! You can also use an ssh agent, in which case you will only have to enter the password once after system boot after which scripts can run without requiring a password.

You can generate passwordless keys using the ssh-keygen tool. Simply press enter when asked for the password:

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa): /home/user/passwordless_rsa
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/passwordless_rsa.
Your public key has been saved in /home/user/passwordless_rsa.pub.
The key fingerprint is:
70:50:84:e3:ea:de:62:43:4e:cf:76:a4:86:8e:c7:29 user@eek

This generates a normal public key and a private key without a password. We can add the public key to a remote machine's list of authorized keys using the ssh-copy-id tool:

$ ssh-copy-id -i /home/user/passwordless_rsa.pub user@remote_host
Now try logging into the machine, with "ssh 'user@remote_host'", and check in:
  .ssh/authorized_keys
to make sure we haven't added extra keys that you weren't expecting.

We specify which public key to transport using the -i /home/user/passwordless_rsa.pub option.

Now we can ssh to the remote_host without using a password. If the private key has been placed in the current user's .ssh directory, ssh will automatically detect it when trying to connect. If you want to be sure it finds the private key to connect with, you can once again specify the -i path_to_private_key option:

$ ssh -i /home/user/passwordless_rsa user@remote_host
user@remote_host$

When using passwordless keys, you may provide the -q and -o "BatchMode=yes" options to SSH in order to make it quiet. Otherwise, errors related to SSH might be interpreted as errors from remote commands.

SSH Agent

The SSH Agent is a tool which can keep your private keys in memory. When connecting to a remote machine, any SSH session (including scp and sftp) will try to contact a running agent on the machine to see if the required private key is already loaded. If it is, it will be used to connect to the remote machine. This way you only have to enter your password for a private key once, instead of each time you want to connect.

You can test if an SSH Agent is already running with the following command:

$ ssh-add -l
Could not open a connection to your authentication agent.

As you can see, no agent is currently running. We can start one with the following command:

$ eval `ssh-agent`
Agent pid 6265

The above command runs the ssh-agent program, which will output some other commands, which are then run by the current shell. This is the actual output of the ssh-agent program:

SSH_AUTH_SOCK=/tmp/ssh-tXJFfB6269/agent.6269; export SSH_AUTH_SOCK;
SSH_AGENT_PID=6270; export SSH_AGENT_PID;
echo Agent pid 6270;

As you can see, the ssh-agent is started. It then creates a socket in the /tmp directory and sets that location as an environment variable so that SSH clients (ssh, scp, sftp) know where to contact the agent.

We can add keys to the agent using the ssh-add tool:

$ ssh-add user.rsa
Enter passphrase for user.rsa:
Identity added: user.rsa (user.rsa)

We can now connect to any remote machine that has the public key counterpart in its authorized_keys file without having to enter our password again.

The ssh_agent program generates a random name for the socket. One frequent problem with starting the SSH agent like we did just now is that only the current shell knows about the socket location. If we open another terminal, it won't know about the running SSH agent, since the environment variable isn't set in the new terminal. We can work around this by specifying our own path to a socket with the -a option.

Combining this knowledge, we can add a few lines to our .profile (or .bashrc) startup script to start an agent if none is running yet. We also check for a forwarded agent and don't do anything if a forwarded agent is found (more on forwarding agents later on).

# Do not start an SSH agent if the user has a forwarded agent.
if [ -z "$SSH_AUTH_SOCK" ]; then
    # Check if a local SSH agent is already running. If not, start one.
    export SSH_AUTH_SOCK="$HOME/.ssh/sshagent.socket"
    if [ ! -S "$SSH_AUTH_SOCK" ]; then
        eval `ssh-agent -a "$SSH_AUTH_SOCK"`
    fi
fi

With this script in your .profile, you will only have to start one agent ever for a given user as long as that machine doesn't reboot (or the agent is killed in some other way). If you log out, the agent is not killed, and will be re-used the next time you log in.

Agent forwarding

Perhaps the most useful feature of SSH is Agent Forwarding. We can tell ssh it should do Agent Forwarding by supplying the -A commandline option when we ssh to remote machine. For this to work, you have to have an SSH agent running locally, or agent forwarding had to be enabled when you SSHed into your current session. (Use ssh-add -l to see if an agent is avaiable).

$ ssh -A user@remote_host

When enabling Agent Forwarding with the -A switch, ssh will automatically create a socket at the remote host and set some environment variables (SSH_AUTH_SOCK, most notably) when you connect to a remote machine. If an ssh session needs to perform authentication, it will first try to reach the SSH agent through the socket. All requests will be sent back to the original SSH agent. This works for as many nested sessions as you'd like.

Using Agent forwarding, you can start a single agent on, say, your desktop machine. Every authentication request made by any SSH session will be sent back to the agent running on your desktop machine, without the need to start additional SSH agents on remote machines and loading keys there. As you can imagine, this is a much better method of keeping private keys secure than storing them on every machine you need to SSH from.

You can enable SSH agent forwarding for all the hosts you SSH to automatically (without the need to specify the -A switch) by putting the following in your ~/.ssh/config:

Host *
        ForwardAgent yes

Authorization

Restricting commands

You can restrict which command can be run by someone logging in with a public/private key in the authorized_keys file. For instance, to restrict a certain public/private key to running the df -h command (to view avaiable diskspace), you add a line to the authorized_keys file like this (Public key shortened for brevity):

command="/bin/df -h" ssh-dss AAAAC8ghi9ldw== user@host

Now when we SSH to that machine (and we have that key loaded):

$ ssh user@remote_host
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/dev-root   39G  2.2G   35G   6% /
Connection to host closed.

Its not possible for a single key to run multiple commands via any normal SSH configuration mechanism. However, SSH will set an environment variable $SSH_ORIGINAL_COMMAND with the command the user tried to run. We can take advantage of that by writing a shellscript. For example, we can create a script called commands.sh on the remote host:

#!/bin/sh

case $SSH_ORIGINAL_COMMAND in
        "diskspace")
                df -h
                ;;
        "dirlist")
                ls -1
                ;;
        "apache_restart")
                /etc/init.d/apache restart
                ;;
        *)
                echo "Unknown command"
esac

Then we restrict the user to running that shellscript:

command="/bin/sh /home/user/commands.sh" ssh-dss AAAAC8ghi9ldw== user@host

The user can now run multiple commands by specifying as an argument after the ssh command (The -q+ option makes ssh quiet, so it doesn't show any output other than that of the remote command):

$ ssh -q user@remote_host diskspace
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/dev-root   39G  2.2G   35G   6% /

$ ssh -q remote_host dirlist
commands.sh
dump.sql

Warning

Make sure you do not include any commands which lets the user run an interactive shell. Also, make sure users cannot overwrite the commands.sh and .ssh/authorized_keys files.

Restricting addition of keys

By default, SSH puts the authorized_keys files in the user's home directory. This allows users to add other keys themselves; a situation you might want to avoid. You can change the location where the key files are kept in the /etc/ssh/sshd_config file, using the AuthorizedKeysFile option:

#AuthorizedKeysFile     %h/.ssh/authorized_keys
AuthorizedKeysFile      /etc/ssh/authorized_keys/%u

The %u will expand to the username. The location of the authorized_keys file for a user named "john" will become /etc/ssh/authorized_keys/john. %h expands to the users homedirectory.

Make sure users can't write to the files in question.

Restricting which users can SSH

You can restrict which users are allowed to use SSH with the AllowUsers option in /etc/ssh/sshd_config:

AllowUsers john pete

Note	if the AllowUsers setting is completely missing from the sshd config file, all users are allowed (see man sshd_config). You may prefer to leave it that way — your choice. I prefer to make the usernames explicit because I'm paranoid ;-)

Input / Output

Like every other Unix tool, SSH can use input and output redirection. When running a command on a remote machine using SSH, it will redirect any input given to it locally to the remote command. Any output from the remote command is redirected back to your local machine. This allows for some very useful time-savers.

For instance, we can run the command du (diskusage) on the remote machine, and locally pipe it into xdu to get a graphical representation on our local X11 desktop of the remote disk usage:

$ ssh remote_host du /var/www/ | xdu

Or suppose we want to transfer a remote directory's contents to the local machine without using scp (for whatever reason). We can remotely create a tar archive of the directory, and instruct tar to write it to the standard output (using the minus as the filename) instead of a file. SSH will transfer the remote standard output of tar to our local machine, where we can untar it in the same manner:

$ ssh remote_host tar -cf - Documents/notes | tar -xf -
$ ls Documents/notes/
dev.c.txt                            sysadmin.networking.txt
dev.git.txt                          sysadmin.openssl.txt
dev.mysql.txt                        sysadmin.solaris.txt

Perhaps we need to create a local copy of a MySQL database on a remote machine. Unfortunatelly, MySQL access is not remotely allowed and the harddisk on the remote machine is full, so we can't create a dump there, transport it to our local machine and read it in. No worry, SSH to the rescue:

$ ssh remote_host mysqldump -u USER -pPASSWORD -h localhost DATABASENAME > dump.sql

Or we can just import it directly:

$ ssh remote_host mysqldump -u USER -pPASSWORD -h localhost DATABASENAME | mysql -u USER -pPASSWORD -h localhost DATABASENAME

Likewise we can locally pipe data into ssh and use it at the remote host. Again, we use the minus-sign to indicate reading from standard in:

$ echo "hello world" | ssh remote_host "cat >foo.txt"

This will put "hello world" in a file called foo.txt on the remote host. We need to quote the parts that contain the redirection on the host ("cat >foo.txt") or it will be picked up by the local shell.

Tunnels and proxies

Local port tunnel

Sometimes you may need to use a certain service on a network, but the network has been firewalled against external connections on ports other than the SSH port. SSH allows us to create a tunnel into the remote network. Suppose we are on a network 192.168.1.x and we want to connect to port 80 on a machine with 192.168.56.3. But the 192.168.56.x network is firewalled, and we can only access it through a bastion host at 192.168.56.1. Here's what we do:

$ ssh -L 80:192.168.56.3:80 user@192.168.56.1

SSH will now create a tunnel to 192.168.56.3 port 80 through 192.168.56.1. The -L option takes three arguments, separated with colons: local_port:remote_host:remote_port. The local_port is where SSH will listen for incoming connections on the machine where you issued this command. remote_host:remote_port is the machine/port to which you wish to create the tunnel. It is important to remember that this is as you'd view it from the server you're ssh-ing too (192.168.56.1 in this case), not as you'd view it from your local machine.

You can additionally specify the -N switch to prevent SSH from actually logging in to 192.168.56.1.

SOCKS5 proxy

We can use SSH as a SOCKS5 proxy. An SOCKS5 proxy works much like a normal tunnel, but works with multiple clients at the same time, and is not restricted to forwarding of a single port. We can start a SOCKS5 using the -D option:

Socks5 is pretty neat, as it allows you to proxy stuff without the server having to know anything about the way the client works. For instance, if we give the following command:

$ ssh -D 8080 remote_host

Now we can configure local clients (such as Firefox, Pidgin Instant Messanger, Chrome, etc) to use the proxy. All network traffic (with the exception of DNS, possibly!) will go through the SOCKS5 proxy. For instance:

$ chromium-browser --proxy-server="socks5://127.0.0.1:8080"

ProxyCommand

Many networks require you to SSH to a bastion (firewall/gateway) server before you're able to SSH to any machine on the network. This becomes tedious quickly, as you have to SSH twice each time. The ProxyCommand is a setting in your ssh configuration file which can do this for you automatically.

Assume we want to SSH to a host 'web1.example.com'. Before we can SSH to this host we first have to SSH to 'example.com'. We can SSH directly to 'web1.example.com' by putting the following in our ~.ssh/config file:

Host web1
        ProxyCommand ssh example.com nc web1.example.com 22

This requires that the nc (netcat) tool is installed on web1.example.com. If we SSH to web1 now, we are automatically sent to web1.example.com. It's even possible to use scp and other SSH tricks directly, thus saving us the trouble of having to transfer files to example.com first, then to our local machine (or vice versa).

ProxyCommand can also be used with other things. To SSH through a HTTP proxy at 192.0.2.0 port 8080:

ProxyCommand /usr/bin/nc -X connect -x 192.0.2.0:8080 %h %p

Configuration

Client configuration

The SSH client configuration lives in the file /home/USER/.ssh/config. It has many useful directives. The basic way it works is we specify a host identifier, and add configuration settings to that host.

For instance, if your local username is john, but on the backup machine backup.example.com it's always backup, you can tell SSH to automatically use that username to log in:

Host backup.example.com
        User backup

You can create aliases (much like the /etc/hosts file) to save some typing:

Host backup
        Hostname backup.example.com
        User backup

If you want to apply a certain configuration option to every host, use the asterisk wildcard:

Host *
        ForwardAgent yes
        ServerAliveInterval 5
        ServerAliveCountMax 720

Check man ssh_config for more useful options in this configuration file.

Transferring files

Secure Copy (scp)

This should be obvious, but you can use the scp tool to transfer files between hosts using SSH. To copy a file localhost.txt to your home directory on the remote host:

scp localfile.txt user@remote_host:

To put the file in a different path:

scp localfile.txt user@remote_host:/path/absolute/to/root/

and

scp localfile.txt user@remote_host:path/in/homedir/

Transferring an entire directory is possible using the -r switch:

scp -r dir/ user@remote_host:

SFTP

OpenSSH (and most other SSH implementations) offer a secure FTP server. With it you can securely (encrypted) transfer files and authenticate using the default SSH authentication methods such as passwords or public keys. Contrary to ordinary FTP server (unless they run on TLS or some other form of encryption), passwords are not sent in plain-text over the network. SFTP also makes it easier for Windows user to transfer files between hosts, as there are many good free SFTP clients available. I personally recommend Filezilla.

The SFTP server needs to be enabled in the SSH server configuration. Edit /etc/ssh/sshd.config on the server and add the following line:

Subsystem sftp /usr/lib/openssh/sftp-server

Restart SSH and you should be able to user the SFTP server:

/etc/init.d/ssh restart

On the client, issue the sftp command:

$ sftp user@remote_host
Connected to host.
sftp> ls
bin          svn.tar.gz   xims
sftp> get svn.tar.gz
Fetching /home/user/svn.tar.gz to svn.tar.gz
/home/user/svn.tar.gz           100%   11KB  11.0KB/s   00:00

You may get an error like this:

subsystem request failed on channel 0
Couldn't read packet: Connection reset by peer

This means that either the sftp-server was not properly configurated, or your login shell on the remote server is outputting some text not expected by the SFTP server (perhaps a welcome message or something). Remove the output and try again.

A different way of enabling the SFTP server is in the authorized_keys file:

command="/usr/lib/openssh/sftp-server" ssh-dss AAAAC8ghi9ldw== user@host

This will restrict any user that logs in using the public key AAAAC8ghi9ldw== to SFTP. The user cannot login using a normal SSH session. This does not require you to enable the SFTP server in /etc/ssh/sshd.conf.

You'll have to make sure that the user can't write to the .ssh directory nor upload any files such as .bashrc, .profile, etc, otherwise the user can overwrite those by uploading their own version, and they can still execute anything they like by just logging in with sftp. You can do this by creating these files and then changing their ownership and rights in such a way that the user can't write to them. Because it's hard to guess what files you should create so that the user can't cause any harm, it's best to simply create a seperate directory in which they can upload stuff, and lock off write access to their entire home directory.

Unlike the ssh and scp commands, sftp does not have a -i switch with which you can manually give the location of the private key to log in with. Fortunately, we can still provide one through the -o (options) switch:

sftp -o IdentityFile=/home/user/.ssh/some_key_rsa username@hostname

This can be convenient in the case of automated tasks. The custom key does not have to have a password and can be placed anywhere. Speaking of automated tasks, here's an example of running sftp in a batch-mode:

echo "PUT myfile" | sftp -o IdentityFile=/home/user/.ssh/some_key_rsa -b - username@hostname

Most normal FTP servers support jailing the user in a certain directory, preventing them from wandering around the file system. SFTP does not have this built-in ability, but we can use normal linux chroot/jails to jail a user to certain directory. For more information, see http://www.electricmonk.nl/log/2007/08/09/jailing-sftpscp/

Remote mount filesystem (SSHfs)

Perhaps the most useful tool in existance: sshfs. SSHfs provides tools to locally mount a directory on a remote server over SSH, much like NFS. SSHfs requires remote support for SFTP. See the previous section on how to enable it. SSHfs is separate from the normal ssh tools, and is implemented as a FUSE (userland) filesystem. On Debian and Ubuntu machines you can easily install it using aptitude:

# aptitude install sshfs

Once installed, we can mount a remote directory using the sshfs tool:

$ sshfs user@remote_host:path/to/dir ./local_mountpoint
$ cd local_mountpoint
$ ls
file1    file2    file3

Unmounting can be done with the fusermount tool:

$ fusermount -u ./local_mountpoint

Easy way to create a Debian package and repository

admin — Mon, 25 Apr 2011 06:56:48 +0000

Interesting article over at Linuxconfig.org:

This article describes a simple way on how to create a home made debian package and include it into a local package repository. Although we could use a existing Debian/Ubuntu package, we will start from scratch by creating our own minimalistic unofficial debian package. Once our package is ready, we will include it into our local package repository. This article illustrates very simplistic approach of creating debian package, however it may serve as a template in many different scenarios.

Vim, X11 and the clipboard (Copy, paste)

admin — Tue, 05 Apr 2011 12:34:06 +0000

A while ago I fiddled around with my Vim configuration, and I removed some things I thought weren't necessary. A little while later I noticed that, when I copied things in a terminal Vim by selecting them with the mouse or a normal visual selection, I couldn't paste them into other X11 programs such as the terminal, Firefox and other Vim instances. Of course, I didn't have a backup of my old vimrc.

After much fiddling around with the various guioptions, mouse and clipboard settings, and many a Google search, I found a page which offered some insight:

For this to work, your Vim has to be compiled with the +xterm_clipboard setting. Run
vim --version | grep xterm_clipboard
to see if that option is compiled in

My Vim (Xubuntu 10.10) showed -xterm_clipboard instead of +xterm_clipboard meaning it didn't support directly cutting/copying to the X11 clipboard. It turns out that I had removed the vim-gtk package and installed the normal vim package, since I never use the GTK/Gnome version anyway.

Re-installing the vim-gtk package solved the problem:

aptitude install vim-gtk

If you want to be able to select with the mouse, you will also need to set the following in your ~/.vimrc:

set mouse=a

If you don't have that you can still select with the mouse, but it won't be Vim who does your selection but your terminal emulator. That results in line numbers being included in your selection (if you've set number) as well as lines longer than the terminal being cut off.

BoxBackup on Debian/Ubuntu

admin — Tue, 01 Feb 2011 15:36:10 +0000

(The lastest version of this article is always available in stand-alone HTML format and in PDF format. The original AsciiDoc source is also available. Please link to the HTML version, not this Blog post!)

1. Introduction

BoxBackup is an online remote backup tool for Unix systems (BSDs, Linux, MacOSX). It is robust, secure, low on resources and can perform both in continues backup mode and snapshotting. In continues backup mode changes will be pushed to the server soon after they happen; in snapshot mode mode BoxBackup behaves more like traditional backup programs and creates snapshots every fixed amount of time.

This article will describe how to set up a BoxBackup server and client on Debian and Ubuntu machines. Much of this article can also be used on other Unix-like systems, however it will not discuss how to compile BoxBackup.

1.1. Assumptions

This article assumes the following:

The server (where backups will be stored) is called server. Its FQDN will be server.example.com.
The client (which will be backed up) is called client. Its FQDN will be client.example.com.
Backups will be stored in a directory /storage/backup on server.

2. For the lazy

This chapter is for the lazy. It quickly describes how to setup BoxBackup, but provides no in-depth information. Please note that commands are prefixed by a prompt that indicates on which machine and in which directory to run the command.

The server will allow the client 100Gb of storage with a hard limit of 120Gb. The client will have account ID 00000001 (more on account IDs later). BoxBackup's user-land RAID-like configuration will not be used. We will have to move some files between the client and the server. For this we use the scp program and the user account named user on both machines.

2.1. Server configuration

Install the server:

server:/root# aptitude install boxbackup-server

Delete Debian's generated configuration (We'll recreate it ourselves):

server:/root# rm -rf /etc/boxbackup/

Find out filesystem block size:

server:/root# /sbin/dumpe2fs -h /dev/sdb1 | grep 'Block size'
dumpe2fs 1.41.4 (27-Jan-2009)
Block size:               4096

Create RAID-like configuration file:

server:/root# raidfile-config  /etc/boxbackup/ 4096 /storage/

Create backup storage directory:

server:/root# mkdir /storage/backup/
server:/root# chown bbstored:bbstored /storage/backup
server:/root# chmod 750 /storage/backup

Create BoxBackup server configuration and certificates:

server:/root# bbstored-config /etc/boxbackup/ server.example.com bbstored
server:/root# bbstored-certs ca init
server:/root# bbstored-certs ca sign-server /etc/boxbackup/bbstored/server.example.com-csr.pem
server:/root# cp ca/servers/server.example.com-cert.pem /etc/boxbackup/bbstored
server:/root# cp ca/roots/clientCA.pem /etc/boxbackup/bbstored
server:/root# chown -R bbstored:bbstored /etc/boxbackup/
server:/root# chmod 700 /etc/boxbackup/

Start the server:

server:/root# /etc/init.d/boxbackup-server start
server:/root# grep "Box Backup" /var/log/syslog
Jan 31 21:45:13 server Box Backup (bbstored)[30775]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbstored.conf

Important

Keep the /root/ca directory around. It is required to sign new BoxBackup clients' certificate files.

2.2. Client configuration

Create an account on the server:

server:/root# bbstoreaccounts create 00000001 0 100g 120g

Install the client software:

client:/root# aptitude install boxbackup-client
client:/root# rm /etc/boxbackup/bbackupd.conf
client:/root# rm /etc/boxbackup/bbackupd/*

Create client configuration and certificates:

client:/root# bbackupd-config /etc/boxbackup/ lazy 00000001 server.example.com /var/run /var/lib/svn/
client:/root# scp /etc/boxbackup/bbackupd/00000001-FileEncKeys.raw user@server.example.com:
client:/root# scp /etc/boxbackup/bbackupd/00000001-csr.pem user@server.example.com:

Sign certificates on the server:

server:/root# bbstored-certs ca sign /home/user/00000001-csr.pem
server:/root# scp ca/clients/00000001-cert.pem user@client.example.com:
server:/root# scp ca/roots/serverCA.pem user@client.example.com:
server:/root# /etc/init.d/boxbackup-server restart

Install signed certificates on the client and start client:

client:/root# mv /home/user/00000001-cert.pem /etc/boxbackup/bbackupd/
client:/root# mv /home/user/serverCA.pem /etc/boxbackup/bbackupd/
client:/root# /etc/init.d/boxbackup-client start
client:/root# grep "Box Backup" /var/log/syslog
Jan 31 21:55:13 client Box Backup (bbackupd)[20975]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbackupd.conf
Jan 31 21:55:13 client Box Backup (bbackupd)[20975]: NOTICE: Beginning scan of local files

Important

Keep the 0000001-FileEncKeys.raw file around. It is required to restore backups.

If anything fails, please take a look at the rest of this document. It explains in more detail how to setup BoxBackup and may provide clues as to the problems you are encountering.

3. Setting up the server (bbstored)

3.1. Installation

The first thing is setting up the server. This is where the backups will be stored. It will run the bbstored program, which is a server daemon that clients can connect to in order to upload data.

The server will be designated as server in the commands given in this chapter. Each command is prefixed with a shell-prompt to indicate on which machine (client or server) the command should be entered. The server's Fully Qualified Domain Name will be server.example.com. The client's will be client.example.com. We will have to move some files (certificates) around between the server and the client in order to sign them. For this we will use a user account with username user. It is assumed the root user on both systems can use secure copy (scp) to move files from one system to the other under this user account.

Let's get started by installing the BoxBackup server. We use Debian, but it should also work under Ubuntu. It is unknown to me which different versions of BoxBackup are compatible with each other, so I assume you'll be using the same version on both the client and the server. If you're not using Debian or Ubuntu, please check your distribution's package list to see if a pre-compiled version of BoxBackup is available. If not, you will have to compile it yourself. This is beyond the scope of this article, so check the BoxBackup documentation for instructions.

server:/root# aptitude install boxbackup-server

Debian/Ubuntu will install the BoxBackup server and some configuration tools. It will also create a user called bbstored. The BoxBackup server (bbstored) will run as this user.

Debian also might generate a configuration file and certificates for you. We will delete those because they confuse the process of installing and configuring BoxBackup (as what Debian does for you and what you need to do manually is terribly documented and it stores things in the wrong place).

server:/root# rm -rf /etc/boxbackup/

3.2. Creating a store

BoxBackup server stores backups in a store. We will configure this store at /storage/backup. This directory will contain a store for each client that backs up using the server. E.g. the backups for the client with account ID 00000001 (more on account IDs later on) will be stored in /storage/backup/00000001/.

BoxBackup provides a RAID-like setup where backups are written to multiple locations; usually different physical disks. This RAID-like setup has nothing to do with actual RAID, so don't get confused by that. This article will not be using the RAID-like facilities of BoxBackup.

Before we can create our store, we must first determine the block size of our file system . This can be done using the dumpe2fs tool. Provide the proper device name to the tool:

server:/root# mount | grep storage
/dev/sdb1 on /storage type ext3 (rw)

In this case the device is /dev/sdb1. Let's get the block size:

server:/root# /sbin/dumpe2fs -h /dev/sdb1 | grep 'Block size'
dumpe2fs 1.41.4 (27-Jan-2009)
Block size:               4096

As we can see, our block size is 4096. We can now configure a Boxbackup store.

server:/root# raidfile-config  /etc/boxbackup/ 4096 /storage/
WARNING: userland RAID is disabled.
Config file written.

If the BoxBackup user (bbstored) has write permissions to the /storage/ directory, you can skip the following steps and continue with the bbstored-config step.

Now we create a directory for the backups and set the proper permissions:

server:/root# mkdir /storage/backup/
server:/root# chown bbstored:bbstored /storage/backup
server:/root# chmod 750 /storage/backup

3.3. Server configuration file / certificates

Time to create the bbstored configuration. We do this with the bbstored-config tool. It takes three parameters:

config_dir: The configuration location. In our case, /etc/boxbackup/.
server_name: The FQDN of the server. We use server.example.com.
username: The user which bbstored will run as. Debian created the bbstored user for us, so we'll use that. It's the same as the daemon name, which is customary.

Run the command:

server:/root# bbstored-config /etc/boxbackup/ server.example.com  bbstored

Checking permissions on /storage//backup
Checking permissions on /storage//backup
Checking permissions on /storage//backup

Setup bbstored config utility

Configuration:
   Writing configuration file: /etc/boxbackup//bbstored.conf
   Writing empty accounts file: /etc/boxbackup//bbstored/accounts.txt
   Server hostname: server.example.com
   RaidFile config: /etc/boxbackup//raidfile.conf

Creating blank accounts file
Generating private key...
Generating RSA private key, 2048 bit long modulus
................................................................
............................................................................+++
.............................................................+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:An optional company name []:

Writing configuration file /etc/boxbackup//bbstored.conf
===================================================================

bbstored basic configuration complete.

What you need to do now...
1) Sign /etc/boxbackup//bbstored/server.example.com-csr.pem
   using the bbstored-certs utility.
2) Install the server certificate and root CA certificate as
      /etc/boxbackup//bbstored/server.example.com-cert.pem
      /etc/boxbackup//bbstored/clientCA.pem
3) You may wish to read the configuration file
      /etc/boxbackup//bbstored.conf
   and adjust as appropraite.
4) Create accounts with bbstoreaccounts
5) Start the backup store daemon with the command
      /usr/local/bin/bbstored /etc/boxbackup//bbstored.conf
   in /etc/rc.local, or your local equivalent.
===================================================================

This will generate a configuration file at /etc/boxbackup/bbstored.conf and generate a server certificate sign request file at /etc/boxbackup/bbstored/server.example.com-csr.pem.

The next step is to create and sign the server certificate. For this we need to generate a root certificate first. We use the bbstored-certs tool to do this:

server:/root# bbstored-certs ca init

Generating RSA private key, 2048 bit long modulus
...................................................................
.................................................................+++
................................................+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:An optional company name []:

Signature ok
subject=/CN=Backup system client root
Getting Private key
Generating RSA private key, 2048 bit long modulus
.........................+++
............................................+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:An optional company name []:

Signature ok
subject=/CN=Backup system server root
Getting Private key

This generates a co directory at our current location. This directory contains a bunch of certificates, including the root certificate. We need this root certificate to sign the server and client certificates.

Important

We need to keep this ca directory around as it is required to sign client certificates! Store it in a secure location!

Okay, let's sign the server certificate using our root certificate:

server:/root# bbstored-certs ca sign-server /etc/boxbackup/bbstored/server.example.com-csr.pem

This certificate is for backup server

server.example.com

Signing the wrong certificate compromises the security of your backup system.

Would you like to sign this certificate? (type 'yes' to confirm)
yes
Signature ok
subject=/CN=server.example.com
Getting CA Private Key

Certificate signed

Install the files

  ca/servers/server.example.com-cert.pem
  ca/roots/clientCA.pem

on the server.

We do as the output says and copy the signed server certificates to the configuration directory:

server:/root# cp ca/servers/server.electricmonk.nl-cert.pem /etc/boxbackup/bbstored
server:/root# cp ca/roots/clientCA.pem /etc/boxbackup/bbstored
server:/root# chown -R bbstored:bbstored /etc/boxbackup/
server:/root# chmod 700 /etc/boxbackup/

3.4. Starting the server

That concludes the configuration part of the server. We will have to create client accounts, but that will be covered in the chapter on client configuration. Let's try out the server:

server:/root# /etc/init.d/boxbackup-server start
Starting boxbackup-server: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbstored.conf
bbstored.

It seems to start. We can check /var/log/syslog to see if it really did:

server:/root# grep "Box Backup" /var/log/syslog
Jan 31 21:53:14 server Box Backup (bbstored)[15884]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbstored.conf

If that's all the output, everything is now okay. Final verifications to see if the server is really running can be made using ps and netstat:

server:/root# ps axf | grep bbstored
 15884 ?        S      0:00 /usr/sbin/bbstored /etc/boxbackup/bbstored.conf
 15885 ?        S      0:14  \_ /usr/sbin/bbstored /etc/boxbackup/bbstored.conf

It's running.

server:/root# netstat -pant | grep bbstored
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 82.171.91.37:2201       0.0.0.0:*               LISTEN      15884/bbstored

It's listening for incoming connections on IP 82.171.91.37, at TCP port 2201. Time to set up the client!

4. Setting up a client (bbackupd)

In this chapter we will set up a client. The client will run the bbackupd program, which is a daemon that scans the files you want to backup and sends them to the server if it encounters new or modified files. It does this efficiently: if there are no changes, nothing is sent to the server. Otherwise it only sends the parts of the files that have changed; not the whole file is transmitted unless it is not yet on the server.

In this chapter we designate the client (the machine you wish to backup) as client. The backup server which will store our backups is designated server. Each command is prefixed with a shell-prompt to indicate on which machine (client or server) the command should be entered. The server's Fully Qualified Domain Name will be server.example.com. The client's will be client.example.com. We will have to move some files (certificates) around between the server and the client in order to sign them. For this we will use a user account with username user. It is assumed the root user on both systems can use secure copy (scp) to move files from one system to the other under this user account.

4.1. Creating an account

Before we start installing things on the client machine, we should first create a BoxBackup store account on the server. We use the bbstoreaccounts tool for this. It takes five parameters:

command: The command determines which action we want to perform on the account. In this case we wish to create an account, so we will use the create command.
account_id: The Account ID is a a hexdecimal account number. We will use 00000001. A second client would be 00000002, a tenth client would be 0000000A, etc.
raid_disk: We must specify a RAID (BoxBackup's userland RAID-like concept) disk on which to store the backups for this client. In our case, we didn't setup multiple RAID disks, so we have only one: 0.
soft_limit: The Soft Limit determines when the client will voluntarily stop sending file changes to the server. Presumably (though I'm not sure) it will not start transferring new changes when the soft limit has been exceeded on the server. However, if a single change exceeds the soft limit after the client has started uploading it, it will continue uploading until it hits the hard_limit. We'll set the soft limit to 100Gb.
hard_limit: The hard limit. This should NEVER be bigger than the free space you have. We set it too 120Gb.

Let's create the account:

server:/root# bbstoreaccounts create 00000001 0 100g 120g
NOTICE: Account 0x00000001 created.

4.2. Installation

Time to move on to the actual client. Note that some of these commands are run on the client and some are run on the server. Double check the prompt to see on which machine the command needs to be run!.

client:/root# aptitude install boxbackup-client

Debian will configure some things for us, but that's just confusing (and didn't work for me), so we remove Debian's configuration and keys:

client:/root# rm /etc/boxbackup/bbackupd.conf
client:/root# rm /etc/boxbackup/bbackupd/*

4.3. Client configuration / certificates

Now we generate the configuration ourselves. We supply the lazy mode parameter which will continuously scan our system for changes. This sounds resource intensive, but it actually spreads the load better over time. We also supply the account ID (which we generated earlier on), the server name (FQDN), a run directory where BoxBackup can store temporary information and a path we want backed up. (We add more later on). We will be backing up /var/lib/svn.

client:/root# bbackupd-config /etc/boxbackup/ lazy 00000001 server.electricmonk.nl /var/run /var/lib/svn/

Setup bbackupd config utility.

Configuration:
   Writing configuration file: /etc/boxbackup//bbackupd.conf
   Account: 00000001
   Server hostname: server.electricmonk.nl
   Directories to back up:
      /var/lib/svn/

Note: If other file systems are mounted inside these directories, then
they will NOT be backed up. You will have to create separate locations for
any mounted filesystems inside your backup locations.

Generating private key...
Generating RSA private key, 2048 bit long modulus
...................................+++
.................+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:An optional company name []:

Generating keys for file backup
Writing notify script /etc/boxbackup//bbackupd/NotifySysadmin.sh
Writing configuration file /etc/boxbackup//bbackupd.conf

===================================================================

bbackupd basic configuration complete

What you need to do now...

1) Make a backup of /etc/boxbackup//bbackupd/00000001-FileEncKeys.raw
   This should be a secure offsite backup.
   Without it, you cannot restore backups. Everything else can
   be replaced. But this cannot.
   KEEP IT IN A SAFE PLACE, OTHERWISE YOUR BACKUPS ARE USELESS.
2) Send /etc/boxbackup//bbackupd/00000001-csr.pem
   to the administrator of the backup server, and ask for it to
   be signed.
3) The administrator will send you two files. Install them as
      /etc/boxbackup//bbackupd/00000001-cert.pem
      /etc/boxbackup//bbackupd/serverCA.pem
   after checking their authenticity.
4) You may wish to read the configuration file
      /etc/boxbackup//bbackupd.conf
   and adjust as appropriate. There are some
   notes in it on excluding files you do not
   wish to be backed up.
5) Review the script
      /etc/boxbackup//bbackupd/NotifySysadmin.sh
   and check that it will email the right person when the store
   becomes full. This is important -- when the store is full, no
   more files will be backed up. You want to know about this.
6) Start the backup daemon with the command
      /usr/local/bin/bbackupd /etc/boxbackup//bbackupd.conf
   in /etc/rc.local, or your local equivalent.
   Note that bbackupd must run as root.
===================================================================

Remember to make a secure, offsite backup of your backup keys,
as described in step 1 above. If you do not, you have no backups.

4.4. Signing certificates

The previous step has generated a file encryption key (/etc/boxbackup/bbackupd/00000001-FileEncKeys.raw) and a client SSL certificate sign request at /etc/boxbackup/bbackupd/00000001-csr.pem. These must be sent to the server so they can be signed and added to the server's known SSL certificates. Here we use scp to copy the files to the server, but you can use any other means to get them there.

client:/root# scp /etc/boxbackup/bbackupd/00000001-FileEncKeys.raw user@server.electricmonk.nl:
    00000001-FileEncKeys.raw           100% 1024     1.0KB/s   00:00
client:/root# scp /etc/boxbackup/bbackupd/00000001-csr.pem user@server.electricmonk.nl:
    00000001-csr.pem                   100%  899     0.9KB/s   00:00

Now we must sign the certificate on the server. Note that the following commands are ran on the server, not the client. Also note that your current working directory must be the one in which we previously ran the bbstored-certs ca init command. (the ca directory must be present).

server:/root# bbstored-certs ca sign /home/user/00000001-csr.pem

This certificate is for backup account

00000001

Ensure this matches the account number you are expecting. The filename is

/home/user/00000001-csr.pem

which should include this account number, and additionally, you should check
that you received it from the right person.

Signing the wrong certificate compromises the security of your backup system

Would you like to sign this certificate? (type 'yes' to confirm)
yes
Signature ok
subject=/CN=BACKUP-00000001
Getting CA Private Key

Certificate signed.

Send the files

  ca/clients/00000001-cert.pem
  ca/roots/serverCA.pem

to the client.

We send the signed certificate and the server's certificate to the client:

server:/root# scp ca/clients/00000001-cert.pem user@client:
    00000001-cert.pem          100%  997     1.0KB/s   00:00
server:/root# scp ca/roots/serverCA.pem user@client:
    serverCA.pem               100% 1021     1.0KB/s   00:00

And on the client we install them in the correct directory:

client:/root# mv /home/user/00000001-cert.pem /etc/boxbackup/bbackupd/
client:/root# mv /home/user/serverCA.pem /etc/boxbackup/bbackupd/

4.5. Starting the client

Now we can start the client:

client:# /etc/init.d/boxbackup-client start

In the syslog on the client we should see:

Jan 31 21:55:13 client Box Backup (bbackupd)[20975]: NOTICE: Starting daemon, version 0.11rc2, config: /etc/boxbackup/bbackupd.conf
Jan 31 21:55:13 client Box Backup (bbackupd)[20975]: NOTICE: Beginning scan of local files
Jan 31 21:55:14 client Box Backup (bbackupd)[20975]: NOTICE: About to notify administrator about event backup-start, running script '/etc/boxbackup//bbackupd/NotifySysadmin.sh backup-start'
Jan 31 21:59:45 client Box Backup (bbackupd)[20975]: NOTICE: Finished scan of local files
Jan 31 21:59:46 client Box Backup (bbackupd)[20975]: NOTICE: About to notify administrator about event backup-finish, running script '/etc/boxbackup//bbackupd/NotifySysadmin.sh backup-finish'
Jan 31 21:59:46 client Box Backup (bbackupd)[20975]: NOTICE: File statistics: total file size uploaded 36012577, bytes already on server 0, encoded size 7161527

In the syslog on the server we should see:

Jan 31 21:55:13 server Box Backup (bbstored)[15884]: WARNING: Message from child process 16488: Incoming connection from 213.19.146.55 port 52963
Jan 31 21:55:13 server Box Backup (bbstored)[16488]: NOTICE: Login from Client ID 0x00000001 Read/Write
Jan 31 21:58:46 server Box Backup (bbstored)[19666]: NOTICE: Session finished for Client ID 0x00000001

4.6. Verifying backups

After the initial backup is done, we can verify that files have been transfered using the bbackupquery tool. This tool should be run on the client:

client:/root# bbackupquery
NOTICE: Box Backup Query Tool v0.11rc2, (c) Ben Summers and contributors 2003-2008
Login complete.

Type "help" for a list of commands.

query > ls
00000002 -d---- var-lib-svn-
00001798 -d---- etc-
query > cd var-lib-svn-
query > ls
00000003 f----- .htpasswd
00000004 f----- .svnaccess
query > exit

Not all files may have been transferred initially. For instance, Box Backup doesn't transfer files if they've been modified recently. It will try again later. If a file is continuously modified, it will eventually (after a max age) upload the file anyway.

4.7. Adding additional paths

If you want to add additional paths which need to backed up, edit the /etc/boxbackup/bbackupd.conf file and search for the BackupLocations directive. You can add paths here. For example, to backup /var/lib/svn and /etc:

BackupLocations
{
        var-lib-svn-
        {
                Path = /var/lib/svn/
        }
        etc-
        {
                Path = /etc/
        }
}

Afterwards simply restart bbackupd:

client:/root# /etc/init.d/bbackupd restart

5. Troubleshooting

The primary methods of troubleshooting are experimentation and the /var/log/syslog file.

5.1. Server connectivity

We can use various tools to test if the server is running and can be reached. To check if the server is running, use ps:

server:/root# ps axf | grep bbstored
 15884 ?        S      0:00 /usr/sbin/bbstored /etc/boxbackup/bbstored.conf
 15885 ?        S      0:14  \_ /usr/sbin/bbstored /etc/boxbackup/bbstored.conf

If it's not running, check /var/log/syslog for any errors.

To see if the server is listening for incoming connections from the clients:

server:/root# netstat -pant | grep bbstored
tcp        0      0 82.171.91.37:2201       0.0.0.0:*               LISTEN      15884/bbstored

This shows the server is listening on IP 82.171.91.37 on TCP port 2201. The daemon should listen on an which can be reached by the clients. If it listens on IP 0.0.0.0, it will accept connections from any machine.

To test if we can reach the server from the client, we can use a port scanner such as nmap:

client:/root#  nmap -p 2201 server.example.com
Starting Nmap 4.62 ( http://nmap.org ) at 2011-02-01 14:21 CET
Interesting ports on server.example.com (82.171.91.37):
PORT     STATE SERVICE
2201/tcp open  ats

If nmap lists the port as open, the client can reach the server.

5.2. Syslog errors and warning

Both the server and the client will log errors to /var/log/syslog. Here are a couple of errors you might run into:

5.2.1. Expired certificate

If you spot this in the syslog:

Jan 31 17:32:45 server Box Backup (bbstored)[30775]: WARNING: Message from child process 31604: Incoming connection from 213.19.146.55 port 59381
Jan 31 17:32:45 server Box Backup (bbstored)[31604]: ERROR: SSL error during Accept: error:14094415:SSL routines:SSL3_READ_BYTES:sslv3 alert certificate expired
Jan 31 17:32:45 server Box Backup (bbstored)[31604]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(245)
Jan 31 17:32:45 server Box Backup (bbstored)[31604]: ERROR: Error in child process, terminating connection: exception Connection TLSHandshakeFailed(7/30)

One of the certificates has expired. You may be dealing with a bug in older Debian/Ubuntu versions. More information is available here:

http://www.mail-archive.com/debian-bugs-rc@lists.debian.org/msg242206.html

5.2.2. Certificates missing

THe following error is reported when certificates can't be loaded. In this case, because they couldn't be found.

Jan 31 18:38:32 server Box Backup (bbstored)[12102]: ERROR: SSL error during Load certificates: error:02001002:system library:fopen:No such file or directory
Jan 31 18:38:32 server Box Backup (bbstored)[12102]: ERROR: SSL error during Load certificates: error:20074002:BIO routines:FILE_CTRL:system lib
Jan 31 18:38:32 server Box Backup (bbstored)[12102]: ERROR: SSL error during Load certificates: error:140DC002:SSL routines:SSL_CTX_use_certificate_chain_file:system lib
Jan 31 18:38:32 server Box Backup (bbstored)[12102]: WARNING: Exception thrown: ServerException(TLSLoadCertificatesFailed) at TLSContext.cpp(118)
Jan 31 18:38:32 server Box Backup (bbstored)[12102]: FATAL: Terminating due to exception Server TLSLoadCertificatesFailed (3/25)

Double check the instructions to make sure you've moved certificates to the correct place.

5.2.3. Permission denied

Various permission denied problems may arise:

Feb  1 10:47:21 server Box Backup[29518]: ERROR: FileHandleGuard: failed to open file '/etc/boxbackup/bbstored.conf': Permission denied
Feb  1 10:47:21 server Box Backup[29518]: WARNING: Exception thrown: CommonException(OSFileOpenError) at ../../lib/common/Guards.h(81)

Check the permission and ownership on the server to make sure that the files in /etc/boxbackup belong to the user bbstored. Also check that the bbstored user can write to /storage/backup.

5.2.4. Certificate verification failure

On the client, you may see the following error:

Jan 31 17:32:46 client Box Backup (bbackupd)[7339]: ERROR: SSL error during Connect: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
Jan 31 17:32:46 client Box Backup (bbackupd)[7339]: WARNING: Exception thrown: ConnectionException(Conn_TLSHandshakeFailed) at SocketStreamTLS.cpp(250)
Jan 31 17:32:46 client Box Backup (bbackupd)[7339]: NOTICE: About to notify administrator about event backup-error, running script '/etc/boxbackup//bbackupd/NotifySysadmin.sh backup-error'
Jan 31 17:32:46 client Box Backup (bbackupd)[7339]: ERROR: Exception caught (Connection TLSHandshakeFailed 7/30), reset state and waiting to retry...
Jan 31 17:32:56 client Box Backup (bbackupd)[7339]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0
Jan 31 17:33:13 client Box Backup (bbackupd)[7339]: NOTICE: Terminating daemon

This might be related to the Debian bug mentioned in the Expired certificate troubleshooting chapter. Check the server's /var/log/syslog for more details.

5.2.5. Network connectivity problem

In case of the following:

Jan 31 18:34:28 client Box Backup (bbackupd)[27060]: ERROR: Failed to connect to socket (type 1, name server.example.com, port 2201): Connection refused (111)
Jan 31 18:34:28 client Box Backup (bbackupd)[27060]: WARNING: Exception thrown: ConnectionException(Conn_SocketConnectError) at SocketStream.cpp(223)
Jan 31 18:34:28 client Box Backup (bbackupd)[27060]: NOTICE: About to notify administrator about event backup-error, running script '/etc/boxbackup/bbackupd/NotifySysadmin.sh backup-error'
Jan 31 18:34:29 client Box Backup (bbackupd)[27060]: ERROR: Exception caught (Connection SocketConnectError (Probably a network issue between client and server, bad hostname, or server not running.) 7/15), reset state and waiting to retry...
Jan 31 18:34:39 client Box Backup (bbackupd)[27060]: NOTICE: File statistics: total file size uploaded 0, bytes already on server 0, encoded size 0

Perform the checks given in Server connectivity part of the troubleshooting chapter.

5.3. Missing files

Sometimes, after a backup, files may not be present on the remote server. Not all files may have been transferred initially. For instance, Box Backup doesn't transfer files if they've been modified recently. It will try again later. If a file is continuously modified, it will eventually (after a max age) upload the file anyway.

5.3.1. Other problems

The BoxBackup homepage lists some other possible problems:

http://boxbackup.org/trouble.html

6. About this document

6.1. Copyright / License

This document may be freely distributed, in part or as a whole, on any medium, without the prior authorization of the author, provided that this Copyright notice remains intact, and there will be no obstruction as to the further distribution of this document. You may not ask a fee for the contents of this document, though a fee to compensate for the distribution of this document is permitted.

Modifications to this document are permitted, provided that the modified document is distributed under the same license as the original document and no copyright notices are removed from this document. All contents written by an author stays copyrighted by that author.

Failure to comply to one or all of the terms of this license automatically revokes your rights granted by this license

All brand and product names mentioned in this document are trademarks or registered trademarks of their respective holders.

6.2. Feedback

All feedback on this document is welcome at .

Modifying Boxee keybindings

admin — Thu, 21 Oct 2010 12:50:09 +0000

Boxee is a pretty awesome media player for Linux. Now, I don't have a remote to control it, but I do have a Logitech DiNovo Wireless keyboard with a detachable numeric pad. Modifying Boxee to support controlling it using the numeric pad is rather easy:

Browse to the map that contains the Boxee installation. For me that's /opt/boxee but it's probably different on your system (try /usr/share/boxee or /usr/lib/boxee). Open up the file system/keymaps/keyboard.xml in your favorite editor. Here you will see a big list of keybindings. Find the section which contains the section. In that section, add the following:

      SkipPrevious
      SkipNext
      Left
      Down
      Right
      BigStepBack
      Up
      BigStepForward
      Pause

Now comment out the following lines:

You may also want to modify the section and add these:

      StepBack
      StepForward

Save the file, exit, and start Boxee. You can now control using your numeric pad.