Blog <-

Archive for the ‘security’ Category

RSS   RSS feed for this category

Script to start a Chrome browser with an SSH Socks5 proxy

Socks5 proxies are great. They allow you to tunnel all traffic for applications that support Socks proxies through the proxy. One example I frequently use is starting a Chrome window that will do everthing as if it was an a remote machine. This is especially useful to bypass firewalls so you can test websites that are only available on localhost on a remote machine, or sites that can only be accessed if you're on a remote network. Basically it's a poor-man's application-specific VPN over SSH.

Normally I run the following:

ssh -D 8000 -N &
chromium-browser --temp-profile --proxy-server="socks5://localhost:8000"

However that quickly becomes tedious to type, so I wrote a script:


if [ -z "$HOST" ]; then
    echo "Usage; $0 <HOST> [SITE]"
    exit 1

while `true`;
    PORT=$(expr 8000 + $RANDOM / 32) # random port in range 8000 - 9000
    if [ \! "$(netstat -lnt | awk '$6 == "LISTEN" && $4 ~ ".$PORT"')" ]; then
        # Port not in use
        ssh -D $PORT -N $HOST &
        chromium-browser --temp-profile --proxy-server="socks5://localhost:$PORT" $SITE
        kill $PID
        exit 0

The script finds a random unused port in the range 8000 – 9000, starts a Socks5 proxy to it and then starts Chromium on that socks proxy.

Together with the excellent Scripts panel plugin for Cinnamon, this makes for a nice menu to easily launch a browser to access remote sites otherwise unreachable:


Update: Added a second optional parameter to specify the site you want the browser to connect too, for added convience.

Can't save imported OpenVPN configuration in Network Manager

I ran into an issue where I couldn't save an imported OpenVPN (.ovpn) configuration in Network Manager. The "Save" button remains disabled:


It turns out I need to enter a password for the Private Key. Ofcourse, this particular private key doesn't have a password, but you can simply enter a single space as your password. After that the "Save" button becomes active.

POODLE: SSLv3 bug summary

Yet Another SSL bug: This time a problem with SSLv3.

Most browsers and web servers support SSLv3. Many don't use it by default; instead opting for higher versions of SSL such as TLS v1.0+. The problem is that attackers can force a downgrade of the negotiated protocol, which will result in the SSLv3 protocol being used to communicate.

No real fixes are available and vendors will probably not be sending out updates to fix this issue. The recommended method of mitigation is to disable SSLv3 on your servers and your browsers. SSLv3 is old and only the following browsers can't work with anything better:

  • Internet Explorer up to (and including) v6
  • Opera v1 t/m 4 (current version is 12)

Other browsers (Firefox, Chrome, etc) have supported TLSv1.0+ from their first release.

To test if you're vulnerable:

openssl s_client -connect HOSTNAME:443 -ssl3

If you do NOT get a message saying something like "ssl handshake failure", your server is vulnerable.

A quick test (which I do not garantuee to be correct) is:

openssl s_client -connect -ssl3 2>/dev/null | grep "Server certificate"

If this returns "Server certificate", you're vulnerable.

To fix this for Apache, edit your SSL module configuration (/etc/apache2/mods-enabled/ssl.conf on Debian-derived systems) and add "-SSLv3" to your protocols to disable SSLv3:

SSLProtocol all -SSLv2 -SSLv3

This disabled SSLv2 and 3, which are both broken. It also means users with Internet Explorer 5 or 6 won't be able to reach your secure website anymore.


Work around insufficient remote permissions when SCPing

Here's a problem I often run into:

  • I need to copy files from a remote system to my local system.
  • I have root access to the remote system via sudo or su, but not directly via SSH.
  • I don't have enough permissions to read the remote files as a normal user; I need to be root.
  • There isn't enough space to copy the files to a temp dir and change their ownership.

One solution is to use sudo tar remotely and output the tar file on stdout:

fboender@local$ ssh "sudo tar -vczf - /root/foo" > foo.tar.gz

This relies on the remote host allowing X11 forwarding though, and you have to have an SSH askpass program installed. Half of the time, I can't get this work properly.

An easier solution is to build a reverse remote tunnel:

fboender@local$ ssh -R 19999:localhost:22

This maps the remote port 19999 on to my local port 22. That means I can now access the SSH server running locally from the remote server by SSHing to port 19999. For example:$ scp -P 19999 -r /root/foo fboender@

There you go. Easy as pie.

How to REALLY test for Bash Shellshock (CVE-2014-6271)

Like always in a crisis, many things go wrong. Everyobody starts chattering, and start deteriorating the signal-to-noise level. I'll keep this brief.

There are a bunch of sites out there that are telling you how to test for the Bash Shellshock vulnerability. Many of the tests are WRONG:

$ env x=’() { ;;}; echo vulnerable’ sh -c “echo this is a test”
syntax error near unexpected token `('

Spot the first problem! First off all, this uses the wrong kind of quotes. That syntax error is NOT an indication that your system isn't vulnerable. It's an indication that the blog where you copied the instruction from doesn't understand what ASCII quotes are.

Now, spot the second problem! Which shell is this calling?? Is it bash? No, it's `sh`. So if `sh` isn't linked to bash, you get this:

$ env x='() { ;;}; echo vulnerable' sh -c “echo this is a test”
sh: x: line 0: syntax error near unexpected token `;;'
sh: x: line 0: `x () { ;;}; echo vulnerable'
sh: error importing function definition for `x'
this: “echo: command not found

"Oh, great, we're not vulnerable", you think. But it's not executing bash at all, it's executing some other shell. Sloppy work.

Here's a way to actually test your system. BUT don't take my word for it, perhaps it is not right either:

# Perhaps correct:
$ env x='() { :;}; echo vulnerable' bash -c 'echo hello'



16 things you should absolutely configure on any new server

It seems even professional sysadmins occasionally forgets the bare minimum configuration that should be done on a new machine. As a developer and part-time system administrator, I can't count the number of times I've had to waste significantly more time Here's a, by no means exhaustive, list of things you should configure on any new machine you deploy.

1. Pick a good hostname

Set a sane hostname on your machine. Something that describes what the machine is or does. Something that uniquely identifies it from any other machines on, at least, the same network. For instance, machine for a client called Megacorp might be called "mc-tst-www-1" to identify the first test WWW server for Megacorp. The primary production loadbalancer might be called "mc-prod-lb-1". Never have your junior sysadmin bring down the master database backend because he thought he was on a different machine.

2. Put all hostnames in /etc/hosts

Put all hostnames your machine uses in the /etc/hosts file to avoid annoying DNS lookup delays and other problems.

3. Install ntpd

Running into problems related to clock drift on your server is not a matter of "if", but a matter of "when". And with clock drift it will be sooner rather than later, depending on which direction your clock is drifting in. Install NTPd, and synchronize it to the same servers as all your other machines. Don't use a default pool if you can avoid it, because they might use Round Robin DNS and give you different servers. Theoretically this shouldn't pose a problem. Theoretically…

If you're running virtual machines, turn off Virtualbox/VMWare/whatever's time synchronization. They've historically been proven to be very unreliable[1]. Install ntpd anyway. And I swear, as a developer, I will kick you in the face if I ever have to diagnose another problem caused by a lack of ntpd.

4. Make sure email can be delivered

This one is simple. Make sure email can be delivered to the outside world. Many programs and scripts will need to be able to send email. Make sure they can. Ideally, you should have a dedicated SMTP server set up on your network that hosts can relay email through. A gateway firewall should prevent all other outgoing traffic for port 25, unless you want your server to be turned into a zombified spam node (which will happen).

5. Cron email

Configure Cron such that output is emailed to an actual person. You want to know about that "No space left on device" error that crashed your cobbled-together backups script. You can specify the email address with the MAILTO directive in the crontab file. Don't forget about user crontabs! Since it's hard to ensure every user crontab has a MAILTO setting, you may want to configure your SMTP server to automatically forward all email to a special email address.

6. Protect the SSH port

Unauthorized probing of the SSH port will happen, unless you prevent it. Weak passwords can be easily guessed in a few hundred tries. Timing attacks can be used to guess which accounts live on the system, even if the attacker can't guess the password. There are several options for securing your SSH port

  • Listen on a different port. This is the least secure option, as it can usually be easily probed using a port scanner. It will fool some of the botnets out in the wild blindly scanning on port 22, but it won't keep out the more advanced attackers. If you go for this option, don't go for port 2222, but pick something arbitrary high, such as 58245.
  • Install Fail2ban. It scans your logs and blocks any IPs that show malicious signs. This is a good idea, regardless of whether you want to secure SSH or not
  • Firewall off the port completely. Only open access from a few select IPs, such as your management network. Use a port knocker to open SSH ports on demand in case you absolutely need access from unpredictable IPs.

7. Configure a firewall

This should go without saying.. install and configure a firewall. Firewall everything. Incoming traffic, outgoing traffic, all of it. Only open up what you need to open. Don't rely on your gateway's firewall to do its job; you will regret it when other machines on your network get compromised.

8. Monitor your system

Monitor your system, even if it's just a simple shell script that emails you about problems. Disks will fill up, services will mysteriously shut down and your CPU load will go to 300. I highly recommend also monitoring important services from a remote location.

9. Configure resource usage

Running Apache, a database or some Java stack? Configure it properly so it utilizes the resources your system has, but doesn't overload it. Configure the minimum and maximum connections Apache will accept, tune the memory your database is allowed to use, etc.

10. Keep your software up-to-date

Install something like apt-dater to keep your software up-to-date. Many server compromises are directly linked to outdated software. Don't trust yourself to keep a machine up to date. You will forget. If you're running third-party software not installed from your package repository, subscribe to their security announcement mailing list and keep a list of all third-party software installed on every server. A tool such as Puppet, Chef or Ansible can help keep your system not only up to date, but uniform.

11. Log rotation

Make sure all logs are automatically rotated, or your disks will fill up. Take a look at /etc/logrotate.d/ to see how. For instance, for Apache vhosts that each have their own log directory, you can add an entry such as:

/var/www/*/logs/*.log {
        rotate 52
        # create 640 root adm # Disabled so old logfile's properties are used.
                if [ -f /var/run/ ]; then
                        /etc/init.d/apache2 restart > /dev/null

12. Prevent users from adding SSH keys

Remove the ability for users to add new authorized keys to their account. Which keys are allowed to connect should be in the admin's hand, not the users. Having the Authorized Keys files scattered all over your system also makes maintenance harder. To do this, change the AuthorizedKeysFile setting in /etc/ssh/sshd_config:

#AuthorizedKeysFile     %h/.ssh/authorized_keys
AuthorizedKeysFile      /etc/ssh/authorized_keys/%u

13. Limit user crontabs

Limit which users can create personal crontab entries by placing only allowed usernames in /etc/cron.allow. This prevents users from creating CPU/IO heavy cronjobs that interfere with your nightly backups.

14. Backups, backups and more backups

Make backups! Keep local backups for easy restoring of corrupt files, databases and other disasters. Databases should be backed up locally each night, if at all possible. Rotate backups on a daily, weekly and monthly cycle. Keep off-site backups too. For small servers I can highly recommend Boxbackup. It keeps remote encrypted backups, does full and incremental backups, keeps a history and does snapshotting as well as continues syncing. Only delta's (changes in files) are transferred and stored, so it is light on resources. wrote an article on setting it up which might prove useful.

15. Install basic tools

Make sure basic tools for daily admin tasks are pre-installed. There's nothing more annoying than having to track down problems and not having the means to do so, especially when your network refuses to come up. Some essential tools:

  • vi
  • iotop
  • strace
  • Whatever more you need..

16. Install fail2ban

I've already mentioned this in the "Protect your SSH port", but it bears mentioning again: install Fail2ban to automatically block offending IPs.


That's it. These are the things I would consider the bare minimum that should be properly configured when you deploy a new machine. It will take a little bit more time up front to configure machines properly, but it will save you time in the end. I can highly recommend using Puppet, Chef or Ansible to help you automate these tasks.


[1]This was the case a few years ago. I'm not sure it still is for VMWare. For VirtualBox, it most certainly is, but you wouldn't run that in a production environment probably. At the very least, install NTPd on your host.

Re-use existing SSH agent (cygwin et al)

(Please note that this post is not specific to Windows nor Cygwin; it'll work on a remote unix machine just as well)

On my netbook, I use Windows XP in combination with Cygwin (A unix environment for Windows) and Mintty for my Unixy needs. From there, I usually SSH to some unix-like machine somewhere, so I can do systems administration or development.

Unfortunately, the default use of an SSH agent under Cygwin is difficult, since there's no parent process that can run it and put the required information (SSH_AUTH_SOCK) in the environment. On most Linux distribution, the SSH agent is started after you log in to an X11 session, so that every child process (terminals you open, etc) inherits the SSH_AUTH_SOCK environment setting and SSH can contact the ssh-agent to get your keys. Result? You have to start a new SSH agent, load your key and enter your password for each Mintty terminal you open. Quite annoying.

The upside is, it's not very hard to configure your system properly so that you need only one SSH agent running on your system, and thus only have to enter your password once.

The key lies in how ssh-agent creates the environment. When we start ssh-agent in the traditional manner, we do:

$ eval `ssh-agent`
Agent pid 1784

The command starts the SSH agent and sets a bunch of environment variables:

$ set | grep SSH_

The SSH_AUTH_SOCK is how the ssh command knows how to contact the agent. As you can see, the socket filename is generated randomly. That means you can't reuse the socket, since you can't guess the socket filename.

Good thing ssh-agent allows us to specify the socket filename, so we can easily re-use it.

Put the following in your ~/.bashrc:

# If no SSH agent is already running, start one now. Re-use sockets so we never
# have to start more than one session.

export SSH_AUTH_SOCK=/home/fboender/.ssh-socket

ssh-add -l >/dev/null 2>&1
if [ $? = 2 ]; then
   # No ssh-agent running
   rm -rf $SSH_AUTH_SOCK
   # >| allows output redirection to over-write files if no clobber is set
   ssh-agent -a $SSH_AUTH_SOCK >| /tmp/.ssh-script
   source /tmp/.ssh-script
   echo $SSH_AGENT_PID >| ~/.ssh-agent-pid
   rm /tmp/.ssh-script

What the script above does is, it sets the socket filename manually to /home/yourusername/.ssh-socket. It then runs ssh-add, which will attempt to connect to the ssh-agent through the socket. If it fails, it means no ssh-agent is running, so we do some cleanup and start one.

Now, all you have to do is start a single terminal, and load your keys once:

$ ssh-add ~/.ssh/fboender\@electricmonk.rsa
Enter passphrase for .ssh/fboender@electricmonk.rsa: [PASSWORD]
Identity added: .ssh/fboender@electricmonk.rsa (.ssh/fboender@electricmonk.rsa)

Now you can start as many new terminals as you'd like, and they'll all use the same ssh-agent, never requiring you to enter your password for that key more than once per boot.


I've updated the script with suggestions from Anthony Geoghegan. It now also works if noclobber is set.

Stop Pingback/Trackback Spam on WordPress

I guess the spammers finally found my blog, cause I've been getting a lot of pignback/trackback spam. I tried some anti-spam plugins, but none really worked, so I disabled pingbacks altogether. Here's how:

First, log into wordpress as an admin. Go to Settings → Discussion, and uncheck the Allow link notifications from other blogs (pingbacks and trackbacks.) box.

That's not all though, cause that just works for new articles. Old ones will still allow ping/trackbacks. As far as I could tell, WordPress doesn't have a feature to disable those for older posts, so we'll have to fiddle around in the database directly.

Connect to your MySQL server using the WordPress username and password. If you no longer remember those, you can find them in the wp-config.php file.

$ mysql -u USERNAME -p -h localhost
Password: PASSWORD
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1684228
Server version: 5.0.51a-24+lenny5 (Debian)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.


At the MySQL prompt, we must now switch to our WordPress database. Again, if you don't remember the name, you can find it in the wp-config.php file. In my case, it is em_wordpress

mysql> USE em_wordpress;
Database changed

Finally, we update all posts and disable ping backs on them:

mysql> UPDATE wp_posts SET ping_status = 'closed';
Query OK, 1084 rows affected (0.10 sec)
Rows matched: 1084  Changed: 1084  Warnings: 0

There we go. No more ping- and trackback spam on old posts.

SSH Tips and Tricks

(The lastest version of this article is always available in stand-alone HTML format and in PDF format. The original AsciiDoc source is also available. Please link to the HTML version, not this Blog post!)

SSH is capable of more than you'd think! This article describes some of the lesser known features and configuration options. It covers authentication, authorization, tunnels and proxies, file transfer and more.

Read the rest of this entry »

Regular expression Denial of Service (ReDoS)

It's only logical, but I hadn't really thought about it much. Turns out Regular Expression can be vulnerable to external Denial of Service attacks.