contact
----------------------------

Blog <-

Archive for the ‘security’ Category

RSS   RSS feed for this category

POODLE: SSLv3 bug summary

Yet Another SSL bug: This time a problem with SSLv3.

Most browsers and web servers support SSLv3. Many don't use it by default; instead opting for higher versions of SSL such as TLS v1.0+. The problem is that attackers can force a downgrade of the negotiated protocol, which will result in the SSLv3 protocol being used to communicate.

No real fixes are available and vendors will probably not be sending out updates to fix this issue. The recommended method of mitigation is to disable SSLv3 on your servers and your browsers. SSLv3 is old and only the following browsers can't work with anything better:

  • Internet Explorer up to (and including) v6
  • Opera v1 t/m 4 (current version is 12)

Other browsers (Firefox, Chrome, etc) have supported TLSv1.0+ from their first release.

To test if you're vulnerable:

openssl s_client -connect HOSTNAME:443 -ssl3

If you do NOT get a message saying something like "ssl handshake failure", your server is vulnerable.

A quick test (which I do not garantuee to be correct) is:

openssl s_client -connect 127.0.0.1:443 -ssl3 2>/dev/null | grep "Server certificate"

If this returns "Server certificate", you're vulnerable.

To fix this for Apache, edit your SSL module configuration (/etc/apache2/mods-enabled/ssl.conf on Debian-derived systems) and add "-SSLv3" to your protocols to disable SSLv3:

SSLProtocol all -SSLv2 -SSLv3

This disabled SSLv2 and 3, which are both broken. It also means users with Internet Explorer 5 or 6 won't be able to reach your secure website anymore.

 

Work around insufficient remote permissions when SCPing

Here's a problem I often run into:

  • I need to copy files from a remote system to my local system.
  • I have root access to the remote system via sudo or su, but not directly via SSH.
  • I don't have enough permissions to read the remote files as a normal user; I need to be root.
  • There isn't enough space to copy the files to a temp dir and change their ownership.

One solution is to use sudo tar remotely and output the tar file on stdout:

fboender@local$ ssh fboender@example.com "sudo tar -vczf - /root/foo" > foo.tar.gz

This relies on the remote host allowing X11 forwarding though, and you have to have an SSH askpass program installed. Half of the time, I can't get this work properly.

An easier solution is to build a reverse remote tunnel:

fboender@local$ ssh -R 19999:localhost:22 fboender@example.com

This maps the remote port 19999 on example.com to my local port 22. That means I can now access the SSH server running locally from the remote server by SSHing to port 19999. For example:

fboender@example.com$ scp -P 19999 -r /root/foo fboender@127.0.0.1
Password: 

There you go. Easy as pie.

How to REALLY test for Bash Shellshock (CVE-2014-6271)

Like always in a crisis, many things go wrong. Everyobody starts chattering, and start deteriorating the signal-to-noise level. I'll keep this brief.

There are a bunch of sites out there that are telling you how to test for the Bash Shellshock vulnerability. Many of the tests are WRONG:

# WROOOOOOOOOOOOOOOOONG
$ env x=’() { ;;}; echo vulnerable’ sh -c “echo this is a test”
syntax error near unexpected token `('

Spot the first problem! First off all, this uses the wrong kind of quotes. That syntax error is NOT an indication that your system isn't vulnerable. It's an indication that the blog where you copied the instruction from doesn't understand what ASCII quotes are.

Now, spot the second problem! Which shell is this calling?? Is it bash? No, it's `sh`. So if `sh` isn't linked to bash, you get this:

# WROOOOOOOOOOOOOOOOOOOOOOOOOOOOOOONG
$ env x='() { ;;}; echo vulnerable' sh -c “echo this is a test”
sh: x: line 0: syntax error near unexpected token `;;'
sh: x: line 0: `x () { ;;}; echo vulnerable'
sh: error importing function definition for `x'
this: “echo: command not found

"Oh, great, we're not vulnerable", you think. But it's not executing bash at all, it's executing some other shell. Sloppy work.

Here's a way to actually test your system. BUT don't take my word for it, perhaps it is not right either:

# Perhaps correct:
$ env x='() { :;}; echo vulnerable' bash -c 'echo hello'
vulnerable
hello

 

 

16 things you should absolutely configure on any new server

It seems even professional sysadmins occasionally forgets the bare minimum configuration that should be done on a new machine. As a developer and part-time system administrator, I can't count the number of times I've had to waste significantly more time Here's a, by no means exhaustive, list of things you should configure on any new machine you deploy.

1. Pick a good hostname

Set a sane hostname on your machine. Something that describes what the machine is or does. Something that uniquely identifies it from any other machines on, at least, the same network. For instance, machine for a client called Megacorp might be called "mc-tst-www-1" to identify the first test WWW server for Megacorp. The primary production loadbalancer might be called "mc-prod-lb-1". Never have your junior sysadmin bring down the master database backend because he thought he was on a different machine.

2. Put all hostnames in /etc/hosts

Put all hostnames your machine uses in the /etc/hosts file to avoid annoying DNS lookup delays and other problems.

3. Install ntpd

Running into problems related to clock drift on your server is not a matter of "if", but a matter of "when". And with clock drift it will be sooner rather than later, depending on which direction your clock is drifting in. Install NTPd, and synchronize it to the same servers as all your other machines. Don't use a default pool if you can avoid it, because they might use Round Robin DNS and give you different servers. Theoretically this shouldn't pose a problem. Theoretically…

If you're running virtual machines, turn off Virtualbox/VMWare/whatever's time synchronization. They've historically been proven to be very unreliable[1]. Install ntpd anyway. And I swear, as a developer, I will kick you in the face if I ever have to diagnose another problem caused by a lack of ntpd.

4. Make sure email can be delivered

This one is simple. Make sure email can be delivered to the outside world. Many programs and scripts will need to be able to send email. Make sure they can. Ideally, you should have a dedicated SMTP server set up on your network that hosts can relay email through. A gateway firewall should prevent all other outgoing traffic for port 25, unless you want your server to be turned into a zombified spam node (which will happen).

5. Cron email

Configure Cron such that output is emailed to an actual person. You want to know about that "No space left on device" error that crashed your cobbled-together backups script. You can specify the email address with the MAILTO directive in the crontab file. Don't forget about user crontabs! Since it's hard to ensure every user crontab has a MAILTO setting, you may want to configure your SMTP server to automatically forward all email to a special email address.

6. Protect the SSH port

Unauthorized probing of the SSH port will happen, unless you prevent it. Weak passwords can be easily guessed in a few hundred tries. Timing attacks can be used to guess which accounts live on the system, even if the attacker can't guess the password. There are several options for securing your SSH port

  • Listen on a different port. This is the least secure option, as it can usually be easily probed using a port scanner. It will fool some of the botnets out in the wild blindly scanning on port 22, but it won't keep out the more advanced attackers. If you go for this option, don't go for port 2222, but pick something arbitrary high, such as 58245.
  • Install Fail2ban. It scans your logs and blocks any IPs that show malicious signs. This is a good idea, regardless of whether you want to secure SSH or not
  • Firewall off the port completely. Only open access from a few select IPs, such as your management network. Use a port knocker to open SSH ports on demand in case you absolutely need access from unpredictable IPs.

7. Configure a firewall

This should go without saying.. install and configure a firewall. Firewall everything. Incoming traffic, outgoing traffic, all of it. Only open up what you need to open. Don't rely on your gateway's firewall to do its job; you will regret it when other machines on your network get compromised.

8. Monitor your system

Monitor your system, even if it's just a simple shell script that emails you about problems. Disks will fill up, services will mysteriously shut down and your CPU load will go to 300. I highly recommend also monitoring important services from a remote location.

9. Configure resource usage

Running Apache, a database or some Java stack? Configure it properly so it utilizes the resources your system has, but doesn't overload it. Configure the minimum and maximum connections Apache will accept, tune the memory your database is allowed to use, etc.

10. Keep your software up-to-date

Install something like apt-dater to keep your software up-to-date. Many server compromises are directly linked to outdated software. Don't trust yourself to keep a machine up to date. You will forget. If you're running third-party software not installed from your package repository, subscribe to their security announcement mailing list and keep a list of all third-party software installed on every server. A tool such as Puppet, Chef or Ansible can help keep your system not only up to date, but uniform.

11. Log rotation

Make sure all logs are automatically rotated, or your disks will fill up. Take a look at /etc/logrotate.d/ to see how. For instance, for Apache vhosts that each have their own log directory, you can add an entry such as:

/var/www/*/logs/*.log {
        weekly
        missingok
        rotate 52
        compress
        delaycompress
        notifempty
        # create 640 root adm # Disabled so old logfile's properties are used.
        sharedscripts
        postrotate
                if [ -f /var/run/apache2.pid ]; then
                        /etc/init.d/apache2 restart > /dev/null
                fi
        endscript
}

12. Prevent users from adding SSH keys

Remove the ability for users to add new authorized keys to their account. Which keys are allowed to connect should be in the admin's hand, not the users. Having the Authorized Keys files scattered all over your system also makes maintenance harder. To do this, change the AuthorizedKeysFile setting in /etc/ssh/sshd_config:

#AuthorizedKeysFile     %h/.ssh/authorized_keys
AuthorizedKeysFile      /etc/ssh/authorized_keys/%u

13. Limit user crontabs

Limit which users can create personal crontab entries by placing only allowed usernames in /etc/cron.allow. This prevents users from creating CPU/IO heavy cronjobs that interfere with your nightly backups.

14. Backups, backups and more backups

Make backups! Keep local backups for easy restoring of corrupt files, databases and other disasters. Databases should be backed up locally each night, if at all possible. Rotate backups on a daily, weekly and monthly cycle. Keep off-site backups too. For small servers I can highly recommend Boxbackup. It keeps remote encrypted backups, does full and incremental backups, keeps a history and does snapshotting as well as continues syncing. Only delta's (changes in files) are transferred and stored, so it is light on resources. wrote an article on setting it up which might prove useful.

15. Install basic tools

Make sure basic tools for daily admin tasks are pre-installed. There's nothing more annoying than having to track down problems and not having the means to do so, especially when your network refuses to come up. Some essential tools:

  • vi
  • iotop
  • strace
  • Whatever more you need..

16. Install fail2ban

I've already mentioned this in the "Protect your SSH port", but it bears mentioning again: install Fail2ban to automatically block offending IPs.

Conclusion

That's it. These are the things I would consider the bare minimum that should be properly configured when you deploy a new machine. It will take a little bit more time up front to configure machines properly, but it will save you time in the end. I can highly recommend using Puppet, Chef or Ansible to help you automate these tasks.

Notes

[1]This was the case a few years ago. I'm not sure it still is for VMWare. For VirtualBox, it most certainly is, but you wouldn't run that in a production environment probably. At the very least, install NTPd on your host.

Re-use existing SSH agent (cygwin et al)

(Please note that this post is not specific to Windows nor Cygwin; it'll work on a remote unix machine just as well)

On my netbook, I use Windows XP in combination with Cygwin (A unix environment for Windows) and Mintty for my Unixy needs. From there, I usually SSH to some unix-like machine somewhere, so I can do systems administration or development.

Unfortunately, the default use of an SSH agent under Cygwin is difficult, since there's no parent process that can run it and put the required information (SSH_AUTH_SOCK) in the environment. On most Linux distribution, the SSH agent is started after you log in to an X11 session, so that every child process (terminals you open, etc) inherits the SSH_AUTH_SOCK environment setting and SSH can contact the ssh-agent to get your keys. Result? You have to start a new SSH agent, load your key and enter your password for each Mintty terminal you open. Quite annoying.

The upside is, it's not very hard to configure your system properly so that you need only one SSH agent running on your system, and thus only have to enter your password once.

The key lies in how ssh-agent creates the environment. When we start ssh-agent in the traditional manner, we do:

$ eval `ssh-agent`
Agent pid 1784

The command starts the SSH agent and sets a bunch of environment variables:

$ set | grep SSH_
SSH_AGENT_PID=1784
SSH_AUTH_SOCK=/tmp/ssh-QzfPveH696/agent.696

The SSH_AUTH_SOCK is how the ssh command knows how to contact the agent. As you can see, the socket filename is generated randomly. That means you can't reuse the socket, since you can't guess the socket filename.

Good thing ssh-agent allows us to specify the socket filename, so we can easily re-use it.

Put the following in your ~/.bashrc:

# If no SSH agent is already running, start one now. Re-use sockets so we never
# have to start more than one session.

export SSH_AUTH_SOCK=/home/fboender/.ssh-socket

ssh-add -l >/dev/null 2>&1
if [ $? = 2 ]; then
   # No ssh-agent running
   rm -rf $SSH_AUTH_SOCK
   # >| allows output redirection to over-write files if no clobber is set
   ssh-agent -a $SSH_AUTH_SOCK >| /tmp/.ssh-script
   source /tmp/.ssh-script
   echo $SSH_AGENT_PID >| ~/.ssh-agent-pid
   rm /tmp/.ssh-script
fi

What the script above does is, it sets the socket filename manually to /home/yourusername/.ssh-socket. It then runs ssh-add, which will attempt to connect to the ssh-agent through the socket. If it fails, it means no ssh-agent is running, so we do some cleanup and start one.

Now, all you have to do is start a single terminal, and load your keys once:

$ ssh-add ~/.ssh/fboender\@electricmonk.rsa
Enter passphrase for .ssh/fboender@electricmonk.rsa: [PASSWORD]
Identity added: .ssh/fboender@electricmonk.rsa (.ssh/fboender@electricmonk.rsa)

Now you can start as many new terminals as you'd like, and they'll all use the same ssh-agent, never requiring you to enter your password for that key more than once per boot.

Update:

I've updated the script with suggestions from Anthony Geoghegan. It now also works if noclobber is set.

Stop Pingback/Trackback Spam on WordPress

I guess the spammers finally found my blog, cause I've been getting a lot of pignback/trackback spam. I tried some anti-spam plugins, but none really worked, so I disabled pingbacks altogether. Here's how:

First, log into wordpress as an admin. Go to Settings → Discussion, and uncheck the Allow link notifications from other blogs (pingbacks and trackbacks.) box.

That's not all though, cause that just works for new articles. Old ones will still allow ping/trackbacks. As far as I could tell, WordPress doesn't have a feature to disable those for older posts, so we'll have to fiddle around in the database directly.

Connect to your MySQL server using the WordPress username and password. If you no longer remember those, you can find them in the wp-config.php file.

$ mysql -u USERNAME -p -h localhost
Password: PASSWORD
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1684228
Server version: 5.0.51a-24+lenny5 (Debian)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql>

At the MySQL prompt, we must now switch to our WordPress database. Again, if you don't remember the name, you can find it in the wp-config.php file. In my case, it is em_wordpress

mysql> USE em_wordpress;
Database changed

Finally, we update all posts and disable ping backs on them:

mysql> UPDATE wp_posts SET ping_status = 'closed';
Query OK, 1084 rows affected (0.10 sec)
Rows matched: 1084  Changed: 1084  Warnings: 0

There we go. No more ping- and trackback spam on old posts.

SSH Tips and Tricks

(The lastest version of this article is always available in stand-alone HTML format and in PDF format. The original AsciiDoc source is also available. Please link to the HTML version, not this Blog post!)

SSH is capable of more than you'd think! This article describes some of the lesser known features and configuration options. It covers authentication, authorization, tunnels and proxies, file transfer and more.

Read the rest of this entry »

Regular expression Denial of Service (ReDoS)

It's only logical, but I hadn't really thought about it much. Turns out Regular Expression can be vulnerable to external Denial of Service attacks.

Security Questions considered harmful

Many online services allow, or even worse, require, the so called "Security Question". It is a question/answer you can enter in case you ever forget your password or can't access your account for some reason. In my opinion, security questions are an incredibly bad idea, from a security perspective.

The usual security questions are things like "What was your mother's maiden name", "What's your pet's name", etcetera. People won't understand that actually supplying a truthful answer to these kind of questions exposes them to an incredible weakness in their account's security. These are all questions to which the answer can be found relatively easy by googling a person or applying a little social engineering. "Hey, I am John, and I think I might be related to you on your mother's side. What's her maiden name"?

The worst part is that every site has basically the same questions from which you can choose. This means that people either have to pick the same question and answer every time, or pick a different one for each account. The first will make them vulnerable to repeated attacks on all their online profiles once an attacker has found the answer. The second will make it very hard for people to remember that they must never let anybody know about their favorite pet's name "Buddy". A lose/lose scenario at best.

As is often the case with security protocols, they must be followed to the letter to be safe. One flaw in the procedure, and the security collapses. Security questions could be a good idea, provided that:

  • The user makes up his own question. No predefined questions should be supplied, and most importantly, different sites shouldn't all use the same questions.
  • The user should never be told what his security question was. If they need to reset their password, they should chose both the security question and the answer. This will make it much harder for a potential attacker to gain accees.

Of course, taking the above in consideration, security questions are just as hard to remember as a password, which makes them kind of pointless. Pointless or insecure, make your pick.

chkrootkit false positives filtering

Chkrootkit is a tool that searches for rootkits, trojans and other signs of break-ins on your system. Like most security scanners, it sometimes generates false positives. Chkrootkit doesn't have a native way to filter those out. From the FAQ:

[Q:] chkrootkit is reporting some files and dirs as suspicious: `.packlist', `.cvsignore', etc. These are clearly false positives. Can't you ignore these?

[A:] Ignoring some files and dirs could impair chkrootkit's accuracy. An attacker might use this, since he knows that chkrootkit will ignore certain files and dirs.

This is true, but getting an email every day is simply too annoying, and makes me skip chkrootkit generated emails on occasion because "It's probably a false positive anyway". So here's a small guide for setting up a filtering of chkrootkit's output.

First, we create a file /etc/chkrootkit.ignore which will hold a bunch of regular expressions that will match everything we don't want to be warned about. For instance, I've got a machine that needs to have a dhcp client installed. Chkrootkit keeps on generating emails with these lines:

eth0: PACKET SNIFFER(/sbin/dhclient[346])
eth1: PACKET SNIFFER(/usr/sbin/dhcpd3[1008])

So what we do is create the file /etc/chkrootkit.ignore and put the following in it:

/etc/chkrootkit.ignore

^eth0: PACKET SNIFFER\(/sbin/dhclient\[[0-9]*\])$
^eth1: PACKET SNIFFER\(/usr/sbin/dhcpd3\[[0-9]*\]\)$

In order to test if the rules we created are correct, we put the two lines with false positives in a separate file (/tmp/chkrk-fp.txt) and run the following:

test:

[root@sharky]/etc# cat /tmp/chkrk-fp.txt | grep -f /etc/chkrootkit.ignore
eth0: PACKET SNIFFER(/sbin/dhclient[346])
eth1: PACKET SNIFFER(/usr/sbin/dhcpd3[1008])

The lines that should be filtered out of the chkrootkit output should appear here. If nothing appears, or if not all of the lines that you want to filter appear, there's a problem. Refine your regular expressions in /etc/chkrootkit.filter until it works.

Now we need to modify the chkrootkit cronjob so that the false positives are filtered. To do this, we edit /etc/cron.daily/chkrootkit. Below is a patch that shows what should be changed. You can apply the patch with the 'patch' command, or you can manually add the lines that start with a '+', replacing the lines with a '-'.

--- /home/root/foo      2007-11-21 11:53:58.532769984 +0100
+++ /etc/cron.daily/chkrootkit  2007-11-21 11:54:00.689442120 +0100
@@ -1,27 +1,28 @@
 #!/bin/sh -e

 CHKROOTKIT=/usr/sbin/chkrootkit
 CF=/etc/chkrootkit.conf
+IGNOREF=/etc/chkrootkit.ignore
 LOG_DIR=/var/cache/chkrootkit

 if [ ! -x $CHKROOTKIT ]; then
   exit 0
 fi

 if [ -f $CF ]; then
     . $CF
 fi

 if [ "$RUN_DAILY" = "true" ]; then
     if [ "$DIFF_MODE" = "true" ]; then
-        $CHKROOTKIT $RUN_DAILY_OPTS > $LOG_DIR/log.new 2>&1
+        $CHKROOTKIT $RUN_DAILY_OPTS | grep -v -f $IGNOREF > $LOG_DIR/log.new 2>&1 || true
         if [ ! -f $LOG_DIR/log.old ] \
            || ! diff -q $LOG_DIR/log.old $LOG_DIR/log.new > /dev/null 2>&1; then
             cat $LOG_DIR/log.new
         fi
         mv $LOG_DIR/log.new $LOG_DIR/log.old
     else
-        $CHKROOTKIT $RUN_DAILY_OPTS
+        $CHKROOTKIT $RUN_DAILY_OPTS | grep -v -f $IGNOREF || true
     fi
 fi

Next, we try running chkrootkit, to see if anything shows up:

[root@sharky]/etc/cron.daily# ./chkrootkit
[root@sharky]/etc/cron.daily#

There is no output, so our false positives are now being ignored.