contact
----------------------------

Blog <-

Archive for the ‘sysadmin’ Category

RSS   RSS feed for this category

Keep an archive of all mails going through a Postfix SMTP server

OB-PZ529_junkma_G_20111006070316

All too often I get asked questions about emails which I can't answer because I can't actually see the emails. While mail logging goes a long way, I'd really like to keep an archive of all mails sent via an SMTP server on that machine. Here's how to do that with Postfix. This was tested on Ubuntu 14.04, but should be applicable to other foonixes without too much trouble. Run all this as the root user.

Add a user to the system so postfix can send BCC's of all emails to it

adduser --system --home /var/archive/mail/ --no-create-home --disabled-password mailarchive

Next, create the Mailbox layout for the mail archive:

mkdir -p /var/archive/mail/tmp
mkdir -p /var/archive/mail/cur
mkdir -p /var/archive/mail/new
chown -R nobody:nogroup /var/archive

Configure Postfix to always send a copy of any emails sent to the mailarchive user:

postconf -e always_bcc=mailarchive@localhost

Configure the mail storage for the mailacrhive user so it uses the Mailbox format. This makes it easier to delete old emails:

# echo "mailarchive: /var/archive/mail/" >> /etc/aliases
# newaliases

Finally, restart postfix

/etc/init.d/postfix restart

Now to test it send an email through the SMTP server. I'll do a manual SMTP session here:

telnet localhost 25
HELO localhost
MAIL FROM: fboender@localhost
RCPT TO: ferry.boender@example.com
DATA
Subject: Mail test
Here's some mail.
.
QUIT

We check the /var/archive/mail/new directory:

ls /var/archive/mail/cur/
1425653888.Vfc00I461477M767603.posttest4:2,

And there is our mail.

To easily view the mail in the archive, install mutt:

apt-get install mutt
mutt -f /var/archive/mail/

You should probably write a cronjob that regularly cleans out the old mail, otherwise your filesystem will fill up. The following cronjob will delete all mail older than 30 days

cat /etc/cron.daily/mailarchive_clean
#!/bin/sh
find /var/archive/mail/ -type f -mtime +30 -exec rm "{}" \;
chmod 755 /etc/cron.daily/mailarchive_clean

Good luck!

Edit: Changed postfix configuration addition to postconf -e

Script to start a Chrome browser with an SSH Socks5 proxy

Socks5 proxies are great. They allow you to tunnel all traffic for applications that support Socks proxies through the proxy. One example I frequently use is starting a Chrome window that will do everthing as if it was an a remote machine. This is especially useful to bypass firewalls so you can test websites that are only available on localhost on a remote machine, or sites that can only be accessed if you're on a remote network. Basically it's a poor-man's application-specific VPN over SSH.

Normally I run the following:

ssh -D 8000 -N remote.example.com &
chromium-browser --temp-profile --proxy-server="socks5://localhost:8000"

However that quickly becomes tedious to type, so I wrote a script:

#!/bin/bash
HOST=$1
SITE=$2

if [ -z "$HOST" ]; then
    echo "Usage; $0 <HOST> [SITE]"
    exit 1
fi

while `true`;
do
    PORT=$(expr 8000 + $RANDOM / 32) # random port in range 8000 - 9000
    if [ \! "$(netstat -lnt | awk '$6 == "LISTEN" && $4 ~ ".$PORT"')" ]; then
        # Port not in use
        ssh -D $PORT -N $HOST &
        PID=$!
        chromium-browser --temp-profile --proxy-server="socks5://localhost:$PORT" $SITE
        kill $PID
        exit 0
    fi
done

The script finds a random unused port in the range 8000 – 9000, starts a Socks5 proxy to it and then starts Chromium on that socks proxy.

Together with the excellent Scripts panel plugin for Cinnamon, this makes for a nice menu to easily launch a browser to access remote sites otherwise unreachable:

lm_panel_scripts

Update: Added a second optional parameter to specify the site you want the browser to connect too, for added convience.

BBCloner v1.4: Bitbucket backup tool

I've released v1.4 of BBCloner.

BBCloner (Bitbucket Cloner) creates mirrors of your public and private Bitbucket Git repositories. It also synchronizes already existing mirrors. Initial mirror setup requires you manually enter your username/password. Subsequent synchronization of mirrors is done using Deployment Keys.

This release features a new flag: –tolerant (-t). It prevents bbcloner from complaining about failed repository synchronisation if a repository fails the first time. Only on the second failure to synchronize the same repository does bbcloner send out an email when using the -t switch. This should save on a lot of unwarranted emails, given that Bitbucket quite regularly seems to bork while syncing repositories.

Get the new release from the Downloads page or directly:

Host inventory overview using Ansible's Facts

Ansible is a multiplexing configuration orchestrator. It allows you to run commands and configure servers from a central system. For example, to run the uname -a command on servers in the group "intranet":

[fboender@jib]~/Projects/ansible$ ansible -m shell -a "uname -a" intranet
host001.example.com | success | rc=0 >>
Linux host001.example.com 2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux

host004.example.com | success | rc=0 >>
Linux vps004c.example.com 2.6.32-55-server #117-Ubuntu SMP Tue Dec 3 17:45:11 UTC 2013 x86_64 GNU/Linux

Ansible can also gather system information using the 'setup' module. It returns the information as a JSON structure:

[fboender@jib]~/Projects/ansible$ ansible -m setup intranet
host001.example.com | success >> {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
            "182.78.44.33", 
            "10.0.0.1"
        ], 
        "ansible_architecture": "x86_64", 
        "ansible_bios_date": "NA", 
        "ansible_bios_version": "NA", 
        ... etc

We can use this to display a short tabulated overview of important system information such as the FQDN, configured IPs, Disk and memory information. I wrote a quick script to do this. The result looks like this:

[fboender@jib]~/Projects/ansible$ ./hostinfo intranet
Name                     FQDN                     Datetime                    OS            Arch    Mem              Disk                 Diskfree          IPs
-----------------------  -----------------------  --------------------------  ------------  ------  ---------------  -------------------  ----------------  -------------------------------------------------------------------------
host001                  host001.example.com      2015-01-20 14:37 CET +0100  Ubuntu 12.04  x86_64  4g (free 0.16g)  80g                  40g               182.78.44.33, 10.0.0.1
host002                  host002.example.com      2015-01-20 14:37 CET +0100  Ubuntu 14.04  x86_64  2g (free 1.21g)  40g                  18g               182.78.44.34, 10.0.0.2
xxxxxx.xxxxxx.xx         xxxxxxx.example.com      2015-01-20 13:37 CET +0000  Ubuntu 10.04  x86_64  2g (free 0.04g)  241g                 20g               192.168.0.2, 10.000.0.4
xxxx.xxxxxx.xxx          xxxx.otherdom.com        2015-01-20 14:37 CET +0100  Ubuntu 13.04  x86_64  8g (free 0.14g)  292g, 1877g, 1877g   237g, 583g, 785g  192.168.1.9, 192.168.1.10, 10.0.0.6
xxxxxxxx.xxxxxx.xx       xxxx.otherdom.com        2015-01-20 14:36 CET +0100  Ubuntu 14.04  i386    6g (free 0.25g)  1860g, 1877g, 1877g  960g, 292g, 360g  10.0.0.5, 10.0.0.14, 192.168.1.12
xxxxx.xxxxx.xxx          test.otherdom.com        2015-01-20 14:37 CET +0100  Ubuntu 9.10   x86_64  2g (free 0.28g)  40g                  16g               10.0.0.15, 10.0.0.9

The script:

#!/usr/bin/python
# MIT license

import os
import sys
import shutil
import json
import tabulate
import pprint

host = sys.argv[1]
tmp_dir = 'tmp_fact_col'

try:
    shutil.rmtree(tmp_dir)
except OSError:
    pass
os.mkdir(tmp_dir)
cmd = "ansible -t {} -m setup {} >/dev/null".format(tmp_dir, host)
os.system(cmd)

headers = [
    'Name', 'FQDN', 'Datetime', 'OS', 'Arch', 'Mem', 'Disk', 'Diskfree', 'IPs',
]
d = []

for fname in os.listdir(tmp_dir):
    path = os.path.join(tmp_dir, fname)
    j = json.load(file(path, 'r'))
    if 'failed' in j:
        continue
    d.append(
        (
            fname,
            j['ansible_facts']['ansible_fqdn'],
            "%s %s:%s %s %s" % (
                j['ansible_facts']['ansible_date_time']['date'],
                j['ansible_facts']['ansible_date_time']['hour'],
                j['ansible_facts']['ansible_date_time']['minute'],
                j['ansible_facts']['ansible_date_time']['tz'],
                j['ansible_facts']['ansible_date_time']['tz_offset'],
            ),
            "%s %s" % (
                j['ansible_facts']['ansible_distribution'],
                j['ansible_facts']['ansible_distribution_version'],
            ),
            j['ansible_facts']['ansible_architecture'],
            '%0.fg (free %0.2fg)' % (
                (j['ansible_facts']['ansible_memtotal_mb'] / 1000.0),
                (j['ansible_facts']['ansible_memfree_mb'] / 1000.0)
                ),
            ', '.join([str(i['size_total']/1048576000) + 'g' for i in j['ansible_facts']['ansible_mounts']]),
            ', '.join([str(i['size_available']/1048576000) + 'g' for i in j['ansible_facts']['ansible_mounts']]),
            ', '.join(j['ansible_facts']['ansible_all_ipv4_addresses']),
        )
    )
    os.unlink(path)
shutil.rmtree(tmp_dir)
print tabulate.tabulate(d, headers=headers)

The script requires the Tabulator python library. Put the script in the directory containing your ansible hosts file, and run it.

 

Can't save imported OpenVPN configuration in Network Manager

I ran into an issue where I couldn't save an imported OpenVPN (.ovpn) configuration in Network Manager. The "Save" button remains disabled:

vpn

It turns out I need to enter a password for the Private Key. Ofcourse, this particular private key doesn't have a password, but you can simply enter a single space as your password. After that the "Save" button becomes active.

Bexec v0.8: Execute a vim buffer and capture output in split window

I released v0.8 of my Bexec vim plugin. The Bexec plugin allows the user to execute the current buffer if it contains a script with a shebang (#!/path/to/interpreter) on the first line or if the default interpreter for the script's type is known by Bexec. The output of the script will be grabbed and displayed in a separate buffer. 

New in this release:

  • Honor splitbelow and splitright vim setting (patch by Christopher Pease).

bexec

Installation instructions:

  1. Download the Vimball
  2. Start vim with: vim bexec-v0.8.vmb
  3. In Vim, type: :source %
  4. Bexec is now installed. Type :Bexec to run it, or use <MapLeader>bx

 

 

POODLE: SSLv3 bug summary

Yet Another SSL bug: This time a problem with SSLv3.

Most browsers and web servers support SSLv3. Many don't use it by default; instead opting for higher versions of SSL such as TLS v1.0+. The problem is that attackers can force a downgrade of the negotiated protocol, which will result in the SSLv3 protocol being used to communicate.

No real fixes are available and vendors will probably not be sending out updates to fix this issue. The recommended method of mitigation is to disable SSLv3 on your servers and your browsers. SSLv3 is old and only the following browsers can't work with anything better:

  • Internet Explorer up to (and including) v6
  • Opera v1 t/m 4 (current version is 12)

Other browsers (Firefox, Chrome, etc) have supported TLSv1.0+ from their first release.

To test if you're vulnerable:

openssl s_client -connect HOSTNAME:443 -ssl3

If you do NOT get a message saying something like "ssl handshake failure", your server is vulnerable.

A quick test (which I do not garantuee to be correct) is:

openssl s_client -connect 127.0.0.1:443 -ssl3 2>/dev/null | grep "Server certificate"

If this returns "Server certificate", you're vulnerable.

To fix this for Apache, edit your SSL module configuration (/etc/apache2/mods-enabled/ssl.conf on Debian-derived systems) and add "-SSLv3" to your protocols to disable SSLv3:

SSLProtocol all -SSLv2 -SSLv3

This disabled SSLv2 and 3, which are both broken. It also means users with Internet Explorer 5 or 6 won't be able to reach your secure website anymore.

 

Pydocmd: Generate Markdown from python source files

I've created pydocmd. It generates Python Module / script documentation in the Markdown (md) format. It was written to automatically generate documentation that can be put on Github or Bitbucket.

It is as of yet not very complete and is more of a Proof-of-concept than a fully-fledged tool. Markdown is also a very restricted format and every implementation works subtly, or completely, different. This means output may be different on different converters. The Bitbucket version doesn't look as nice as the Github version.

Get it from bitbucket.

Example outputs are available:

Work around insufficient remote permissions when SCPing

Here's a problem I often run into:

  • I need to copy files from a remote system to my local system.
  • I have root access to the remote system via sudo or su, but not directly via SSH.
  • I don't have enough permissions to read the remote files as a normal user; I need to be root.
  • There isn't enough space to copy the files to a temp dir and change their ownership.

One solution is to use sudo tar remotely and output the tar file on stdout:

fboender@local$ ssh fboender@example.com "sudo tar -vczf - /root/foo" > foo.tar.gz

This relies on the remote host allowing X11 forwarding though, and you have to have an SSH askpass program installed. Half of the time, I can't get this work properly.

An easier solution is to build a reverse remote tunnel:

fboender@local$ ssh -R 19999:localhost:22 fboender@example.com

This maps the remote port 19999 on example.com to my local port 22. That means I can now access the SSH server running locally from the remote server by SSHing to port 19999. For example:

fboender@example.com$ scp -P 19999 -r /root/foo fboender@127.0.0.1
Password: 

There you go. Easy as pie.

How to REALLY test for Bash Shellshock (CVE-2014-6271)

Like always in a crisis, many things go wrong. Everyobody starts chattering, and start deteriorating the signal-to-noise level. I'll keep this brief.

There are a bunch of sites out there that are telling you how to test for the Bash Shellshock vulnerability. Many of the tests are WRONG:

# WROOOOOOOOOOOOOOOOONG
$ env x=’() { ;;}; echo vulnerable’ sh -c “echo this is a test”
syntax error near unexpected token `('

Spot the first problem! First off all, this uses the wrong kind of quotes. That syntax error is NOT an indication that your system isn't vulnerable. It's an indication that the blog where you copied the instruction from doesn't understand what ASCII quotes are.

Now, spot the second problem! Which shell is this calling?? Is it bash? No, it's `sh`. So if `sh` isn't linked to bash, you get this:

# WROOOOOOOOOOOOOOOOOOOOOOOOOOOOOOONG
$ env x='() { ;;}; echo vulnerable' sh -c “echo this is a test”
sh: x: line 0: syntax error near unexpected token `;;'
sh: x: line 0: `x () { ;;}; echo vulnerable'
sh: error importing function definition for `x'
this: “echo: command not found

"Oh, great, we're not vulnerable", you think. But it's not executing bash at all, it's executing some other shell. Sloppy work.

Here's a way to actually test your system. BUT don't take my word for it, perhaps it is not right either:

# Perhaps correct:
$ env x='() { :;}; echo vulnerable' bash -c 'echo hello'
vulnerable
hello