contact
----------------------------

Blog <-

Bexec v0.8: Execute a vim buffer and capture output in split window

I released v0.8 of my Bexec vim plugin. The Bexec plugin allows the user to execute the current buffer if it contains a script with a shebang (#!/path/to/interpreter) on the first line or if the default interpreter for the script's type is known by Bexec. The output of the script will be grabbed and displayed in a separate buffer. 

New in this release:

  • Honor splitbelow and splitright vim setting (patch by Christopher Pease).

bexec

Installation instructions:

  1. Download the Vimball
  2. Start vim with: vim bexec-v0.8.vmb
  3. In Vim, type: :source %
  4. Bexec is now installed. Type :Bexec to run it, or use <MapLeader>bx

 

 

POODLE: SSLv3 bug summary

Yet Another SSL bug: This time a problem with SSLv3.

Most browsers and web servers support SSLv3. Many don't use it by default; instead opting for higher versions of SSL such as TLS v1.0+. The problem is that attackers can force a downgrade of the negotiated protocol, which will result in the SSLv3 protocol being used to communicate.

No real fixes are available and vendors will probably not be sending out updates to fix this issue. The recommended method of mitigation is to disable SSLv3 on your servers and your browsers. SSLv3 is old and only the following browsers can't work with anything better:

  • Internet Explorer up to (and including) v6
  • Opera v1 t/m 4 (current version is 12)

Other browsers (Firefox, Chrome, etc) have supported TLSv1.0+ from their first release.

To test if you're vulnerable:

openssl s_client -connect HOSTNAME:443 -ssl3

If you do NOT get a message saying something like "ssl handshake failure", your server is vulnerable.

A quick test (which I do not garantuee to be correct) is:

openssl s_client -connect 127.0.0.1:443 -ssl3 2>/dev/null | grep "Server certificate"

If this returns "Server certificate", you're vulnerable.

To fix this for Apache, edit your SSL module configuration (/etc/apache2/mods-enabled/ssl.conf on Debian-derived systems) and add "-SSLv3" to your protocols to disable SSLv3:

SSLProtocol all -SSLv2 -SSLv3

This disabled SSLv2 and 3, which are both broken. It also means users with Internet Explorer 5 or 6 won't be able to reach your secure website anymore.

 

Pydocmd: Generate Markdown from python source files

I've created pydocmd. It generates Python Module / script documentation in the Markdown (md) format. It was written to automatically generate documentation that can be put on Github or Bitbucket.

It is as of yet not very complete and is more of a Proof-of-concept than a fully-fledged tool. Markdown is also a very restricted format and every implementation works subtly, or completely, different. This means output may be different on different converters. The Bitbucket version doesn't look as nice as the Github version.

Get it from bitbucket.

Example outputs are available:

Work around insufficient remote permissions when SCPing

Here's a problem I often run into:

  • I need to copy files from a remote system to my local system.
  • I have root access to the remote system via sudo or su, but not directly via SSH.
  • I don't have enough permissions to read the remote files as a normal user; I need to be root.
  • There isn't enough space to copy the files to a temp dir and change their ownership.

One solution is to use sudo tar remotely and output the tar file on stdout:

fboender@local$ ssh fboender@example.com "sudo tar -vczf - /root/foo" > foo.tar.gz

This relies on the remote host allowing X11 forwarding though, and you have to have an SSH askpass program installed. Half of the time, I can't get this work properly.

An easier solution is to build a reverse remote tunnel:

fboender@local$ ssh -R 19999:localhost:22 fboender@example.com

This maps the remote port 19999 on example.com to my local port 22. That means I can now access the SSH server running locally from the remote server by SSHing to port 19999. For example:

fboender@example.com$ scp -P 19999 -r /root/foo fboender@127.0.0.1
Password: 

There you go. Easy as pie.

How to REALLY test for Bash Shellshock (CVE-2014-6271)

Like always in a crisis, many things go wrong. Everyobody starts chattering, and start deteriorating the signal-to-noise level. I'll keep this brief.

There are a bunch of sites out there that are telling you how to test for the Bash Shellshock vulnerability. Many of the tests are WRONG:

# WROOOOOOOOOOOOOOOOONG
$ env x=’() { ;;}; echo vulnerable’ sh -c “echo this is a test”
syntax error near unexpected token `('

Spot the first problem! First off all, this uses the wrong kind of quotes. That syntax error is NOT an indication that your system isn't vulnerable. It's an indication that the blog where you copied the instruction from doesn't understand what ASCII quotes are.

Now, spot the second problem! Which shell is this calling?? Is it bash? No, it's `sh`. So if `sh` isn't linked to bash, you get this:

# WROOOOOOOOOOOOOOOOOOOOOOOOOOOOOOONG
$ env x='() { ;;}; echo vulnerable' sh -c “echo this is a test”
sh: x: line 0: syntax error near unexpected token `;;'
sh: x: line 0: `x () { ;;}; echo vulnerable'
sh: error importing function definition for `x'
this: “echo: command not found

"Oh, great, we're not vulnerable", you think. But it's not executing bash at all, it's executing some other shell. Sloppy work.

Here's a way to actually test your system. BUT don't take my word for it, perhaps it is not right either:

# Perhaps correct:
$ env x='() { :;}; echo vulnerable' bash -c 'echo hello'
vulnerable
hello

 

 

SSH port forwarding: bind: Cannot assign requested address

Just now I tried seting up an SSH tunnel. Something I must have done for at least a few tens of thousands of times in my career. But suddenly, it didn't work anymore:

$ ssh -L 8080:127.0.0.1:80 dev.local
bind: Cannot assign requested address

After checking that the local port was free, and the remote port had Apache listening on it, and testing some other things, I turned on verbose logging for SSH:

$ ssh -v -L 8080:127.0.0.1:80 dev.local
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Local connections to LOCALHOST:8080 forwarded to remote
        address 127.0.0.1:80
debug1: Local forwarding listening on 127.0.0.1 port 8080.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on ::1 port 8080.
bind: Cannot assign requested address

 

The problem becomes obvious. It's trying to bind on an IPv6 address, not IPv4. And since IPv6 is a flaming heap of shit not production ready yet, it fails miserably. 

You can force the use of IPv4 on the commandline with the -4 switch:

$ ssh -4 -L 8080:127.0.0.1:80 dev.local
Linux dev.local 2.6.32-5-amd64 #1 SMP Mon Sep 23 22:14:43 UTC 2013 x86_64
[fboender@dev]~$

To permanently disable IPv6, edit your ~/.ssh/config and add:

$ vi ~/.ssh/config
Host *
    AddressFamily inet

That will make sure SSH never even tries anything with IPv6.

Scripting a Cisco switch with Python and Expect

WS-C3750G-48TS-S2In the spirit of "Automate Everything" I was tasked with scripting some oft needed tasks on Cisco Switches. It's been a while since I've had to do anything even remotely related to switches, so I thought I'd start by googling for some ways to automate tasks on switches. What I found:

Both seemed to be able to get the job done quite well. Unfortunately it turns out that the source for sw_script is actually nowhere to be found and Trigger wouldn't even install properly, giving me a whole plethora of compiler errors. Since I was rather time constrained, I decided to fall back to good old Expect.

Expect

Expect s a framework to automate interactive applications. Basically what it does is let the user insert text into the input of the program, and then watches the output of the program for specific occurrences of text, hence the name "Expect". For example, consider a program that requires the user to enter a username and password. It lets the user know this by giving us prompts:

$ ftp host.local
Username: 
Password:

We can use Expect to scan the output of the program and respond with the username and password when appropriate:

spawn ftp host.local
expect "Username:"
send "fboender\r"
expect "password:"
send "sUp3rs3creT\r"

It's a wonderful tool, but error handling can be somewhat tricky, as you'll see further in this article.

Scripting a Cisco switch

There is an excellent Expect library for Python called Pexpect. Installation on Debian-derived systems is as easy as "aptitude install python-pexpect".

Here's an example session on a Cisco switch we'll automate with Expect in a bit:

$ ssh user@10.0.0.1
Password:
Switch>enable
Password:
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)#interface Gi2/0/2 
Switch(config-if)#switchport access vlan 300
Switch(config-if)#no shutdown
Switch(config-if)#end
Switch#wr mem
Building configuration...
[OK]
Switch#quit

This is a simple manual session that changes the Vlan of switch port "Gi2/0/2" to Vlan 300. So how do we go about automating this with PExpect?

Logging in

The first step is to log in. This is fairly easy:

import pexpect

switch_ip = "10.0.0.1"
switch_un = "user"
switch_pw = "s3cr3t"
switch_port = "Gi2/0/2"
switch_vlan = 300

child = pexpect.spawn('ssh %s@%s' % (switch_un, switch_ip))
child.logfile = sys.stdout
child.timeout = 4
child.expect('Password:')
child.sendline(switch_pw)
child.expect('>')

First we import the pexpect module. Then we spawn a new process "ssh user@10.0.0.1". We set the process' logfile to sys.stdout. This is merely for debugging purposes. It tells PExpect to show all the output it's receiving on our terminal. The default timeout is set to 4 seconds.

Then comes the first juicy bit. We let Expect know that we expect to see a 'Password:' prompt. If something goes wrong, for instance the switch at 10.0.0.1 is down, expect will wait for 4 seconds, looking for the text 'Password:' in SSH's output. Of course, it won't get that prompt since the switch is down. It will then raise a pexpect.TIMEOUT exception after 4 seconds. If it does detect the 'Password:' prompt, it will then send the switch password and wait until it detects the prompt.

Catching errors

If we want to catch errors and show the user somewhat helpful error messages, we can use try/except clauses:

try:
  child.expect('Password:')
except pexpect.TIMEOUT:
  raise OurException("Login prompt not received")

​After the password prompt, we send the password. If all goes well, we'll receive the prompt. Otherwise the switch will ask for the password again. We don't "expect" this, so PExpect will timeout once again while waiting for the ">" prompt.

try:
  child.sendline(switch_pw)
  child.expect('>')
except pexpect.TIMEOUT:
  raise OurException("Login failed")

Let's jump ahead a bit and look at the next interesting problem. What if we supply the wrong port? The switch will respond like so:

Switch(config)#interface Gi2/0/2 
                         ^
% Invalid input detected at '^' marker.

If, on the other hand, our port is correct, we'll simply get a prompt:

Switch(config-if)#

So here we have two possible scenario's. Something goes wrong, or it goes right. How do we detect this? We can tell Expect that we expect two different scenario's:

o = child.expect(['\(config-if\)#', '% Invalid'])
if o != 0:
  raise OurException("Unknown switch port '%s'" % (port))

The first scenario '\(config-if\)#' is our successful one. The second is when an error occurred. We then simply check that we got the successful one, and otherwise raise an error.

​The rest of the script is just straight-forward expects and sendline's.

The full script

Here's the full script:

import pexpect

switch_ip = "10.0.0.1"
switch_un = "user"
switch_pw = "s3cr3t"
switch_enable_pw = "m0r3s3cr3t"
port = "Gi2/0/2"
vlan = 300

try:
  try:
    child = pexpect.spawn('ssh %s@%s' % (switch_un, switch_ip))
    if verbose:
        child.logfile = sys.stdout
    child.timeout = 4
    child.expect('Password:')
  except pexpect.TIMEOUT:
    raise OurException("Couldn't log on to the switch")

  child.sendline(switch_pw)
  child.expect('>')
  child.sendline('terminal length 0')
  child.expect('>')
  child.sendline('enable')
  child.expect('Password:')
  child.sendline(switch_enable_pw)
  child.expect('#')
  child.sendline('conf t')
  child.expect('\(config\)#')
  child.sendline('interface %s' % (port))
  o = child.expect(['\(config-if\)#', '% Invalid'])
  if o != 0:
      raise Exception("Unknown switch port '%s'" % (port))
  child.sendline('switchport access vlan %s' % (vlan))
  child.expect('\(config-if\)#')
  child.sendline('no shutdown')
  child.expect('#')
  child.sendline('end')
  child.expect('#')
  child.sendline('wr mem')
  child.expect('[OK]')
  child.expect('#')
  child.sendline('quit')
except (pexpect.EOF, pexpect.TIMEOUT), e:
    error("Error while trying to move the vlan on the switch.")
    raise

Conclusion

It's too bad that I couldn't use any of the existing frameworks. I could have tried getting Trigger to compile, but I was time constrained so I didn't bother. There are other ways of configuring Switches too. SNMP is one way, but it is complex and prone to errors. I believe it's also possible to retrieve the entire configuration from a switch, modify it and put it back. This is partly what Rancid does. However that would require even more time.

Expect was a good fit in this case. Although it too is rather error prone, it's fairly easy to catch errors as long as you're expecting (no pun intended) them. I strongly suggest you give Trigger a try before falling back to Expect. It seems like a very decent tool.

Multi-column data plotting with Gnuplot

In my previous post I showed how to generate good looking charts with Gnuplot. Those were simple bar charts with a single bar. In this post I want to show you how to plot bar charts with multiple bars. Such charts take multiple columns of data and plot them grouped in the chart. We'll be working with the following data:

2013-4    271467    250500
2013-5    217188    198030
2013-6    192770    163000
2013-7    242761    233000
2013-8    192893    189603
2013-9    209139    154500
2013-10    235128   202300
2013-11    264841   250070
2013-12    290258   270699
2014-1    249561    209000
2014-2    225669    185000
2014-3    247809    212000

We will re-use the Gnuplot settings from the previous post. In the previous post, we used the "plot … with boxes" method of plotting bar charts. To plot a second set of data, we just add another plotting rule after the first one:

plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090",\
     "" using 3 lt rgb "#40FF00"

We don't need to specify a data file. Gnuplot will simply reuse the first one. We define the third column and set a different color for the second set of data. This is what it produces:

multibar_1

As you can see, it plots the third column of data as "X"s on top of the current bar. If we add the "with boxes" option to the second plotting rule, we'll see that it creates overlapping bars:

plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090",\
     "" using 3 with boxes lt rgb "#40FF00"

multibar_2

While that could be a useful way to represent your data, what if you want the bars to appear next to each other? Now we run into a problem. The plotting method we use can't deal with that. We need to plot our data as a histogram. Let's first see how that works with a single bar:

set boxwidth 1
set style data histograms
plot "registrations.dat" using 2:xtic(1) lt rgb "#406090"

We change the boxwidth to "1" (from 0.6 in in the previous post). We then set the plotting style to "histograms". The rest remains the same.

multibar_3

This looks a lot like the final result from the previous post. Now we're all set to start plotting multiple bars. Let's add a second plot:

set style data histograms
plot "registrations.dat" using 2:xtic(1) lt rgb "#406090",\
     "" using 3 lt rgb "#40FF00"

This results in the following chart:

multibar_4

The bars slightly overlap, which we can fix by changing the box width to a slightly smaller value:

set boxwidth 0.8

multibar_5

In charts with multiple columns of data it would be smart to add the legend that we removed in the previous post.

#set nokey 
plot "registrations.dat" using 2:xtic(1) title "Total" lt rgb "#406090",\
     "" using 3 title "From web" lt rgb "#40FF00"

multibar_6

There you have it. Multiple bars. Adding even more bars is a simple as simply adding another plot.

Generating good-looking charts with Gnuplot

Gnuplot is a tool for plotting graphs. It was originally created to allow scientists and students to visualize mathematical functions and data interactively, but has grown to support many non-interactive uses such as web scripting. It is excellent for generating all kinds of charts. Unfortunately, the defaults for Gnuplot don't generate very appealing charts:

set terminal png size 640,300
set output "1.png"
plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090"

 

That's uh.. pretty ugly. Maybe that looks great if you're going for the whole sciency look, but we want something better. Let's fix 'r up!

The first things we'll do is change that ugly font to something better and get rid of the text inside the graph.

# Output to PNG, with Verdana 8pt font
set terminal png nocrop enhanced font "verdana,8" size 640,300

# Don't show the legend in the chart
set nokey

Here's what it looks like now. Slightly better:

2

Let's do something about those horrid bars. We'll make them thinner and fill them in.

# Thinner, filled bars
set boxwidth 0.4
set style fill solid 1.00 

3

Okay. Now we'll do something about the labels. We'll add a title to the chart, set a label on the Y-axis and make the values on the Y-axis better readable.

# Set a title and Y label. The X label is obviously months, so we don't set it.
set title "Number of registrations per month"
set ylabel "Registrations (thousands)"

# Rotate X labels and get rid of the small striped at the top (nomirror)
set xtics nomirror rotate by -45

# Show human-readable Y-axis. E.g. "100 k" instead of 100000.
set format y '%.0s %c'

It now looks like this​:

4

Now we're getting somewhere! One of the rules of creating nice charts is "less is more". This means you should remove everything that is not vital to getting your message across. See the  little stripes on the Y-axis on the left and right side of the chart? Useless. Border? Useless. Let's get rid of them:

# Replace small stripes on the Y-axis with a horizontal gridlines
set tic scale 0
set grid ytics

# Remove border around chart
unset border

5

Much nicer. Let's do a few small optimizations and we'll be done. We increase the title's font size and color, change the range of the Y-axis from 190k-300k to 100k-300k and make the horizontal grid lines a little less pronounced. 

# Set a title and Y label. The X label is obviously months, so we don't set it.
set title "Number of registrations per month" font ",14" tc rgb "#606060"

# Lighter grid lines
set grid ytics lc rgb "#C0C0C0"

# Manual set the Y-axis range
set yrange [100000 to 300000]

The final result:

6

One final improvement we can make is changing the rendering engine from the default to the Cairo engine. Whether this works for you depends on if your Gnuplot was compiled with Cairo enabled. The Cairo engine does a much better job of anti-aliasing the chart, which is especially important for line charts. Here's how to enable rendering with the Cairo engine:

set terminal pngcairo nocrop enhanced font "verdana,8" size 640,300
set grid ytics lc rgb "#505050"

We also had to modify the grid color, because it was barely distinguishable in the Cairo render. This is what it looks like:

7

Here's the full GnuPlot file:

# Output to PNG, with Verdana 8pt font
set terminal pngcairo nocrop enhanced font "verdana,8" size 640,300

# Don't show the legend in the chart
set nokey 

# Thinner, filled bars
set boxwidth 0.4
set style fill solid 1.00 

# Set a title and Y label. The X label is obviously months, so we don't set it.
set title "Number of registrations per month" font ",14" tc rgb "#606060"
set ylabel "Registrations (thousands)"

# Rotate X labels and get rid of the small stripes at the top (nomirror)
set xtics nomirror rotate by -45

# Show human-readable Y-axis. E.g. "100 k" instead of 100000.
set format y '%.0s %c'

# Replace small stripes on the Y-axis with a horizontal gridlines
set tic scale 0
set grid ytics lc rgb "#505050"

# Remove border around chart
unset border

# Manual set the Y-axis range
set yrange [100000 to 300000]

set output "6.png"
plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090"

The data used (registrations.dat):

2013-4    271467
2013-5    217188
2013-6    192770
2013-7    242761
2013-8    192893
2013-9    209139
2013-10    235128
2013-11    264841
2013-12    290258
2014-1    249561
2014-2    225669
2014-3    247809

 

You can save the Gnuplot directives to a file called "base.gnuplot" and include it in other files to get the desired result for multiple charts:

load "base/charts/base.gnuplot"

Gnuplot has a huge number of  options and abilities. Here's a few resources that show how Gnuplot can be used:

I hope you found this post useful.

 

Upload a file by command line via sftp.

If you want to upload a file by commandline via SFTP, you may end up on this StackOverflow page. The answer there is WRONG. Those are not using the SFTP subsystem, they use SSH and process output redirection. Using scp will result in an error if the server only allows the SFTP subsystem: 

This service allows sftp connections only.

Instead, use this:

$ echo "put system.log" | sftp upload@sftp.example.com:logs/
Connected to example.com.
Changing to: /home/upload/logs/
sftp> put system.log
Uploading system.log to /home/upload/logs/system.log
system.log                100%   61     0.1KB/s   00:00    

to upload a local file "system.log" to the remote SFTP host in the logs/ directory. You can also use wildcards.

The above command generates some output you may not want. In that case you can use the -q switch and redirect output to /dev/null:

$ echo "put system.log" | sftp -q upload@sftp.example.com:logs/ > /dev/null
Connected to example.com

As you can see, this still generates a little output. To fix this, we have to use batch mode:

$ cat batch.txt 
put *.log
$ sftp -q -b batch.txt upload@sftp.example.com:logs/  > /dev/null
$

Batch mode requires the use of a passwordless private key. If you don't want to load it into a key agent (for automated scripts, etc), you can use the "-o IdentityFile" option:

$ sftp -q -b batch.txt -o IdentityFile=key.rsa upload@sftp.example.com:logs/  > /dev/null