Ferry Boender
Programmer, DevOpper, Open Source enthusiast.
Wednesday, January 21st, 2015
I ran into an issue where I couldn’t save an imported OpenVPN (.ovpn) configuration in Network Manager. The “Save” button remains disabled:
It turns out I need to enter a password for the Private Key. Ofcourse, this particular private key doesn’t have a password, but you can simply enter a single space as your password. After that the “Save” button becomes active.
Sunday, October 19th, 2014
I released v0.8 of my Bexec vim plugin. The Bexec plugin allows the user to execute the current buffer if it contains a script with a shebang (#!/path/to/interpreter) on the first line or if the default interpreter for the script’s type is known by Bexec. The output of the script will be grabbed and displayed in a separate buffer.
New in this release:
Installation instructions:
vim bexec-v0.8.vmb:source %:Bexec to run it, or use <MapLeader>bx
Wednesday, October 15th, 2014
Yet Another SSL bug: This time a problem with SSLv3.
Most browsers and web servers support SSLv3. Many don’t use it by default; instead opting for higher versions of SSL such as TLS v1.0+. The problem is that attackers can force a downgrade of the negotiated protocol, which will result in the SSLv3 protocol being used to communicate.
No real fixes are available and vendors will probably not be sending out updates to fix this issue. The recommended method of mitigation is to disable SSLv3 on your servers and your browsers. SSLv3 is old and only the following browsers can’t work with anything better:
Other browsers (Firefox, Chrome, etc) have supported TLSv1.0+ from their first release.
To test if you’re vulnerable:
openssl s_client -connect HOSTNAME:443 -ssl3
If you do NOT get a message saying something like “ssl handshake failure”, your server is vulnerable.
A quick test (which I do not garantuee to be correct) is:
openssl s_client -connect 127.0.0.1:443 -ssl3 2>/dev/null | grep "Server certificate"
If this returns “Server certificate”, you’re vulnerable.
To fix this for Apache, edit your SSL module configuration (/etc/apache2/mods-enabled/ssl.conf on Debian-derived systems) and add “-SSLv3” to your protocols to disable SSLv3:
SSLProtocol all -SSLv2 -SSLv3
This disabled SSLv2 and 3, which are both broken. It also means users with Internet Explorer 5 or 6 won’t be able to reach your secure website anymore.
Sunday, October 5th, 2014
I’ve created pydocmd. It generates Python Module / script documentation in the Markdown (md) format. It was written to automatically generate documentation that can be put on Github or Bitbucket.
It is as of yet not very complete and is more of a Proof-of-concept than a fully-fledged tool. Markdown is also a very restricted format and every implementation works subtly, or completely, different. This means output may be different on different converters. The Bitbucket version doesn’t look as nice as the Github version.
Get it from bitbucket.
Example outputs are available:
Tuesday, September 30th, 2014
Here’s a problem I often run into:
One solution is to use sudo tar remotely and output the tar file on stdout:
fboender@local$ ssh fboender@example.com "sudo tar -vczf - /root/foo" > foo.tar.gz
This relies on the remote host allowing X11 forwarding though, and you have to have an SSH askpass program installed. Half of the time, I can’t get this work properly.
An easier solution is to build a reverse remote tunnel:
fboender@local$ ssh -R 19999:localhost:22 fboender@example.com
This maps the remote port 19999 on example.com to my local port 22. That means I can now access the SSH server running locally from the remote server by SSHing to port 19999. For example:
fboender@example.com$ scp -P 19999 -r /root/foo fboender@127.0.0.1 Password:
There you go. Easy as pie.
Thursday, September 25th, 2014
Like always in a crisis, many things go wrong. Everyobody starts chattering, and start deteriorating the signal-to-noise level. I’ll keep this brief.
There are a bunch of sites out there that are telling you how to test for the Bash Shellshock vulnerability. Many of the tests are WRONG:
# WROOOOOOOOOOOOOOOOONG
$ env x=’() { ;;}; echo vulnerable’ sh -c “echo this is a test”
syntax error near unexpected token `('
Spot the first problem! First off all, this uses the wrong kind of quotes. That syntax error is NOT an indication that your system isn’t vulnerable. It’s an indication that the blog where you copied the instruction from doesn’t understand what ASCII quotes are.
Now, spot the second problem! Which shell is this calling?? Is it bash? No, it’s `sh`. So if `sh` isn’t linked to bash, you get this:
# WROOOOOOOOOOOOOOOOOOOOOOOOOOOOOOONG
$ env x='() { ;;}; echo vulnerable' sh -c “echo this is a test”
sh: x: line 0: syntax error near unexpected token `;;'
sh: x: line 0: `x () { ;;}; echo vulnerable'
sh: error importing function definition for `x'
this: “echo: command not found
“Oh, great, we’re not vulnerable”, you think. But it’s not executing bash at all, it’s executing some other shell. Sloppy work.
Here’s a way to actually test your system. BUT don’t take my word for it, perhaps it is not right either:
# Perhaps correct:
$ env x='() { :;}; echo vulnerable' bash -c 'echo hello'
vulnerable
hello
Wednesday, September 24th, 2014
Just now I tried seting up an SSH tunnel. Something I must have done for at least a few tens of thousands of times in my career. But suddenly, it didn’t work anymore:
$ ssh -L 8080:127.0.0.1:80 dev.local bind: Cannot assign requested address
After checking that the local port was free, and the remote port had Apache listening on it, and testing some other things, I turned on verbose logging for SSH:
$ ssh -v -L 8080:127.0.0.1:80 dev.local OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014 debug1: Local connections to LOCALHOST:8080 forwarded to remote address 127.0.0.1:80 debug1: Local forwarding listening on 127.0.0.1 port 8080. debug1: channel 0: new [port listener] debug1: Local forwarding listening on ::1 port 8080. bind: Cannot assign requested address
The problem becomes obvious. It’s trying to bind on an IPv6 address, not IPv4. And since IPv6 is a flaming heap of shit not production ready yet, it fails miserably.
You can force the use of IPv4 on the commandline with the -4 switch:
$ ssh -4 -L 8080:127.0.0.1:80 dev.local Linux dev.local 2.6.32-5-amd64 #1 SMP Mon Sep 23 22:14:43 UTC 2013 x86_64 [fboender@dev]~$
To permanently disable IPv6, edit your ~/.ssh/config and add:
$ vi ~/.ssh/config
Host *
AddressFamily inet
That will make sure SSH never even tries anything with IPv6.
Saturday, July 26th, 2014
In the spirit of “Automate Everything” I was tasked with scripting some oft needed tasks on Cisco Switches. It’s been a while since I’ve had to do anything even remotely related to switches, so I thought I’d start by googling for some ways to automate tasks on switches. What I found:
Both seemed to be able to get the job done quite well. Unfortunately it turns out that the source for sw_script is actually nowhere to be found and Trigger wouldn’t even install properly, giving me a whole plethora of compiler errors. Since I was rather time constrained, I decided to fall back to good old Expect.
Expect s a framework to automate interactive applications. Basically what it does is let the user insert text into the input of the program, and then watches the output of the program for specific occurrences of text, hence the name “Expect”. For example, consider a program that requires the user to enter a username and password. It lets the user know this by giving us prompts:
$ ftp host.local
Username:
Password:
We can use Expect to scan the output of the program and respond with the username and password when appropriate:
spawn ftp host.local
expect "Username:"
send "fboender\r"
expect "password:"
send "sUp3rs3creT\r"
It’s a wonderful tool, but error handling can be somewhat tricky, as you’ll see further in this article.
There is an excellent Expect library for Python called Pexpect. Installation on Debian-derived systems is as easy as “aptitude install python-pexpect“.
Here’s an example session on a Cisco switch we’ll automate with Expect in a bit:
$ ssh user@10.0.0.1 Password: Switch>enable Password: Enter configuration commands, one per line. End with CNTL/Z. Switch(config)#interface Gi2/0/2 Switch(config-if)#switchport access vlan 300 Switch(config-if)#no shutdown Switch(config-if)#end Switch#wr mem Building configuration... [OK] Switch#quit
This is a simple manual session that changes the Vlan of switch port “Gi2/0/2” to Vlan 300. So how do we go about automating this with PExpect?
Logging in
The first step is to log in. This is fairly easy:
import pexpect
switch_ip = "10.0.0.1"
switch_un = "user"
switch_pw = "s3cr3t"
switch_port = "Gi2/0/2"
switch_vlan = 300
child = pexpect.spawn('ssh %s@%s' % (switch_un, switch_ip))
child.logfile = sys.stdout
child.timeout = 4
child.expect('Password:')
child.sendline(switch_pw)
child.expect('>')
First we import the pexpect module. Then we spawn a new process “ssh user@10.0.0.1“. We set the process’ logfile to sys.stdout. This is merely for debugging purposes. It tells PExpect to show all the output it’s receiving on our terminal. The default timeout is set to 4 seconds.
Then comes the first juicy bit. We let Expect know that we expect to see a ‘Password:’ prompt. If something goes wrong, for instance the switch at 10.0.0.1 is down, expect will wait for 4 seconds, looking for the text ‘Password:’ in SSH’s output. Of course, it won’t get that prompt since the switch is down. It will then raise a pexpect.TIMEOUT exception after 4 seconds. If it does detect the ‘Password:’ prompt, it will then send the switch password and wait until it detects the prompt.
Catching errors
If we want to catch errors and show the user somewhat helpful error messages, we can use try/except clauses:
try:
child.expect('Password:')
except pexpect.TIMEOUT:
raise OurException("Login prompt not received")
After the password prompt, we send the password. If all goes well, we’ll receive the prompt. Otherwise the switch will ask for the password again. We don’t “expect” this, so PExpect will timeout once again while waiting for the “>” prompt.
try:
child.sendline(switch_pw)
child.expect('>')
except pexpect.TIMEOUT:
raise OurException("Login failed")
Let’s jump ahead a bit and look at the next interesting problem. What if we supply the wrong port? The switch will respond like so:
Switch(config)#interface Gi2/0/2
^
% Invalid input detected at '^' marker.
If, on the other hand, our port is correct, we’ll simply get a prompt:
Switch(config-if)#
So here we have two possible scenario’s. Something goes wrong, or it goes right. How do we detect this? We can tell Expect that we expect two different scenario’s:
o = child.expect(['\(config-if\)#', '% Invalid'])
if o != 0:
raise OurException("Unknown switch port '%s'" % (port))
The first scenario ‘\(config-if\)#’ is our successful one. The second is when an error occurred. We then simply check that we got the successful one, and otherwise raise an error.
The rest of the script is just straight-forward expects and sendline’s.
The full script
Here’s the full script:
import pexpect
switch_ip = "10.0.0.1"
switch_un = "user"
switch_pw = "s3cr3t"
switch_enable_pw = "m0r3s3cr3t"
port = "Gi2/0/2"
vlan = 300
try:
try:
child = pexpect.spawn('ssh %s@%s' % (switch_un, switch_ip))
if verbose:
child.logfile = sys.stdout
child.timeout = 4
child.expect('Password:')
except pexpect.TIMEOUT:
raise OurException("Couldn't log on to the switch")
child.sendline(switch_pw)
child.expect('>')
child.sendline('terminal length 0')
child.expect('>')
child.sendline('enable')
child.expect('Password:')
child.sendline(switch_enable_pw)
child.expect('#')
child.sendline('conf t')
child.expect('\(config\)#')
child.sendline('interface %s' % (port))
o = child.expect(['\(config-if\)#', '% Invalid'])
if o != 0:
raise Exception("Unknown switch port '%s'" % (port))
child.sendline('switchport access vlan %s' % (vlan))
child.expect('\(config-if\)#')
child.sendline('no shutdown')
child.expect('#')
child.sendline('end')
child.expect('#')
child.sendline('wr mem')
child.expect('[OK]')
child.expect('#')
child.sendline('quit')
except (pexpect.EOF, pexpect.TIMEOUT), e:
error("Error while trying to move the vlan on the switch.")
raise
It’s too bad that I couldn’t use any of the existing frameworks. I could have tried getting Trigger to compile, but I was time constrained so I didn’t bother. There are other ways of configuring Switches too. SNMP is one way, but it is complex and prone to errors. I believe it’s also possible to retrieve the entire configuration from a switch, modify it and put it back. This is partly what Rancid does. However that would require even more time.
Expect was a good fit in this case. Although it too is rather error prone, it’s fairly easy to catch errors as long as you’re expecting (no pun intended) them. I strongly suggest you give Trigger a try before falling back to Expect. It seems like a very decent tool.
Monday, July 14th, 2014
In my previous post I showed how to generate good looking charts with Gnuplot. Those were simple bar charts with a single bar. In this post I want to show you how to plot bar charts with multiple bars. Such charts take multiple columns of data and plot them grouped in the chart. We’ll be working with the following data:
2013-4 271467 250500 2013-5 217188 198030 2013-6 192770 163000 2013-7 242761 233000 2013-8 192893 189603 2013-9 209139 154500 2013-10 235128 202300 2013-11 264841 250070 2013-12 290258 270699 2014-1 249561 209000 2014-2 225669 185000 2014-3 247809 212000
We will re-use the Gnuplot settings from the previous post. In the previous post, we used the “plot … with boxes” method of plotting bar charts. To plot a second set of data, we just add another plotting rule after the first one:
plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090",\
"" using 3 lt rgb "#40FF00"
We don’t need to specify a data file. Gnuplot will simply reuse the first one. We define the third column and set a different color for the second set of data. This is what it produces:
As you can see, it plots the third column of data as “X”s on top of the current bar. If we add the “with boxes” option to the second plotting rule, we’ll see that it creates overlapping bars:
plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090",\
"" using 3 with boxes lt rgb "#40FF00"
While that could be a useful way to represent your data, what if you want the bars to appear next to each other? Now we run into a problem. The plotting method we use can’t deal with that. We need to plot our data as a histogram. Let’s first see how that works with a single bar:
set boxwidth 1 set style data histograms plot "registrations.dat" using 2:xtic(1) lt rgb "#406090"
We change the boxwidth to “1” (from 0.6 in in the previous post). We then set the plotting style to “histograms”. The rest remains the same.
This looks a lot like the final result from the previous post. Now we’re all set to start plotting multiple bars. Let’s add a second plot:
set style data histograms
plot "registrations.dat" using 2:xtic(1) lt rgb "#406090",\
"" using 3 lt rgb "#40FF00"
This results in the following chart:
The bars slightly overlap, which we can fix by changing the box width to a slightly smaller value:
set boxwidth 0.8
In charts with multiple columns of data it would be smart to add the legend that we removed in the previous post.
#set nokey
plot "registrations.dat" using 2:xtic(1) title "Total" lt rgb "#406090",\
"" using 3 title "From web" lt rgb "#40FF00"
There you have it. Multiple bars. Adding even more bars is a simple as simply adding another plot.
Update: Thanks to Mario Domenech Goulart for spotting that the link to the previous GNUplot post was broken!
Saturday, July 12th, 2014
Gnuplot is a tool for plotting graphs. It was originally created to allow scientists and students to visualize mathematical functions and data interactively, but has grown to support many non-interactive uses such as web scripting. It is excellent for generating all kinds of charts. Unfortunately, the defaults for Gnuplot don’t generate very appealing charts:
set terminal png size 640,300 set output "1.png" plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090"
That’s uh.. pretty ugly. Maybe that looks great if you’re going for the whole sciency look, but we want something better. Let’s fix ‘r up!
The first things we’ll do is change that ugly font to something better and get rid of the text inside the graph.
# Output to PNG, with Verdana 8pt font set terminal png nocrop enhanced font "verdana,8" size 640,300 # Don't show the legend in the chart set nokey
Here’s what it looks like now. Slightly better:
Let’s do something about those horrid bars. We’ll make them thinner and fill them in.
# Thinner, filled bars set boxwidth 0.4 set style fill solid 1.00
Okay. Now we’ll do something about the labels. We’ll add a title to the chart, set a label on the Y-axis and make the values on the Y-axis better readable.
# Set a title and Y label. The X label is obviously months, so we don't set it. set title "Number of registrations per month" set ylabel "Registrations (thousands)" # Rotate X labels and get rid of the small striped at the top (nomirror) set xtics nomirror rotate by -45 # Show human-readable Y-axis. E.g. "100 k" instead of 100000. set format y '%.0s %c'
It now looks like this:
Now we’re getting somewhere! One of the rules of creating nice charts is “less is more”. This means you should remove everything that is not vital to getting your message across. See the little stripes on the Y-axis on the left and right side of the chart? Useless. Border? Useless. Let’s get rid of them:
# Replace small stripes on the Y-axis with a horizontal gridlines set tic scale 0 set grid ytics # Remove border around chart unset border
Much nicer. Let’s do a few small optimizations and we’ll be done. We increase the title’s font size and color, change the range of the Y-axis from 190k-300k to 100k-300k and make the horizontal grid lines a little less pronounced.
# Set a title and Y label. The X label is obviously months, so we don't set it. set title "Number of registrations per month" font ",14" tc rgb "#606060" # Lighter grid lines set grid ytics lc rgb "#C0C0C0" # Manual set the Y-axis range set yrange [100000 to 300000]
The final result:
One final improvement we can make is changing the rendering engine from the default to the Cairo engine. Whether this works for you depends on if your Gnuplot was compiled with Cairo enabled. The Cairo engine does a much better job of anti-aliasing the chart, which is especially important for line charts. Here’s how to enable rendering with the Cairo engine:
set terminal pngcairo nocrop enhanced font "verdana,8" size 640,300 set grid ytics lc rgb "#505050"
We also had to modify the grid color, because it was barely distinguishable in the Cairo render. This is what it looks like:
Here’s the full GnuPlot file:
# Output to PNG, with Verdana 8pt font set terminal pngcairo nocrop enhanced font "verdana,8" size 640,300 # Don't show the legend in the chart set nokey # Thinner, filled bars set boxwidth 0.4 set style fill solid 1.00 # Set a title and Y label. The X label is obviously months, so we don't set it. set title "Number of registrations per month" font ",14" tc rgb "#606060" set ylabel "Registrations (thousands)" # Rotate X labels and get rid of the small stripes at the top (nomirror) set xtics nomirror rotate by -45 # Show human-readable Y-axis. E.g. "100 k" instead of 100000. set format y '%.0s %c' # Replace small stripes on the Y-axis with a horizontal gridlines set tic scale 0 set grid ytics lc rgb "#505050" # Remove border around chart unset border # Manual set the Y-axis range set yrange [100000 to 300000] set output "6.png" plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090"
The data used (registrations.dat):
2013-4 271467 2013-5 217188 2013-6 192770 2013-7 242761 2013-8 192893 2013-9 209139 2013-10 235128 2013-11 264841 2013-12 290258 2014-1 249561 2014-2 225669 2014-3 247809
You can save the Gnuplot directives to a file called “base.gnuplot” and include it in other files to get the desired result for multiple charts:
load "base/charts/base.gnuplot"
Gnuplot has a huge number of options and abilities. Here’s a few resources that show how Gnuplot can be used:
I hope you found this post useful.
Search this blog:
The text of all posts on this blog, unless specificly mentioned otherwise, are licensed under this license.