contact
----------------------------

Blog <-

Archive for the ‘graphics’ Category

RSS   RSS feed for this category

Multi-column data plotting with Gnuplot

In my previous post I showed how to generate good looking charts with Gnuplot. Those were simple bar charts with a single bar. In this post I want to show you how to plot bar charts with multiple bars. Such charts take multiple columns of data and plot them grouped in the chart. We'll be working with the following data:

2013-4    271467    250500
2013-5    217188    198030
2013-6    192770    163000
2013-7    242761    233000
2013-8    192893    189603
2013-9    209139    154500
2013-10    235128   202300
2013-11    264841   250070
2013-12    290258   270699
2014-1    249561    209000
2014-2    225669    185000
2014-3    247809    212000

We will re-use the Gnuplot settings from the previous post. In the previous post, we used the "plot … with boxes" method of plotting bar charts. To plot a second set of data, we just add another plotting rule after the first one:

plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090",\
     "" using 3 lt rgb "#40FF00"

We don't need to specify a data file. Gnuplot will simply reuse the first one. We define the third column and set a different color for the second set of data. This is what it produces:

multibar_1

As you can see, it plots the third column of data as "X"s on top of the current bar. If we add the "with boxes" option to the second plotting rule, we'll see that it creates overlapping bars:

plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090",\
     "" using 3 with boxes lt rgb "#40FF00"

multibar_2

While that could be a useful way to represent your data, what if you want the bars to appear next to each other? Now we run into a problem. The plotting method we use can't deal with that. We need to plot our data as a histogram. Let's first see how that works with a single bar:

set boxwidth 1
set style data histograms
plot "registrations.dat" using 2:xtic(1) lt rgb "#406090"

We change the boxwidth to "1" (from 0.6 in in the previous post). We then set the plotting style to "histograms". The rest remains the same.

multibar_3

This looks a lot like the final result from the previous post. Now we're all set to start plotting multiple bars. Let's add a second plot:

set style data histograms
plot "registrations.dat" using 2:xtic(1) lt rgb "#406090",\
     "" using 3 lt rgb "#40FF00"

This results in the following chart:

multibar_4

The bars slightly overlap, which we can fix by changing the box width to a slightly smaller value:

set boxwidth 0.8

multibar_5

In charts with multiple columns of data it would be smart to add the legend that we removed in the previous post.

#set nokey 
plot "registrations.dat" using 2:xtic(1) title "Total" lt rgb "#406090",\
     "" using 3 title "From web" lt rgb "#40FF00"

multibar_6

There you have it. Multiple bars. Adding even more bars is a simple as simply adding another plot.

Generating good-looking charts with Gnuplot

Gnuplot is a tool for plotting graphs. It was originally created to allow scientists and students to visualize mathematical functions and data interactively, but has grown to support many non-interactive uses such as web scripting. It is excellent for generating all kinds of charts. Unfortunately, the defaults for Gnuplot don't generate very appealing charts:

set terminal png size 640,300
set output "1.png"
plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090"

 

That's uh.. pretty ugly. Maybe that looks great if you're going for the whole sciency look, but we want something better. Let's fix 'r up!

The first things we'll do is change that ugly font to something better and get rid of the text inside the graph.

# Output to PNG, with Verdana 8pt font
set terminal png nocrop enhanced font "verdana,8" size 640,300

# Don't show the legend in the chart
set nokey

Here's what it looks like now. Slightly better:

2

Let's do something about those horrid bars. We'll make them thinner and fill them in.

# Thinner, filled bars
set boxwidth 0.4
set style fill solid 1.00 

3

Okay. Now we'll do something about the labels. We'll add a title to the chart, set a label on the Y-axis and make the values on the Y-axis better readable.

# Set a title and Y label. The X label is obviously months, so we don't set it.
set title "Number of registrations per month"
set ylabel "Registrations (thousands)"

# Rotate X labels and get rid of the small striped at the top (nomirror)
set xtics nomirror rotate by -45

# Show human-readable Y-axis. E.g. "100 k" instead of 100000.
set format y '%.0s %c'

It now looks like this​:

4

Now we're getting somewhere! One of the rules of creating nice charts is "less is more". This means you should remove everything that is not vital to getting your message across. See the  little stripes on the Y-axis on the left and right side of the chart? Useless. Border? Useless. Let's get rid of them:

# Replace small stripes on the Y-axis with a horizontal gridlines
set tic scale 0
set grid ytics

# Remove border around chart
unset border

5

Much nicer. Let's do a few small optimizations and we'll be done. We increase the title's font size and color, change the range of the Y-axis from 190k-300k to 100k-300k and make the horizontal grid lines a little less pronounced. 

# Set a title and Y label. The X label is obviously months, so we don't set it.
set title "Number of registrations per month" font ",14" tc rgb "#606060"

# Lighter grid lines
set grid ytics lc rgb "#C0C0C0"

# Manual set the Y-axis range
set yrange [100000 to 300000]

The final result:

6

One final improvement we can make is changing the rendering engine from the default to the Cairo engine. Whether this works for you depends on if your Gnuplot was compiled with Cairo enabled. The Cairo engine does a much better job of anti-aliasing the chart, which is especially important for line charts. Here's how to enable rendering with the Cairo engine:

set terminal pngcairo nocrop enhanced font "verdana,8" size 640,300
set grid ytics lc rgb "#505050"

We also had to modify the grid color, because it was barely distinguishable in the Cairo render. This is what it looks like:

7

Here's the full GnuPlot file:

# Output to PNG, with Verdana 8pt font
set terminal pngcairo nocrop enhanced font "verdana,8" size 640,300

# Don't show the legend in the chart
set nokey 

# Thinner, filled bars
set boxwidth 0.4
set style fill solid 1.00 

# Set a title and Y label. The X label is obviously months, so we don't set it.
set title "Number of registrations per month" font ",14" tc rgb "#606060"
set ylabel "Registrations (thousands)"

# Rotate X labels and get rid of the small stripes at the top (nomirror)
set xtics nomirror rotate by -45

# Show human-readable Y-axis. E.g. "100 k" instead of 100000.
set format y '%.0s %c'

# Replace small stripes on the Y-axis with a horizontal gridlines
set tic scale 0
set grid ytics lc rgb "#505050"

# Remove border around chart
unset border

# Manual set the Y-axis range
set yrange [100000 to 300000]

set output "6.png"
plot "registrations.dat" using 2:xticlabels(1) with boxes lt rgb "#406090"

The data used (registrations.dat):

2013-4    271467
2013-5    217188
2013-6    192770
2013-7    242761
2013-8    192893
2013-9    209139
2013-10    235128
2013-11    264841
2013-12    290258
2014-1    249561
2014-2    225669
2014-3    247809

 

You can save the Gnuplot directives to a file called "base.gnuplot" and include it in other files to get the desired result for multiple charts:

load "base/charts/base.gnuplot"

Gnuplot has a huge number of  options and abilities. Here's a few resources that show how Gnuplot can be used:

I hope you found this post useful.

 

Recording Linux desktop with PulseAudio

For a while now I've been trying to record my Ubuntu desktop with various levels of success. I've been using recordmydesktop to do my recording. Recordmydesktop is a simple commandline tool that outputs an .ogv (Ogg Video) file which is a container format that uses the Theora video codec and the Vorbis audio codec. Most modern video/audio applications on closed-source desktops such as Windows and MacOS-X won't be able to read those, but we convert them to other formats using a range of open tools. We'll get to that later.

Recording the desktop

Recording the desktop is fairly easy. We can either record the entire desktop:

recordmydesktop -o /home/todsah/out.ogv

or we can record a single window. For this we need the window id, which you can get with a commandline tool called xwininfo:

$ xwininfo
xwininfo: Please select the window about which you
          would like information by clicking the
          mouse in that window.

Now we click on the window, and xwininfo will give us the window ID:

xwininfo: Window id: 0x9c00003 "[root@eek]/home/fboender$"

Now we can tell recordmydesktop that we want to record just that window by giving it the window ID:

recordmydesktop --windowid 0x9c00003 -o /home/todsah/out.ogv

Once you're done recording, simply press ctrl-c in the recordmydesktop terminal, and it will start encoding your recording to out.ogv.

Sound issues with PulseAudio

Okay, so far so good, except for the sound, which is simply missing. We can see that recordmydesktop did record some kind of sound by looking at the output of the encoding:

Done.
Written 1967932 bytes
(1845110 of which were video data and 122822 audio data)

So why isn't there any sound in the video? The problem is PulseAudio (again and again and again). PulseAudio acts as a layer between all the applications, all the different sound architectures (ALSA, OSS, etc) and between the hardware. In my case, the problem was that recordmydesktop was recording from the incorrect device. Of course the default Sound Preference of Ubuntu 10.04 didn't list any Recording devices, which is why it took me so long to find out what the problem was. To fix this, install the PulseAudio Volume Controller:

# aptitude install pavucontrol

You will now get a PulseAudio Volume Controller in the menu:

Start it up, and select the 'Recording' tab:

Now, make sure you've already started recording your desktop, or the application that's trying to record sound will NOT show up in this list! You should also start playing a song or generate some other audio so it's easier to spot when things are working. As you can see, it shows per application where that application is recording sound from. Select the source it will record from by clicking on the button that says 'Monitor of Internal Audio Analog Stereo'. A list will appear with options. Try each option until the volume bar below the application comes to life. I have not yet experimented with microphones, but I wouldn't be surprised if this was also the place to select that as a recording source.

recordmydesktop will now probably record sound also. You may have to tell it which channel to record sound from. This command at least worked for me:

recordmydesktop -o /home/todsah/out.ogv --fps 15 --channels 1 --freq 22050 --device pulse --v_quality 63 --s_quality 10 --workdir /tmp

You can also configure 'pulse' in the GTK front-end for recordmydesktop by going to AdvancedSoundDevice. Enter 'pulse' there.

Converting to other formats

Converting to other formats is, frankly, a real pain in the ass. There are basically three applications which you can use:

  • mencoder
  • ffmpeg
  • vlc

The main problem with encoding is codecs. A movie contains video and audio, and both of those are basically stuffed into a single file. Such a file is called a container. Examples of containers are Ogg, AVI and mp4. The video and audio files in these containers are called streams. The streams are almost always encoded in order to compress them or to allow for additional information to be embedded in the streams. Encoding these streams is done with codecs. And this is where we get in trouble. You see, most codecs are patented, trademarked, copyright and generally completely unsuitable for Free Software usages. This is why mencoder, ffmpeg and vlc usually have trouble with them.

Any of the above programs may not support the input container/codecs (Ogg container, Theora video, Vorbis audio) or may not support the output container/codecs such as the TS/Avi/Ogg/MP4/ASF containers which contain the Theora, Mpeg-4, Mpeg-2, WMV, etc video codecs and the AAC, MP3, etc audio codecs. VLC can usually play just about everything (because it provides many illegal decoding codecs), but it has trouble encoding in many format.

To my great regret, it's up to you to find out which combinatation of containers and codecs works for you. You can check the Medibuntu website for instructions on how to add illegal codecs to your Ubuntu installation.

For me, it seems mencoder works the best. For instance, to convert the out.ogv file to an AVI file:

mencoder -idx out.ogv  -ovc lavc -oac mp3lame -o output.avi

Gimp Animation frame speed

The Gimp can do animations by simply using stacked layers as animations. You can preview the animation in the Filters → Animation → Playback. It's rather to figure out how to change the time each frame is displayed though.

It turns out you have to change the layer name in order to change the time each frame is being displayed. To do this, open the Layers dialog, double click on a layer's name, and add: (500ms) at the end of the layer name to make the current layer be displayed for 500 Miliseconds.

frames

Change blindness

This is pretty cool. Try to see if you can spot the change that occurs during the flicker (after about 1.5 seconds) in this image:

Kayak

Now try it in the same image without the flicker:

Kayak

More information in this paper

Some (Debian) backgrounds

I found some Debian (and one Linux) backgrounds I had created some time ago on my computer. Decided to put them online.

Vexel artsy thingy

Yesterday evening, being the lazy slacker that I am, instead of working on my project, I was surfing around. I went to freshmeat, saw some interesting project, visited the author's homepage, saw a link, got to some other guys website where I read that he was the inventor of the word Vexel.

There was a link on his site to the Vexel site and I spent some time surfing that, viewing different Vexels. Then I came across this artist's profile. First check out this thumbnail. Looks like a photographs, doesn't it? But then check out the full version! It's actually a hand-made vector-oriented image! It was even just her fourth Vexel. Took 900 layers to create. That might not mean much to you if you're not into computer graphics, but it's a lot. The most layers I ever used in a computer generated image was about 200. And about a fifth of those were automatically generated and another fifth were just throw-away layers.

Now that's talent. Her online portfolio contains more great art, btw.

Reminds me of that demo-scene pixel-artist guy's work I saw a couple of years ago. The guy used some age-old DOS port of an Amiga image editing program to draw, completely by hand, an almost picture-perfect image of a woman. There were even individual little blond hairs on her arms. Amazing what some people can do with some free time and some artsy skills. Wish I could do it.

Smoke looks cool

Smoke looks cool.

Via J-walk blog

Optical illusion

I'm normally not too impressed with optical illusions, but this one's pretty cool.

The cool thing here is how the magenta spots disappear when you stare focus on the cross in the center and all you see is a moving green spot. I'm not sure how it works, but it seems to me the brain is filtering out the magenta spots because it believes they are nothing more than overexposed spots on the retina. Fascinating. Who knows what kind of things our brains filter out without us knowing it?

Via J-walk blog