Electricmonk

Ferry Boender

Programmer, DevOpper, Open Source enthusiast.

Blog

Simple function caching in Python

Tuesday, September 15th, 2009

Python dynamic nature continues to astound me. I was working on a small library when I noted it did a lot of redundant IO calls. Now I’m not one for premature optimization, but a while ago I was thinking about writing a decorator or something that would wrap around functions and methods, and cache the returns. Turns out it’s way easier than I thought, and I whipped this up in a couple of minutes:

#!/usr/bin/python
#
# Public Domain.

__cache = {} # Global cache

def fncache(fn):
   """
   Function caching decorator. Keeps a cache of the return 
   value of a function and serves from cache on consecutive
   calls to the function. 
   
   Cache keys are computed from a hash of the function 
   name and the parameters (this differentiates between 
   instances through the 'self' param). Only works if 
   parameters have a unique repr() (almost everything).

   Example:

   >>> @fncache
   ... def greenham(a, b=2, c=3):
   ...   print 'CACHE MISS'
   ...   return('I like turtles')
   ... 
   >>> print greenham(1)           # Cache miss
   CACHE MISS
   I like turtles
   >>> print greenham(1)           # Cache hit
   I like turtles
   >>> print greenham(1, 2, 3)     # Cache miss (even though default params)
   CACHE MISS
   I like turtles
   >>> print greenham(2, 2, ['a']) # Cache miss
   CACHE MISS
   I like turtles
   >>> print greenham(2, 2, ['b']) # Cache miss
   CACHE MISS
   I like turtles
   >>> print greenham(2, 2, ['a']) # Cache hit
   I like turtles
   """
   def new(*args, **kwargs):
      h = hash(repr(fn) + repr(args) + repr(kwargs))
      if not h in __cache:
         __cache[h] = fn(*args, **kwargs)
      return(__cache[h])
   new.__doc__ = "%s %s" % (fn.__doc__, "(cached)")
   return(new)

if __name__ == '__main__':
   import doctest
   doctest.testmod()

Save to a file named ‘fncache.py’ and import it into your program. Then decorate your functions and methods with it, and their output will be cached. Python rocks. Remember, only use for functions that do heavy calculations, file or network IO. Determining the uniqueness of the function call is rather expensive.

HIrcd – minimal IRC server in Python

Monday, September 14th, 2009

I wrote a little IRC server in Python:

HIrcd is a minimal, hacky implementation of an IRC server daemon written in Python in about 400 lines of code, including comments, etc.

It is mostly useful as a testing tool or perhaps for building something like a private proxy on. Do NOT use it in any kind of production code or anything that will ever be connected to by the public.

Direct link to the source code, for those interested.

Software Bashing

Thursday, September 3rd, 2009

We started a new site where we can vent our rage on all things sucky about software:

Software Bashing: We hate software. With a passion:

Fact: All software sucks. We’re here to show you exactly why, and just how much it truly sucks. We don’t discriminate against vendor or development model; all software sucks. We are relentless. We show no mercy. If the software exists, we will find its suckage, no matter how much it leverages synergetic business potential. Be prepared.

Hopefully we can reach some software authors and make them see the light. If not, at least I was able significantly reduce my blood pressure through the site.

TCP performance

Monday, August 31st, 2009

Here’s an interesting article on trying to understand TCP performance. It discusses how the TCP flow-control window (receive buffer), window scaling, selective ACKs and some other TCP features/options affect your link’s speed. Of course it’s only a tiny fraction of the total picture that is link performance, but it’s an interesting read.

There’s also a webpage which inspects the TCP options your browser sends and calculates it’s maximum theoretical speed.

Teleportation

Monday, August 3rd, 2009

I teleported home one night
With Ron and Sid and Meg.
Ron stole Meggie’s heart away
And I got Sidney’s leg.

— Douglas Adams, The Restaurant at the End of the Universe.

Game review: Midnightclub Los Angeles (SUCKS)

Thursday, July 23rd, 2009

I’m not normally into the game reviewing thing, but I’ll make an exception for Midnightclub Los Angeles because it is, without a single doubt, the worst ‘racing’ (and I use the word ‘racing’ loosely here) game I have ever wasted money on.

The story: None. But that’s okay. It’s a racing game, it doesn’t require a story.

The rest of the game, though, sucks so hard, it created a small black hole in my playstation 3 which then proceeded to break the game CD into 5 pieces. Oh, wait, no, that was me. This game is absolutely terrible. Know that old arcade racing game called ‘Outrun’? It would have been better if they slapped some new graphics onto that and otherwise release it exactly the same as it was.

Mightnightclub Los Angeles truly sucks from every side you look at it. There’s no manual gear change, the cars handle like shit, you keep having to spend your hard earned money on repairing your car every single time some puts drives into it, but it never tells you how much you have to pay to repair it. It’s an arcade racer for crying out loud! Cars should be indestructible (and have rockets on them, preferably). There are no prices listed for the upgrades, nor are the effects they’ll have on the car made clear. The customization is a wreck, the menu’s are a disaster, and mobile phone that constantly pops up must have been inspired by the one in GTA IV. It makes you want to pull out your hair and rip out your eyes.

When it comes to racing, Midnightclub Los Angeles is nothing more than an extreme exercise in frustration. During the daytime races, it’s almost always near impossible to see where you’re going. But since the game is called Midnightclub Los Angeles, it’s always impossible to see where you’re going. The roads are packed with traffic, which can be fun, but not when there’s shitloads of traffic in every single race. Now I’ve played ‘traffic’ racers before, but there were always at least some races without it. Given the fact that in this game traffic is just suddenly spawned in the middle of the road, there’s no way to avoid it. They probably put this in because the actually driving, and by driving I mean cornering, in Midnightclub is so immensely boring that during the fifth race I literally fell asleep, and when I woke up, I had still won. True story. Okay, okay, not a true story.. it turns out I hit thirty-nine lampposts and sixthousand and one other cars on the road, but that’s still the same as every other race.

Did I mention this game is utterly frustrating yet? But wait, there’s more! You see, it doesn’t actually matter at all how well you do in this game. Why? Because it has that nice catch-up built in. You know, the kind where it doesn’t matter if you give your opponents a head start of twenty-five minutes, you’ll still be able to catch up with them (and they with you if the roles are reversed) before the race ends. A true challenge. Fortunately, the huge, yellow, view-obstructing smoke plume makes the game lot’s more interesting. You never know what you’ll find behind it! (A traffic jam, probably, or a house). Another thing that makes the game much more frustrating and a totally worthless piece of shit that needs to be sent to hell and suffer an eternity just like its creators fun is that, while you have no idea where you’re going and keep bumping into all kinds of shit that shouldn’t have just spawned there, your opponents, being artificial intelligent with pre-knowledge of the entire track, every little shortcut, the traffic and everything else, never have this problem.

I could go on and on about this game, but what it comes down to is that it’s simply not a racing game. It’s not a racing simulator (which I certainly wasn’t expecting), but it’s also not an arcade racing game. This game doesn’t have a single racing element in it. The best way to sum it up is: A “look at the map while avoiding cars and steer left/right depending on where the next blob on the radar tells you to go” game. There’s no way to learn the track, since you always have to play multiple races right after each other. There’s no way to see where you need to go, because the game only tells you one checkpoint in advance where the actually racing track is. (sometimes these points are about 30 meters apart in the game, in a corner). The game is simply no fun at all. All it is, is… you’ve guessed it…

frustrating.

If I awarded a score, stars, or funny little cars between the number of 0 to five, I’d give this game a -15 gazillion and a half. But since I don’t do that, and hence cannot get the satisfaction of drilling this game six feet under, I’ll just say that I literally (and when I say ‘literally’, I do mean that literally, not figuratively) spit on this game and then proceeded to break the CD in half (in five, actually) and tossing it in the trash. I felt somewhat sorry for the other items in there having to share their nice garbage-bin with such a piece of true trash, but what can you do if you don’t have an incinerator at home.

The most annoying thing about email…

Tuesday, July 7th, 2009

The most annoying thing about email…

When someone sends you an email, and not five minutes later proceeds to call you up or visit you to ask if you’ve already read their e-mail and what your response is gonna be. Then makes you explain your entire response and says “send me an email about that, will you”.

Simple Revision Control with RCS

Tuesday, June 9th, 2009

I do a lot of stuff in text file on my system. I keep notes and todo’s in them, I write drafts using a text editor and even complete user guides and other documentation. I also keep my personal planning in Gnome Planner which stores its data in an XML file, which in essence is also text. Every so often I wish I could revert a draft back to an older version, or perhaps just have little peak at how it was a couple of days ago. I also really wanted to be able to look back at my planning to see where I made mistakes and underestimated the time required for a particular task.

A revision control system would be ideal in these circumstances. However a full-featured revision control system such as Subversion or Git would be a little overkill in this situation. Enter our old and forgotten friend RCS. RCS has been around since at least 1982, and has since been deprecated by CVS, Subversion and the various other centralized and decentralized versioning systems. It is however perfectly suited for keeping revision histories of single or a small collection of files. RCS is easier to understand than just about any other revision control system too.

Let’s look at how it works. First, we have to install it. It used to be installed by default on most systems, but that’s no longer the case. If you’re running a Debian-based distribution (such as Ubuntu), you can simply install it by typing:

aptitude install rcs

We’ve now got the RCS revision system installed, and we can start using it.

Let’s create an empty text file, called ‘simplerevisioncontrolwithrcs.txt’ in the ‘drafts’ directory:

~$ cd drafts
~/drafts$ touch simple_revision_control_with_rcs.txt

RCS consists of a couple of simple command-line utilities:

  • ‘ci’: Check In RCS revisions. This allows you to save the changes made to a file that is managed by RCS.
  • ‘co’: Check out RCS working copy. This command retrieves a specific version (the latest by default) of a file in a RCS repository.

So let’s convert our empty draft to an RCS repository by checking it in. This means RCS will create a file based on the file you want to check in with the appropriate information such as when it has been been changed, what changed, etc.

~/drafts/$ ci simple_revision_control_with_rcs.txt
simple_revision_control_with_rcs.txt,v  <--  simple_revision_control_with_rcs.txt
enter description, terminated with single '.' or end of file:
NOTE: This is NOT the log message!
>> Simple revision control with RCS
>> .
initial revision: 1.1
done

There. Our initial empty file is gone, and in its place is a file under RCS revision control: ‘simplerevisioncontrolwithrcs.txt,v’. It has been given a revision number by RCS: revision 1.1.

We can now make a working copy of this file by doing a checkout. This will retrieve the latest modifications from the text file and create a new file with the original name in the current directory. The first thing we should do though, is to turn off locking. We’re working on this file ourselves, and it will never be shared, so we don’t care about locking at all. Locking can be turned off using the ‘-U’ option of the ‘rcs’ tool:

~/drafts$ rcs -U ./simple_revision_control_with_rcs.txt,v 
RCS file: ./simple_revision_control_with_rcs.txt,v
done

Now we can do a checkout so we can edit our draft. Let’s do this, and add a line to it:

~/drafts$ co simple_revision_control_with_rcs.txt,v 
simple_revision_control_with_rcs.txt,v  -->  simple_revision_control_with_rcs.txt
revision 1.1
done
~/drafts$ echo "Hello world" > simple_revision_control_with_rcs.txt

Now let’s commit this change to the repository so RCS knows about it:

~/drafts$ ci simple_revision_control_with_rcs.txt
simple_revision_control_with_rcs.txt,v  <--  simple_revision_control_with_rcs.txt
new revision: 1.2; previous revision: 1.1
enter log message, terminated with single '.' or end of file:
>> Added greeting.
>> .
done

RCS keeps an entire history of all changes made to the files it keeps under revision each time you do a check in. Take a look at the history of the file with the command ‘rlog’:

~/drafts$ rlog ./simple_revision_control_with_rcs.txt,v
RCS file: ./simple_revision_control_with_rcs.txt,v
Working file: simple_revision_control_with_rcs.txt
head: 1.2
branch:
locks: non-strict
access list:
symbolic names:
keyword substitution: kv
total revisions: 2; selected revisions: 2
description:
Simple revision control with RCS
----------------------------
revision 1.2
date: 2009/06/09 07:48:50;  author: todsah;  state: Exp;  lines: +1 -0
Added greeting.
----------------------------
revision 1.1
date: 2009/06/09 07:46:24;  author: todsah;  state: Exp;
Initial revision
=============================================================================

That lists some basic information on the file, the repository and the two revisions we’ve made so far. We can now easily check out earlier revisions of the file by specifying a specific revision, or a date:

~/drafts$ co -r1.1 simple_revision_control_with_rcs.txt
simple_revision_control_with_rcs.txt,v  -->  simple_revision_control_with_rcs.txt
revision 1.1
done
~/drafts$ cat simple_revision_control_with_rcs.txt
~/drafts$

RCS has given us back our file as it existed at revision 1.1, which was the empty file. Let’s retrieve the version of the file as it was at 14:00 today:

~/drafts$ co -d"14:00" simple_revision_control_with_rcs.txt,v
simple_revision_control_with_rcs.txt,v  -->  simple_revision_control_with_rcs.txt
revision 1.2
writable simple_revision_control_with_rcs.txt exists; remove it? [ny](n): y
done

In conclusion: RCS may appear to be completely dead, but it still has its uses. I know many people out there will think to themselves “Why not just use Git or Subversion”? I understand the attraction of very powerful tools, but I also really believe in keeping things simple, and using the right tool for the job. I don’t know much about Git, so perhaps it is ideal for such a simple requirement, but I don’t really feel like learning it just to version control a single file.

Encodings in Python

Friday, May 22nd, 2009

There’s a whole slew of information regarding all kinds of encoding issues out there on the big bad Internet. Some deal with how unicode works, some with what UTF-8 is and how it relates to other encodings and some with how to transform from one encoding to another. All that theory is nice, but I’ve found a rather worrying lack of practical, understandable and contextual information on dealing with encodings in Python, leading me to think I’d never be able to properly deal with encodings in Python.

So I took the plunge, and tried to find some stuff out. Here’s what I came up with. All of this might be terribly wrong though. Encodings are a complicated subject if you ask me, so feel free to correct me if I’m wrong.

NOTICE: This article uses special HTML entities in various places to show output. Depending on your browser, the encodings it supports and the font you are using and its capabilities of showing UTF-8 characters, you may or may not be able to properly see these characters. In these cases a description of the character is given between parenthesis right after the character.

(more…)

Gift certificates

Sunday, May 17th, 2009

I don’t understand gift certificates. I mean, the idea is quite good: A piece of paper that represents a certain value, and which you can then trade for goods of some kind. Much better than dragging all that gold around all the time. So in that regard, gift certificates are an awesome idea. Except that we already have this thingy which is made of paper (most of the time) and that represents a certain value. It’s called “money”.

The best thing about money is that you can spend it on everything, everywhere. Whereas most gift certificates are only valid in certain stores. The only reason gift certificates make sense is if the giver wants you to spend it in a certain store. I guess that’s the reason gift certificates exist: vendor lock-in. Another brilliant way of controlling how and where we spend our money. Capitalism, yay.

I prefer money.

The text of all posts on this blog, unless specificly mentioned otherwise, are licensed under this license.