Here's what I did:

def test_file_error(self):
    self.assertRaises(IOError, file('/foo', 'r'))

I mistakenly thought this would work, in that assertRaises would notice the IOError exception and mark the test as passed. Naturally, it doesn't:

ERROR: test_file_error (__main__.SomeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test.py", line 10, in test_file_error
    self.assertRaises(IOError, file('/foo', 'r'))
IOError: [Errno 2] No such file or directory: '/foo'

The problem is that I'm a dumbass and I didn't read the documentation carefully enough:


assertRaises(exception, callable, *args, **kwds)
Test that an exception is raised when callable is called with any positional or keyword arguments that are also passed to assertRaises().


If you look carefully, you'll notice that I did not pass in a callable. Instead, I passed in the result of a callable! Here's the correct code:

def test_file_error(self):
    self.assertRaises(IOError, file, '/foo', 'r')

The difference is that this time I pass a callable (file) and the arguments ('/foo' and 'r') that the test case should pass to that callable. self.AssertRaises will then call it for me with the specified arguments and catch the IOError. In the first scenario (the wrong code), the call is made before the unit test is actually watching out for it.

My interest in Evolutionary Algorithms started when I read On the Origin of Circuits over at DamnInteresting.com. I always wanted to try something like that out for myself, but never really found the time. Now I have, and I think I've found some interesting results.

Disclaimer: I know next to nothing about Evolutionary Algorithms. Everything you read in here is the product of my own imagination and tests. I may use the wrong algorithms, nomenclature, methodology and might just be getting very bad results. They are, however, interesting to me, and I do know something about evolution, so here it is anyway.

How Evolution Works

So, how does an Evolutionary Algorithm work? Why, the same as normal biological evolution, mostly! Very (very) simply said, organism consist of DNA, which determine their characteristics. When organisms reproduce, there is a chance their offspring's DNA contains a mutation, which can lead to difference in characteristics. Sufficiently negative changes in offspring make that offspring less fit to survive, causing it, and the mutation, to die out eventually. Positive changes are passed on to future offspring. So through evolution an set of DNA naturally tends to grow towards its "goal", which is ultimate fitness for its environment. Now this is not an entirely correct description, but for our purposes it is good enough.

A simple evolutionary algorithm

There is nothing stopping us from using the same technique to evolve things towards goals set by a programmer. As can be seen from the Antenna example in the DamnInteresting article, this can sometimes even produce better things than engineers can come up with. In this post, I'm going to evolve the string "Hello, World!" from random garbage. The first example won't be very interesting, but it demonstrates the concept rather well.

First, lets define our starting point and end goal:

source = "jiKnp4bqpmAbp"
target = "Hello, World!"

Our evolutionary algorithm will start with "jiKnp4bqpmAbp", which we can view as the DNA of our "organism". It will then randomly mutate some of the DNA, and judge the new mutated string's fitness. But how do we determine fitness? This is probably the most difficult part of any evolutionary algorithm.

Lucky for us, there's an easy way to do this with strings. All we have to do is take the value of each character in the mutated string, and see how much it differs from the same character in the target string. This is called the distance between two characters. We then add all those differences, which leads us to a single value which is the fitness of that string. A fitness of 0 is perfect, and means that both strings are exactly the same. A fitness of 1 means one of the characters is off by one. For instance, the strings "Hfllo" and "Hdllo" both have a fitness of one. The higher the fitness number, the less fit it actually is!

Here's the fitness function.

def fitness(source, target):
   fitval = 0
   for i in range(0, len(source)):
      fitval += (ord(target[i]) - ord(source[i])) ** 2
   return(fitval)

If you look closely, you'll notice that for each character, I square the difference. This is to convert any negative numbers to positive ones, and to put extra emphasis on larger differences. If we don't do this, the string "Hannp" would have a fitness of 0. You see, the difference between 'e' and 'a' is -5, between 'l' and 'n' is +2 (which we have twice) and between 'o' and 'p' is +1. Adding these up yields a fitness of 0, but it's not the string we want at all. If we square the differences, they become 25, 4, 4 and 1, which yields a fitness of 34. Effectively, we square each difference so that they can't cancel each other out.

Edit: In the mutation algorithm below, I only mutate one character by one value at a time. It has been pointed out that, unless I actually allow for larger mutations, squaring the distance is largely pointless, since new mutations will always only differ by one value. At the time I wrote this fitness function, I had no idea how the rest of the algorithm would look like. It seemed like a good idea.

Now we need to introduce mutations into our string. This is rather easy. We simply pick a random character in the string, and either increment or decrease it by one, or leave it alone:

def mutate(source):
   charpos = random.randint(0, len(source) - 1)
   parts = list(source)
   parts[charpos] = chr(ord(parts[charpos]) + random.randint(-1,1))
   return(''.join(parts))

Time to tie the whole shabang together!

fitval = fitness(source, target)
i = 0
while True:
   i += 1
   m = mutate(source)
   fitval_m = fitness(m, target)
   if fitval_m < fitval:
      fitval = fitval_m
      source = m
      print "%5i %5i %14s" % (i, fitval_m, m)
   if fitval == 0:
      break

This should be easy enough to understand. For each iteration of the While-loop, we mutate the string and then calculate its fitness. If it is fitter then the original string (the parent), we make the child the new string. Otherwise, we throw it away. If the fitness is 0, we're done!

Lets look at some output. I'm snipping out some intermediary output cause it's not terribly interesting.

At generation 1, we have a fitness of 15491, and the string looks nothing like "Hello, World!". The same for generation 20, 40, 60, etc.

    1 15491  jjKnp4bqpmAbp
   20 15400  jiKnp3bppoAbp
   40 15377  jiKlo2bpooAdp
   60 15130  iiKlo2aoooAdp

Not much progress so far. At generation 500 it's still a load of nonsense:

  500  9986  \eTlo,YaorNdf

Generation 1200, we start to see something that looks like "Hello, World!":

 1200  4186  Heglo,LWorhdP

Generation 1500, we're getting very close!

 1500  3370  Hello,GWorldL

It still takes a good 1500 generations more before we're finally there:

 3078     2  Hello, Vorld"
 3079     2  Hfllo, World"
 3080     2  Hfllo, World"
 3081     0  Hello, World!

There it is!

A better, more interesting, algorithm

Okay, so that worked. But... it was kinda lame. Nothing interesting to see, really, was there? That's because our algorithm was a little too simplistic. Only one "organism" in the gene pool, only one character mutated at any time. We can do better than that, so let's modify the program to make it more interesting.

We're not going to touch our fitness function, since that works rather well. Instead, lets introduce a gene pool. Instead of having only one string, why not have a whole bunch or randomly generated strings and let them duke it out among themselves. That sounds a bit more real-life, doesn't it?

GENSIZE = 20
genepool = []
for i in range(0, GENSIZE):
   dna = [random.choice(string.printable[:-5]) for j in range(0, len(target))]
   fitness = calc_fitness(dna, target)
   candidate = {'dna': dna, 'fitness': fitness }
   genepool.append(candidate)

This little snippet generates a gene pool with 20 random strings and their fitnesses. In an official implementation, the gene pool would be called the population. (Thanks, reddit!)

Now, lets modify our mutation function. Instead of mutating one single character, we feed it two parents, picked at random from the genepool, and it will mix their DNA together a bit. This is called crossover. It will also randomly mutate one character in the resulting DNA. It then returns the newly fabricated child, including its fitness.

def mutate(parent1, parent2):
   child_dna = parent1['dna'][:]

   # Mix both DNAs
   start = random.randint(0, len(parent2['dna']) - 1)
   stop = random.randint(0, len(parent2['dna']) - 1)
   if start > stop:
      stop, start = start, stop
   child_dna[start:stop] = parent2['dna'][start:stop]

   # Mutate one position
   charpos = random.randint(0, len(child_dna) - 1)
   child_dna[charpos] = chr(ord(child_dna[charpos]) + random.randint(-1,1))
   child_fitness = calc_fitness(child_dna, target)
   return({'dna': child_dna, 'fitness': child_fitness})

We also need a routine to pick two random parents from the genepool. Now, we could just pick them completely random, but what you really want is for parents with a good fitness to have a better chance of offspring. This is called elitism If we sort the genepool list by fitness, we can use a uniform product distribution to make sure that parents with better fitness get chosen more often.

Now you might ask, what the hell is a uniform product distribution? When you randomly pick a number between, say, one and ten, each number has the same chance of being picked. This is called a "uniform distribution". But when you pick two random numbers, and you multiply them, there's a much bigger chance of getting a bigger number than a smaller number. Hence the name "uniform product distribution". Here's how that looks:

So our random parent picker will do just that. We select two random real numbers between 0 and 1, multiple those two random numbers and then scale the result up to our poolsize by multiplying the result with the size of the pool. We return that parent from the pool.

def random_parent(genepool):
   wRndNr = random.random() * random.random() * (GENSIZE - 1)
   wRndNr = int(wRndNr)
   return(genepool[wRndNr])

There! Now it's time for our main loop

while True:
   genepool.sort(key=lambda candidate: candidate['fitness'])

   if genepool[0]['fitness'] == 0:
      # Target reached
      break

   parent1 = random_parent(genepool)
   parent2 = random_parent(genepool)

   child = mutate(parent1, parent2)
   if child['fitness'] < genepool[-1]['fitness']:
      genepool[-1] = child

For each iteration of the While True loop, we first sort the genepool by fitness so that the most fit parents are at the top. We check to see if the fittest happens to be the target string we're looking for. If so, we stop the loop.

Then we select two parents from the genepool using the uniform product distribution so that fitter parents are chosen more often. We create a bastard mutated child that will mix both parents' DNA together and introduce a little mutation. If the new child is more fit than the worst in the genepool, it will replace that degenerate one in the genepool. In the next iteration, the pool is sorted again on fitness so that the new child takes its rightful place.

Results

Now it's time to run this puppy and see what it does. Again, I snip out some of the less interesting stuff.

Here's the genepool in the beginning. The first number is the generation (the number of times the While-loop has run), the second number the fitness and the third column is the DNA for that entry in the genepool.

     1   7617   'iSx{$,K`u~(B
     1   9284   SQf`1N#UdrPlT
     1  12837   sYIu<E"Fq'^_.
     1  15531   DC8Dg1I$*mUs-
     1  16064   L~*}JBVdF7bu2
     1  16533   1,XU%)5$q[YuO
     1  16588   ff],ceW<0fud&
     1  17316   [V3@2'VgY\{KV
     1  17356   kWw#v/P<#apG9
     1  17581   <Lrh(1hN_Bd)3
     1  18777   TM]_]TbtxFY:q
     1  19656   $zS+EI?BS>%z(
     1  19841   =S;B~((W8 D,6
     1  20398   P_A$D|NPJPio/
     1  21957   J&f=O:g\8'{S2
     1  22543   5*T2c"pMZ80L'
     1  24954   A&lZ#A_}MxI"P
     1  25186   &9MrI|0&x)q,N
     1  28110   OlXT/Q{y3{"LR
     1  29656   8WB99hx%0]}h[

One big random jumbled mess. Note the ones I've emphasized. These are the parents that were selected for the new child in the next generation. Lets see how it looks after one generation:

     2   7617   'iSx{$,K`u~(B
     2   8742   SQf`1N#UdfumT
     2   9284   SQf`1N#UdrPlT
     2  12837   sYIu<E"Fq'^_.
     2  15531   DC8Dg1I$*mUs-
     2  16064   L~*}JBVdF7bu2
     2  16533   1,XU%)5$q[YuO
     2  16588   ff],ceW<0fud&
     2  17316   [V3@2'VgY\{KV
     2  17356   kWw#v/P>#apG9
     2  17581   <Lrh(1hN_Bd)3
     2  18777   TM]_]TbtxFY:q
     2  19656   $zS+EI?BS>%z(
     2  19841   =S;B~((W8 D,6
     2  20398   P_A$D|NPJPio/
     2  21957   J&f=O:g\8'{S2
     2  22543   5*T2c"pMZ80L'
     2  24954   A&lZ#A_}MxI"P
     2  25186   &9MrI|0&x)q,N
     2  28110   OlXT/Q{y3{"LR

Two random parents from the previous generation have their DNA mixed, and have generated an offspring (the bold one) which is better then both of them. It comes in second with a fitness of 8742, while its parents only had fitness of 9284 and 16588. Lets skip ahead a bit and look at the 6th generation:

     6   7617   'iSx{$,K`u~(B
     6   8742   SQf`1N#UdfumT
     6   9284   SQf`1N#UdrPlT
     6  10198   SQfD1N#UdfumT
     6  12837   sYIu<E"Fq'^_.
     6  15531   DC8Dg1I$*mUs-
     6  16064   L~*}JBVdF7bu2
     6  16387   SQf`1N"MZ80LT
     6  16533   1,XU%)5$q[YuO
     6  16588   ff],ceW<0fud&
     6  17316   [V3@2'VgY\{KV
     6  17356   kWw#v/P>#apG9
     6  17356   kWw#v/P>#apG9
     6  17581   <Lrh(1hN_Bd)3
     6  18777   TM]_]TbtxFY:q
     6  19656   $zS+EI?BS>%z(
     6  19841   =S;B~((W8 D,6
     6  20287   fe],1eW<0fud&
     6  20398   P_A$D|NPJPio/
     6  21957   J&f=O:g\8'{S2

As you can see, the "SQf" has reproduced again with success, and there are now four variants of it in the genepool. We also note the "kWw#", which there are two identical ones of. This can happen when the entire DNA of one parent is copied and no mutation occurs. In our mutate function, we use the first parent's DNA as a base and then randomly overlay some of the seconds parent's DNA. This can anything from the entire second parent's DNA, or nothing at all. But generally, the chance is higher that the first parent's DNA survives largely in tact.

The next interesting generation is 13:

    13   4204   RQf`{$,KdfumT
    13   7617   'iSx{$,K`u~(B
    13   7617   'iSx{$,K`u~(B
    13   8742   SQf`1N#UdfumT
    13   8742   SQf`1N#UdfumT
    13   9284   SQf`1N#UdrPlT
    13   9284   SQf`1N#UdrPlT
    13  10198   SQfD1N#UdfumT
    13  12837   sYIu<E"Fq'^_.
    13  15531   DC8Dg1I$*mUs-
    13  15838   L~*xJBVdG7bu2
    13  15856   $zS+<E"Fq(^_(
    13  15883   L~*xJCVdG7bu2
    13  16064   L~*}JBVdF7bu2
    13  16387   SQf`1N"MZ80LT
    13  16533   1,XU%)5$q[YuO
    13  16588   ff],ceW<0fud&
    13  17316   [V3@2'VgY\{KV
    13  17356   kWw#v/P>#apG9
    13  17356   kWw#v/P>#apG9

Wow! "SQf" has been really busy and now almost rules the genepool. "iSx" is second and third, but has lost its number one position to the "RQf" variant of "SQf". "RQf" was introduced in the 12th generation as a child of an "iSx" and "SQf" variant. We see that "kWv" has been knocked almost to the end of the list by more fit candidates. It is very obvious that this pool is no longer random. Patterns are starting to emerge all over it.

By the time we reach generation 40:

    40   3306   RQSw{$-KcfumB
    40   4204   RQf`{$,KdfumT
    40   4229   RQf`|$,KdfumT
    40   4242   RQe`|$,KdfumT
    40   4795   RQSw{$-KdfumT
    40   4971   RQSwz$*K`uSnT
    40   4973   RQSwz$+K`uSmT
    40   4992   RQSwz$+K`uSnT
    40   5017   SQSxz$+K`uSmT
    40   5017   SQSxz$+K`uSmT
    40   5951   (QSxz$+KdfSmT
    40   5985   'QSxz$+K`uSmT
    40   6421   SQfx{$+K`u~(B
    40   6444   TQf`{$+K`u~(B
    40   6489   SQfx{$+KdfS(B
    40   6492   TQf`{$-K`u~(B
    40   7034   SQSxy$+KdfS(B
    40   7617   'iSx{$,K`u~(B
    40   7617   'iSx{$,K`u~(B
    40   7625   'iS`{$,Kdg~(B

The genepool is now almost entirely dominated by the "RQf" variants. Forms of its original parents "SQf" and "iSx" can still be found here and there, although "iSx" is almost entire gone from the pool. An interesting thing is that we can see combinations of letters (bold) that keep reappearing. These are almost like actual genes! Combinations of DNA that work well together and therefor stay in the genepool in that combination. It takes lots of generations to make variants of these genes that are more fit then previous versions.

The next milestone is found in the 67th generation:

    67   3138   RQSw{$+KdfukA
    67   3161   RQSw{$+KcfukA
    67   3176   RQSw{$,KdfulA
    67   3176   RQSw{$+KcfulA
    67   3218   RQSw{$-LcfumA
    67   3222   RQSw{%,KefumB
    67   3237   RQSw{$-LcfvmA
    67   3241   RQSw{$-KcfumA
    67   3241   RQSw{$-KcfumA
    67   3266   RQSw{$-KceumA
    67   3266   RQSw{$-KceumA
    67   3267   RRSw{$-KcfumB
    67   3289   RQSw{%,KefumC
    67   3306   RQSw{$-KcfumB
    67   3306   RQSw{$-KcfumB
    67   3323   RQSw{#-KcfumB
    67   3324   RPSw{$-KdfumB
    67   3331   RQSw{$-KbfumB
    67   3348   RQSw{#-KbfumB
    67   3489   RQSw{$+KdfumA

This marks the first generation where there are no other variations then the RQS one. But immediately, we see the next generation in which a new number one is found:

    68   3119   QQSw{$+KdfukA
    68   3138   RQSw{$+KdfukA
    68   3161   RQSw{$+KcfukA

By the 96th generation, QQS has taken over the top:

    96   3060   QQSw{%+KdhukA
    96   3065   QRSw{%+KdfukA
    96   3081   QQSw{%+KdgukA
    96   3081   QQSw{%+KdgukA
    96   3081   QQSw{%+KdgukA
    96   3096   QQSw{$+KdgukA
    96   3104   QQSw{%+KdfukA
    96   3119   QQSw{$+KdfukA
    96   3119   QQSw{$+KdfukA
    96   3119   QQSw{$+KdfukA
    96   3137   RRSw{$,KdfulA
    96   3137   RRSw{$,KdfulA
    96   3138   RQSw{$+KdfukA
    96   3138   RQSw{$+KdfukA
    96   3138   RQSw{$+KdfukA
    96   3138   RQSw{$+KdfukA
    96   3138   RQSw{$+KdfukA
    96   3142   QQSw{$,KdfukA
    96   3142   QQSw{$+KcfukA
    96   3144   QQSw|$+KdfukA

This is where the race gets boring. Every now and then a new, better, mutation will arise and take over the genepool. Change is slow though, and no big surprised are left. The candidates slowly but surely mutate until the reach something resembling the "Hello, World!" we are looking for in generation 1600:

  1600     19   Hdllo+ Worle%
  1600     20   Hdklo+ Worle%
  1600     20   Hdklo+ Worle%
  1600     20   Hdklo+ Worle%
  1600     20   Hdklo+ Worle%
  1600     20   Hdklo+ Workd%

It takes almost another half-thousand generation to get to the final target:

  1904      0   Hello, World!
  1904      1   Hello, World"
  1904      1   Hello, World"
  1904      2   Hello, Wprld"
  1904      2   Helmo, World"
  1904      2   Helmo, World"
  1904      2   Hdllo, World"
  1904      2   Hello, Worle"

Here are the program so you can download them and play with it a bit (ignore the SSL warning; it's a self-signed certificate):

• Naive Algorithm
• Traditional Algorithm

Interesting (if you're boring like me and you like this kind of stuff) facts:

• It usually takes anywhere between 2500 and 4000 generations to evolve the target.
• On average, it takes approximately 3100 generations to evolve the target.
• If we remove the parent DNA mixing and rely solely on mutations, it takes on average 3650 generations to evolve the target.
• The parent DNA mixing is only really useful in the beginning. In the first generations, it can quickly propel a new mix of DNA to the top of the list, but later on random mutations instead of mixing DNA becomes the main driving force between the evolution. (this doesn't have to be the case in real life evolution, naturally)
• Sometimes "beneficial" mutations disappear. For instance, the word "World" already appeared in mutation 1469, but was overtaken by other mutations quickly. It was pushed out of the genepool at generation 1486, only to reappear in generation 1659. From then on, it quickly rose to the top and dominated the top 5 positions of the genepool within 10 generations.

Update: It has rightly been pointed out that are much more efficient methods of this algorithm. Please keep in mind that I had absolutely no idea what I was doing. :-D I'm surprised I got so close to how one would properly implement an Evolutionary Algorithm.

Also, here are some more interesting statistics. I modified the mutation function a number of times, and these are the results:

• One char, -1, 0 or +1 ascii-value: 3100 generations
• Two chars, -1, 0 or +1 assii-value: 1924 generations
• Three chars, -1, 0 or +1 ascii-values: 1734 generations
• Four chars, -1, 0 or +1 ascii-values: 1706 generations
• One char, between -4 and +4 ascii-values: 1459 generations
• two chars: between -4 and +4 ascii-values: 2122 generations
• Three chars, between -4 and +4 ascii-values: 4490 generations

You can also read the
Reddit discussion and the Hacker News discussion for some nice insights. One of the most interesting comments mentions:

FWIW, for this problem, at least the way the OP set it up, the "naive" algorithm is actually a very good way to go - when I increase the population size to 20, and set the mutation/selection/crossover policies OP used, I find that the average number of fitness checks required to hit "Hello, World" (about 3510) is actually higher than the number in the naive version (in the neighborhood of 3k, usually a bit under). Also, the real time taken is larger. Which means that adding "genetic" to the algorithm has actually hurt us...
In fact, even with my full GA codebase in hand (not a substantial one, I wrote it in response to this post, but it's more flexible than the OP's), I couldn't find any situation where having a population size more than a few members helped - single member mutation (which is accepted/rejected if better/worse) always won. This is a good indication that this type of problem is vastly better suited to gradient descent than it is to a genetic algorithm.

Cool stuff.

Traceback (most recent call last):
  File "./webkit.py", line 7, in 
    import webkit
  File "/home/todsah/webkit.py", line 18, in 
    class BrowserPage(webkit.WebView):
AttributeError: 'module' object has no attribute 'WebView'

… make sure you haven't named your script 'webkit.py', and there is no other script with the same name in that directory. Also delete any webkit.pyc pic files. Do'h!

You can download it from its home page, and here's a screenshot of the thing:

Additionally, I was quite amazed at how easy it is to write GUI applications using just GTK in combination with Glade. Writing this tool took me only about an hour, with no previous knowledge. All it really required was creating a GTK Status Icon with an active signal handler. The handler pops up an interface put together in Glade by loading the gcountdown.glade file using gtk.glade.XML(). Connecting signals to the widgets is also super easy with signal_autoconnect().

Take a look at the source code. It's only a measly 136 lines.

import logging
import sys
 
class StreamToLogger(object):
   """
   Fake file-like stream object that redirects writes to a logger instance.
   """
   def __init__(self, logger, log_level=logging.INFO):
      self.logger = logger
      self.log_level = log_level
      self.linebuf = ''
 
   def write(self, buf):
      for line in buf.rstrip().splitlines():
         self.logger.log(self.log_level, line.rstrip())
 
logging.basicConfig(
   level=logging.DEBUG,
   format='%(asctime)s:%(levelname)s:%(name)s:%(message)s',
   filename="out.log",
   filemode='a'
)
 
stdout_logger = logging.getLogger('STDOUT')
sl = StreamToLogger(stdout_logger, logging.INFO)
sys.stdout = sl
 
stderr_logger = logging.getLogger('STDERR')
sl = StreamToLogger(stderr_logger, logging.ERROR)
sys.stderr = sl
 
print "Test to standard out"
raise Exception('Test to standard error')

We define a custom file-like object called StreamToLogger object which sends anything written to it to a logger instead. We then create two instances of that object and replace sys.stdout and sys.stderr with our fake file-like instances.

The output logfile looks like this:

2011-08-14 14:46:20,573:INFO:STDOUT:Test to standard out
2011-08-14 14:46:20,573:ERROR:STDERR:Traceback (most recent call last):
2011-08-14 14:46:20,574:ERROR:STDERR:  File "redirect.py", line 33, in 
2011-08-14 14:46:20,574:ERROR:STDERR:raise Exception('Test to standard error')
2011-08-14 14:46:20,574:ERROR:STDERR:Exception
2011-08-14 14:46:20,574:ERROR:STDERR::
2011-08-14 14:46:20,574:ERROR:STDERR:Test to standard error

1. Introduction

if (logged_in()) {
    if ($action == "logout") {
        act_logout();
    } else {
        # Show a form with a logout button
        display_logout();
    }
} else {
    if ($username != NULL && $password != NULL) {
        act_login();
    } else {
        display_login();
    }
}

2. Uses

3. Theory

4. A simple transducer example

Let's look at an example of a simple state machine. I've already mentioned the quote-example, so here it is in the flesh:

#!/usr/bin/python

s = "ls -la 'My Documents' /home /etc"

STATE_UNQUOTED = 1
STATE_QUOTED = 2

CHAR_QUOTE = "'"
CHAR_SPACE = " "

words = []
cur_state = STATE_UNQUOTED
cur_word = ''

# Break s up in words. Words are delimited by
# spaces, unless we're between quotes.
for char in s:
    if cur_state == STATE_QUOTED:
        if char == CHAR_QUOTE:
            words.append(cur_word)
            cur_word = ''
            cur_state = STATE_UNQUOTED
        else:
            cur_word += char
    elif cur_state == STATE_UNQUOTED:
        if char == CHAR_QUOTE:
            cur_state = STATE_QUOTED
        elif char == CHAR_SPACE:
            if cur_word:
                words.append(cur_word)
            cur_word = ''
        else:
            cur_word += char
words.append(cur_word)

print words

The output of the above State Machine is:

['ls', '-la', 'My Documents', '/home', '/etc']

As you can see, "My Documents" has been successfully parsed as a single argument.

Let's examine the State Machine up close. There are two possible states: UNQUOTED and QUOTED. We start in the UNQUOTED state and then loop over each of the characters in the string. These characters are the input into our state machine, and thus are our symbols. If we're currently in the UNQUOTED state, and the character we read is a quote, we transition to the QUOTED state. Otherwise, if the character is a space, we add the current to the list of words. This is an action. If the character is anything else, we append it to the current word (also an action)

If the State Machine is currently in the QUOTED state, we do basically the same thing as the UNQUOTED state, except that the space-character is not treated as a word-delimiter. Instead, it is just append to the current word. Encountering another quote while already in the QUOTED state means we've reached the end of the quoted word, so we append it to the list of words and transition back to the UNQUOTED state.

4.1. State Diagrams

A State Diagram is a visual method of describing a State Machine. There are many varieties of State Diagrams, each with different rules. The basic form though can be seen in figure 1, which describes the state machine from the previous example.

Figure 1: Quoted string parser State Diagram

In this example diagram the start state (UNQUOTED) is indicated by the arrow coming out of nothing going into the UNQUOTED state. The "quote char" arrows are transition conditions. Whenever a quote character is encountered, the State Machine changes state. It is debatable whether we should include the "non-quote char" arrows, since they do not indicate a state transition. However, since we want to model all the input we can receive, we will include them. This particular State Diagram does not model the action of adding words to the list when a white space occurs. This is because actions that are internal to a state are not modelled in classical State Diagrams. When modeling State Machines as implemented in computer programs, you may therefor want to make use of UML State Diagrams.

4.2. Abstracting the example

Our simple example above is a completely hand-written custom implementation of a State Machine. The state transitions and actions are all handled within the for loop. This can quickly become messy when we're dealing with bigger state machines, so we want to create an Abstract State Machine handler. There are many ways to implement Abstract State Machines, and here is an example of one:

class TransducerError(Exception):
    pass

class Transducer(object):
    def __init__(self, input, start_state):
        self.input = input
        self.output = []
        self.cur_state = start_state

    def run(self):
        for symbol in self.input:
            method = getattr(self, 'state_%s' % (self.cur_state), None)
            if not method:
                raise TransducerError('No method handler found for state \'%s\'' % (self.cur_state))
            method(symbol)
        return(self.output)

    def transition(self, new_state):
        handler = getattr(self, 'action_%s_exit' % (self.cur_state), None)
        if handler:
            handler()
        handler = getattr(self, 'action_transition', None)
        if handler:
            handler(self.cur_state, new_state)
        handler = getattr(self, 'action_%s_enter' % (new_state), None)
        if handler:
            handler()
        self.cur_state = new_state

This Abstract State Machine makes use of Python's powerful meta-programming capabilities to handle states and state transitions. The run() method reads tokens from the input. For each token it determines the current state the state machine is in and tries to find a method called state_CURRENT_STATE on the current object instance. We can then extend the Transducer class and define methods for each different state. When we transition into a different state, the Transducer class automatically tries to find entry, transition and exit methods. The exit methods (for example: action_quoted_exit) are called when we exit that particular state. The action_transition method, if found, will be called whenever we transition state, regardless of the state we came from and are going to. It is called with the previous and the new state as parameters. Finally, the entry action (for example: action_quoted_enter) is called when we transition to that particular state.

Here's how we would implement our string parser example using the abstract Transducer State Machine class.

s = "ls -la 'My Documents' /home /etc"

CHAR_QUOTE = "'"
CHAR_SPACE = " "

class Splitwords(Transducer):
    def __init__(self, s):
        Transducer.__init__(self, s, 'unquoted')
        self.output.append('')

    def state_unquoted(self, c):
        if c == CHAR_QUOTE:
            self.transition('quoted')
        elif c == CHAR_SPACE:
            self.append_word()
        else:
            self.append_char(c)

    def state_quoted(self, c):
        if c == CHAR_QUOTE:
            self.transition('unquoted')
        else:
            self.append_char(c)

    def append_word(self):
        if self.output[-1]:
            self.output.append('')

    def append_char(self, c):
        self.output[-1] += c

sw = Splitwords(s)
print sw.run()

The output:

['ls', '-la', 'My Documents', '/home', '/etc']

You can get the full example to try it out.

As you can see, the abstracted implementation of the State Machine is much clearer. The current state is automatically handled by calling the correct methods. Actions and state transitions are clearly visible in each state.

4.3. Another example: SQL

Here's another example which shows how you can parse structured statements. In this case, we parse an SQL statement.

s = "SELECT a, b FROM table WHERE a > 5 ORDER BY b"

class SQL(Transducer):
    def __init__(self, s):
        Transducer.__init__(self, s, 'select')
        self.output = {
            'select': [],
            'from': [],
            'where': [],
            'order': [],
        }

    def state_select(self, token):
        if token == "FROM":
            self.transition('from')
        elif token == "SELECT":
            pass
        else:
            self.output['select'].append(token)

    def state_from(self, token):
        if token == "ORDER":
            self.transition('order')
        elif token == "WHERE":
            self.transition('where')
        else:
            self.output['from'].append(token)

    def state_where(self, token):
        if token == "ORDER":
            self.transition('order')
        else:
            self.output['where'].append(token)

    def state_order(self, token):
        if token == 'BY':
            pass
        else:
            self.output['order'].append(token)

sw = SQL(s.split())

print sw.run()

This example produces the following output:

{'where': ['a', '>', '5'], 'from': ['table'], 'order': ['b'], 'select': ['a,', 'b']}

Of course this example is far from complete. It lacks proper syntax checking, is case-sensitive and doesn't properly sanitize various values. It demonstrates how one could create a very clear parser for structured statements.

5. Conclusion

Before we start with why a closure is useful, we might first need to understand what exactly a closure is.

Fist-class functions

In order to understand what a closure is, we must realize that in many, if not most, languages we can not just call functions, but we can also pass references to a function around in a variable. If a language supports that, it is said to have first-class functions. This can be used, amongst other things, to implement callbacks: you pass a reference to a function to a part of the program, which can then later call the function and obtain the results.

A common example of something that uses callback functions is a sorting routine that takes a comparison function. Such a function is called a higher-order function. For instance, Python's sorted function:

sorted(iterable, cmp=None, key=None, reverse=False) --> new sorted list

The cmp parameter is a callback function. If we have a list of custom objects:

class MyPerson():
   def __init__(name, age):
      self.name = name
      self.age = age

people = [
   MyPerson('john', 24),
   MyPerson('santa', 100'),
   MyPerson('pete', 30),
]

and we want to sort people by age, we can do so by defining our own custom comparison function and pass it to sorted:

def my_cmp(a, b):
   return(cmp(a.age, b.age))

sorted(people, my_cmp)

The sorted function will now loop through the items in people and call the callback function my_cmp for two items in the list at a time. If one is bigger/smaller than the other, it swaps them in order to sort people. Note that we are not calling my_cmp! We're simply passing a reference to the function to sorted.

Nested functions

Okay, so that covers first-class functions. Many languages also support nested functions. Example:

def get_cmp_func(key='age'):

   def my_cmp_name(a, b):
      return(cmp(a.name, b.name))

   def my_cmp_age(a, b):
      return(cmp(a.age, b.age))

   if key == 'name':
      return my_cmp_name
   elif key == 'age':
      return my_cmp_age

The get_cmp_func returns a function that can be used to compare things depending on what you pass as the key parameter. get_cmp_func is also a higher-order function because it returns a reference to a function. Of course in this use-case there are better ways of sorting the list, but it's just an example.

Anonymous functions

Anonymous functions are not a requirement for closures, but it may be a good idea to explain what they are nonetheless, as there's a lot of confusion over when exactly something is an anonymous function.

Anonymous functions, sometimes also called lambda's, are simply that: anonymous. They have no name. Looking at previous examples in this post, we see function names such as my_cmp, get_cmp_func and even nested functions with names: my_cmp_age. Anonymous functions have no name. That doesn't mean they can't be passed around as a reference though! Example:

sorted(people, lambda a, b: cmp(a.age, b.age))

The anonymous function here is: lambda a, b: cmp(a.age, b.age). As you can see, it looks a lot like our first my_cmp function, except it has no name and doesn't seem to return anything. That's because an anonymous (lambda) function in Python always implicitly returns its first statement. In fact, you can only have one statement in a lambda in Python. (Other languages allow for more advanced anonymous functions; Python likes to keep it simple).

Okay, so why exactly would you need anonymous functions? Well, if your language already supports first-class functions (passing around references to a function), there really isn't a need for anonymous functions, except that it saves some typing. Lambda functions are syntactic sugar for first-class functions.

Scope

So.. a closure, what is it? Again, before we can understand closures, we need to understand scope. Scope determines when we can access defined variables and functions at a certain location in our code. When a function is called, the programming language allocates a piece of memory where parameters to the function are stored and local variables can be stored by the function. This piece of memory (called the stack) is automatically cleared when the function returns. This is called the local scope.

Functions usually can also reference variable of the parent scope. For example:

a = 10

def print_a():
   print a

print_a() # output: 10

The print_a function has access to the a variable in the parent scope. But if we define a in a function's local scope, we'll get an error:

def define_a():
   a = 10

def print_a():
   print a

print_a() # NameError: global name 'a' is not defined

We get a NameError when we try to print a's value, because it is defined in define_a's local scope, which will be destroyed as soon as define_a stops running. This is called going out of scope. Anything a piece of code can access (local scope, parent scope) is defined as being within scope.

Closures

Now, finally, closures!

A closure is a special way in which scopes are handled. Instead of a function going out of scope and all the variables/functions its scope (both the local, as the parent, as the grand-parent, etc scope) being destroyed, the scope is kept around for later usage. Let's look at an example:

def define_a():
   a = 10

   def print_a():
      print a

   return(print_a)

var_print_a = define_a()
var_print_a() # output: 10

This outputs 10. Let's take a look at what's happening. We define a function define_a and set a = 10 in its local scope. We then define a nested function that prints a from the parent scope. The define_a function then returns a reference to that function.

Next, we call define_a, which returns a reference to print_a and assigns it to variable var_print_a. Then we call var_print_a as a function (this is called deferencing). By all accounts it shouldn't work, because define_a has already stopped running. It has gone out of scope and its scope (containing a) should have been destroyed. But it's not, because Python kept its scope around. This is a closure. The variables that were in scope at the time the closure was generated are still accessible for the function, and are now known as free variables.

The use-case

So, when are closures useful? Why not just use an Object and store the value in the object, along with a method that uses the object?

Let's say we have a multithreaded program that handles requests. Data is stored in a database. The request handlers need to access the data in the database, but each thread has to have its own handler to the database, or they might accidentally overwrite each other's data. So our multithreaded program allows us to register a callback function which will be called when a new thread starts. The callback function should return a new database connection for use in the thread.

def make_db_connection():
   return(db.conn(host='localhost', username='john', passwd='f00b4r'))

app = MyMultiThreadedApp(on_new_thread_cb = make_db_connection)
app.serve()

MyMultiThreadedApp will call make_db_connection for each new thread it starts, and the thread can then use the database connection returned by make_db_connection. But there is a problem! The database connection information (host, username, passwd) is hard-coded, but we want to get it from a configuration file instead!

So? We just pass some paramters to the make_db_connection right? Wrong!

def make_db_connection(host, username, passwd):
   return(db.conn(host=host, username=username, passwd=passwd))

app = MyMultiThreadedApp(on_new_thread_cb = make_db_connection)
app.serve()

This example wont work! Why not? Because MyMultiThreadedApp has absolutely no idea it should pass parameters to make_db_connection. Remember that we're not calling the function ourselves, we're just passing a reference to the MyMultiThreadedApp, which will call it eventually. There's no way for it to know which parameters it should pass, because that depends on how your database needs to be set up. SQLite only needs a path parameter, but MySQL also needs username, password, and a host.

This is where closures step in:

def gen_db_connector(host, username, passwd):
   def make_db_connection():
      return(db.conn(host=host, username=username, passwd=passwd))
   return(make_db_connection)

callback_func = gen_db_connector('localhost', 'john', 'f00b4r')
app = MyMultiThreadedApp(on_new_thread_cb = callback_func)
app.serve()

The gen_db_connector function generates a closure (make_db_connection) which has access to host, username and passwd. We then get a reference to the closure, put it in callback_func and pass that to MyMultiThreadedApp. Now when a new thread is created, and the callback function is called, it will have access to the host, username and passwd information, without MyMultiThreadedApp needing to know which params it should pass on.

An alternative to closures

There's a different way of accomplishing this though. By using objects:

class DBConnector():
   def __init__(self, host, username, passwd):
      self.host = host
      self.username = username
      self.passwd = passwd

   def connect(self):
      return(db.conn(
         host=self.host,
         username=self.username,
         passwd=self.passwd)
      )

db_conn = DBConnector('localhost', 'john', 'f00b4r')
app = MyMultiThreadedApp(on_new_thread_cb = db_conn.connect)
app.serve()

However, this is a lot more lines, and wheter it works depends on if your programming language allows first-class methods. That is, passing references around to methods on an object, while also allowing you to call them as an instance method (instead of just as a static method).

I'd personally argue for the Object way. Closures are a concept which is very hard to understand for less experienced programmers. It is a matter of debate on whether closures hide state in an unpredictable way. I tend to think they do, and I'm not much of a fan of free variables since it is hard to guess where they came from. At any rate, objects are easier to understand than closures, so if at all possible, go for the object-way.

#!/usr/bin/python
#
# Public Domain.
 
__cache = {} # Global cache
 
def fncache(fn):
   """
   Function caching decorator. Keeps a cache of the return 
   value of a function and serves from cache on consecutive
   calls to the function. 
 
   Cache keys are computed from a hash of the function 
   name and the parameters (this differentiates between 
   instances through the 'self' param). Only works if 
   parameters have a unique repr() (almost everything).
 
   Example:
 
   >>> @fncache
   ... def greenham(a, b=2, c=3):
   ...   print 'CACHE MISS'
   ...   return('I like turtles')
   ... 
   >>> print greenham(1)           # Cache miss
   CACHE MISS
   I like turtles
   >>> print greenham(1)           # Cache hit
   I like turtles
   >>> print greenham(1, 2, 3)     # Cache miss (even though default params)
   CACHE MISS
   I like turtles
   >>> print greenham(2, 2, ['a']) # Cache miss
   CACHE MISS
   I like turtles
   >>> print greenham(2, 2, ['b']) # Cache miss
   CACHE MISS
   I like turtles
   >>> print greenham(2, 2, ['a']) # Cache hit
   I like turtles
   """
   def new(*args, **kwargs):
      h = hash(repr(fn) + repr(args) + repr(kwargs))
      if not h in __cache:
         __cache[h] = fn(*args, **kwargs)
      return(__cache[h])
   new.__doc__ = "%s %s" % (fn.__doc__, "(cached)")
   return(new)
 
if __name__ == '__main__':
   import doctest
   doctest.testmod()

Save to a file named 'fncache.py' and import it into your program. Then decorate your functions and methods with it, and their output will be cached. Python rocks. Remember, only use for functions that do heavy calculations, file or network IO. Determining the uniqueness of the function call is rather expensive.

HIrcd is a minimal, hacky implementation of an IRC server daemon written in Python in about 400 lines of code, including comments, etc.

It is mostly useful as a testing tool or perhaps for building something like a private proxy on. Do NOT use it in any kind of production code or anything that will ever be connected to by the public.

Direct link to the source code, for those interested.

So I took the plunge, and tried to find some stuff out. Here's what I came up with. All of this might be terribly wrong though. Encodings are a complicated subject if you ask me, so feel free to correct me if I'm wrong.

NOTICE: This article uses special HTML entities in various places to show output. Depending on your browser, the encodings it supports and the font you are using and its capabilities of showing UTF-8 characters, you may or may not be able to properly see these characters. In these cases a description of the character is given between parenthesis right after the character.

Basics

When we're talking about text, we're really talking about two things:

• Byte representation of text.
• Encoded representation of text.

Byte representation of text doesn't give a crap about what language or encoding something is. A byte is a byte: 8 bits, 256 different values.

The text's encoding doesn't exist. That is, a text is encoded in what we say it is encoded in. If I take a piece of UTF-8 text, and say "this is encoded as latin-1", that's just fine. There is nothing inherently UTF-8y about the text. The encoding merely lets us know what little symbol we should show when we encounter a certain byte (or range of bytes, for that matter). But pretending that a piece of UTF-8 text is encoded in latin-1 can of course cause problems when the text contains bytes which latin-1 doesn't have. This is what causes encoding/decoding errors in Python.

A piece of text consisting of the byte with decimal value 163 does not exist in ASCII (as it is heigher than 128), represents '£' (Pound sterling) in latin-1, 'Ģ' (Capital G Cedilla) in Cyrillic (ISO-8859-5) and also doesn't exist in UTF-8. As you can see, it all depends on how you interpret it, and encodings are names for ways to interpret.

There are an almost infinite number of encodings out there. If you wanted to, you could make your own. Some of the ones I'll be using as an example in this little adventure are:

• ASCII

  A 7-bit encoding (the 8th bit is unused) that doesn't know jack about swishy characters with accents and stuff.

• Latin-1, ISO 8859-1, Windows-1252

  An 8-bit encoding that does know about accents and stuff. Backwards compatible with ASCII in that its first 7 bits (byte-values lower than 128) map to the same characters as ASCII does. Higher characters map to all kinds of stuff like characters with accents on them, pound signs, etc.

• UTF-8

  A variable-length encoding. It can consist of 1 byte or more, up to 4 bytes. Also backwards compatible with ASCII in the same way that latin-1 is.

When to deal with encodings

In any given software system, we have to deal with a lot of different potential encodings. Here are some general ones we have to worry about:

• The default system encoding of the Operating System.
• The encoding of any input into the program.
• The default encoding of our programming language.
• The encodings our software libraries can work with.
• The supported encoding of the tools we use to work with data in our program (debuggers, etc).
• The encoding of the destination when we output data to it.

As we can see, there are a lot of things that can go wrong. All this sounds rather despairing doesn't it? The truth of the matter is that most of the time, encodings don't matter at all. In reality, we only need to take encoding into account when:

• We want to operate on the actual meanings of the text, instead of just transporting bytes around.
• We output text to a system which needs to operate on the actual meanings of the text (or which cares about the encoding its input comes in).

Suppose we're reading a file and counting the number of bytes in that file. Does the encoding matter? No, we care about bytes, not characters. If, however, we want to count the frequency of characters, we'll need to deal with encodings, since an 'e' without an accent is not the same as an 'é' ('e' with an accent). When we're outputting text, it depends on the system we're outputting to whether we need to deal with encodings. If we're inserting data into a database, and that database expects UTF-8, we'll need to make sure we also output UTF-8. If we're printing to the console, and the system default encoding is ASCII, we'll need to make sure we're outputting ASCII.

As a software developer, you'll mostly have to deal with the encodings of input and output of your program. This presents us with two major problems:

• We need to know the encoding of our input.
• We need to know the encoding we need to output.

Sometimes, however, we simply don't know the encodings. When we read input from a text file that contains bytes with values higher than 127, we might simply not know what encoding it is. It could be in the system's default encoding, but it does not have to be. These can be tricky problems. For instance, my system default encoding is ASCII at the moment. Yet running the command apt-cache dumpavail produces output with bytes with a value higher than 128. Which encoding is it in? It's in ASCII encoding of course! Except that ASCII doesn't support bytes with larger then 128 values, so it is actually output with invalid characters.

Usually these problems are solved by making industry standards. For instance, some things always have to be in a certain encoding. Other things may let us know up-front what encoding something is in. An HTML document, for instance, should mention its encoding on the first line of the file (which should always be in just ASCII characters so everything can read it).

How to deal with encodings

WARNING: You may encounter what is sometimes called a 'heisenbug'. A heisenbug is a bug which normally does not appear when your program is running, but only appears when you try to look at the data. In the case of encodings, suppose you try to print a string, purely for debugging reasons, which has the UTF-8 encoding. The console you're printing on may not be able to deal with UTF-8, and as such Python tries to encode the UTF-8 string to ASCII and fails. This bug would not occur had you not tried to print the data to the console.

Onto the meat of the article! How do we deal with encodings?

First, some basics. Normal strings in Python don't care about encoding:

>>> s = 'Andr\xE9' # \xE9 == hex for 233 == latin-1 for e-acute.
>>> type(s)
<type 'str'>
>>> print s
Andr�

Note that I'm outputting this to an ASCII terminal, yet python does not complain about the unsupported character in 's'. It simply show some garbled text 'Andr�' ('i' with trema, reversed questionmark and a 1/2 character).

Python also has support for unicode strings. These unicode strings can contain just about any character in the entire world.

>>> s = u'Andr\xE9'
>>> type(s)
<type 'unicode'>

In this case, we define a unicode string 'André' ('e' acute) by specifying the 'é' as hexidecimal value E9. Since unicode strings can contain just about everything, they are ideal as intermediate storage for strings. It is therefor always a good idea to decode any input you receive from its encoding to unicode, and to encode it to the proper encoding when you output it again.

Now let's see what happens when we print the unicode string we just created:

>>> print s
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 4: ordinal not in range(128)

As you can see in the previous examples, when we use normal python strings, we don't get an error. But when we use the Unicode string, we get an encoding error. Like I said: normal strings in Python don't care about encoding, but unicode strings do. So when we try to print it, Python will notice our terminal is in ASCII, and tries to convert the string from Unicode to ASCII. This, naturally, fails as ASCII doesn't know about character \xE9. We'll learn how to deal with that in a moment. First, let's look at how to handle input.

So what do we do when reading in an ASCII string when it contains invalid characters? We've got three options:

• Ignore the entire encoding (if we can).
• Ignore any invalid characters.
• Replace any invalid characters with a placeholder character.

Ignoring invalid characters simply removes them from the text as the text is decoded from the source encoding to the target encoding. Replacement will replace the unknown character with a placeholder character. When encoding to unicode, it will be replaced with the character with hexidecimal value XFFFD. This is a two-byte character which will be rendered (if your font supports it) as a square or diamond with a questionmark in it.

So, let's decode our input (containing invalid characters) from ascii to unicode:

>>> s = 'Andr\xE9'               # ASCII with invalid char \xE9.
>>> s.decode('ascii', 'ignore')
u'Andr'

The 'é' is dropped from the output string as it does not exist in ASCII. We can also replace characters:

>>> s.decode('ascii', 'replace')
u'Andr\ufffd'

This works great. The 'é' is replaced with \uFFFFD: the UTF-8 symbol � (Black diamond with a questionmark) representing an unsupported character. The decode() method on normal strings decodes from the encoding you specify to a unicode string. But look at what happens when we do the same thing, but print the variable?:

>>> print s.decode('ascii', 'replace')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 4: ordinal not in range(128)

How is this possible? Did we do something wrong? No, everything is alright. This is the heisenbug. What happened here is that the 'replace' option will replace any unknown characters in the ASCII string to the \xFFFD unicode character. When we try to print the resulting unicode string to the terminal, Python will try to convert the string to our default system encoding (ASCII) and print it. This will fail, because \xFFFD isn't a character in ascii either. If we want to print it (for debugging purposes or something) we have to encode it back to ASCII before we can:

>>> print s.decode('ascii', 'replace').encode('ascii', 'replace')
Andr?

The Python manual mentions: "When a Unicode string is printed, written to a file, or converted with str(), conversion takes place using this default [ASCII] encoding". Since Python converts without using any of the replacement options, UnicodeEncodeErrors can occur. We have to encode it ourselves if our data contains invalid characters. The encode() method of a unicode string encodes the string from unicode to the encoding you specify. If we were to output the string to a HTML file that is in the UTF-8 encoding, we would do this instead:

s = 'Andr\xE9'

line = s.decode('ascii', 'replace')
f_out = file('foo.html', 'w')
f_out.write('''<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html
   PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
   "DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<body>
    %s
</body>
</html>''' % (line.encode('utf-8')))

f_out.close()

This will output a HTML document with the string 'Andr�' (where the last character is a diamond with a questionmark in it, depending on how your browser displays the UTF-8 replacement character). We first decode the line from ASCII to Unicode and when we output it, we encode it from Unicode to UTF-8. Rmemeber: UTF-8 isn't the same as Unicode! There's also UTF-16, UTF-32, etc.

If we know that the string 's' is actually in the latin-1 encoding, we can replace the line:

line = s.decode('ascii', 'replace')

with:

line = s.decode('latin-1', 'replace')

and the output in UTF-8 will become 'André'.

Another problem that can occur is when you try to run the decode() method on strings which are already unicode:

>>> s = 'Andr\xE9'
>>> u = s.decode('ascii', 'replace')
>>> u
u'Andr\ufffd'
>>> u.decode('ascii', 'replace')
Traceback (most recent call last):
  File "", line 1, in 
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 4: ordinal not in range(128)
>>> u.decode('utf-8', 'replace')
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.5/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 4: ordinal not in range(128)

As you can see, calling the decode() method on a Unicode string always seems to fail. I have no idea why the Unicode strings in Python have this method, but it's probably either because they inherit from the String type, or this is the 'internal' method Python uses when it needs to display or convert a Unicode string. If I remember correctly, the decode() will be removed from Unicode strings in the future.

Conclusion

It turns out that dealing with encodings in Python is tricky since we have to deal with a large number of potential problem areas. Once you're familier with the ins and outs though, dealing with encodings becomes rather easy. In general:

• If input is just 'passing through' your program, just treat it as binary instead of text. Don't decode or encode at all. But be careful about the target output (including the terminal when debugging with print or something) as it may require a specific encoding.

For input:

• First decode input from the encoding it is in to Unicode.
• If you know the input encoding: s = input.decode(input_encoding).
• If you do not know the input encoding, take the safe route and decode from ASCII with the 'replace' option: s = input.decode('ascii', 'replace')
• If the input might contain invalid characters for the encoding it is in, use the replace option. (always a good idea)

For output:

• If you know the output encoding: s.encode(output_encoding).
• If you do not know the output encoding, and there is an agreement about the default encoding, use that. Otherwise, take the safe route and use ASCII.
• If the target encoding might not support all the characters that are in the internal representation, use the 'replace' option.

I hope I got all this right, and that it makes dealing with encodings in Python a little clearer.

Further reading

Further reading:

An Informal Introduction to Python: Unicode-strings