Sunday, August 3rd, 2008
There’s something I have to get off my chest. I HATE it when things in Python pretend to be something they are not. I often use IPython to introspect things so I know how to use them. Lately, I’ve been running into more and more libraries which return things that pretend to be one thing, but which really are something else. Observe:
import optparse
parser = optparse.OptionParser()
parser.add_option("-p", dest="path", action="store", type="string")
(options, args) = parser.parse_args()
print options
Output:
{'path': None}
“Ah”, I think, “a dictionary. Swell!”, and go ahead and do:
print options.get('path', 'default')
And then it responds:
Traceback (most recent call last):
File "/home/todsah/foo.py", line 7, in <module>
print options.get('path', 'default')
AttributeError: Values instance has no attribute 'get'
What the hell? A dictionary with no get attribute? Then, when you print the type, it turns out it’s not a dictionary at all: “<type 'instance'>“. It’s a plain old object instance, and you have to use getattr(), etc instead of foo[key] and foo.get(). Fortunately, I’ve seen this trap now for so many times, the first thing I do is check the type of the thing I’m having problems with, but occasionally it still bites me in the ass.
I am seeing this more and more often, and it annoys me to no end. Python is supposed to be a rapid application development language, and things like this are extremely annoying. Don’t pretend to be something you’re not. If an object is going to behave dictionary-like, why not just extend the real dict object? Wouldn’t that be so much easier?
So, developers: Please don’t make things appear like things they’re not. It causes confusion, and possibly even bugs.
Tuesday, July 29th, 2008
Want laten we eerlijk zijn, rokers hebben humor. Rokers hebben stijl. Rokers zijn sexy motherfuckers. Na het neuken rook je samen bezweet een sigaret in bed. Niet-rokers aaien samen een kat.
Beste.. column.. ooit: Rokers zijn sexy
Friday, July 25th, 2008
Okay. So what’s cool about Python? I can’t count the number of times I’ve had to show skeptics why Python is cool, what Python can do that their favorite language can’t do. So I’m writing a bunch of articles showing off Python’s Awesomeness.
All articles in this series:
(more…)
Thursday, July 10th, 2008
I’ll be on vacation from now till the 26th of July.
Monday, July 7th, 2008
I hardly ever use destructors in Python objects. I guess Python’s dynamic nature often negates the need for destructors. Today though, I needed to write some data to disk when an object was destroyed, or more accurately, when the program exited. So I defined a destructor in the main controller object using the __del__ magic method. To my surprise, the destructor was never called. Not only was it never called upon program exit, but also not when I deleted it manually (using del). The code I needed this in was written a while ago, so I wasn’t intimately familiar with it anymore, leading me to think it was some strange bug in my program. I eventually traced the problem to some code which basically did this:
class Foo:
def __init__(self, x):
print "Foo: Hi"
self.x = x
def __del__(self):
print "Foo: bye"
class Bar:
def __init__(self):
print "Bar: Hi"
self.foo = Foo(self) # x = this instance
def __del__(self):
print "Bar: Bye"
bar = Bar()
# del bar # This doesn't work either.
What the above code does is that the Foo instance keeps a reference to its creator class, which is an instance of Bar. The output is:
Bar: Hi
Foo: Hi
As you can see, the destructors are never called, not even when we add a del bar at the end of the program. Removing the self.x = x solves (well, makes it disappear) the problem.
Garbage collection
The reason that __del__ is never called suddenly becomes obvious when looking at the above code. It’s a ‘problem’ with certain garbage collectors, namely: circular referencing. Python uses a reference counting garbage collecting algorithm. Such an garbage collection algorithm increases a counter on each data instance for each reference that exists to that data instance and decreases the counter when a reference to the data instance is removed. When the counter reaches zero, the data instance is garbage collected because nothing points to it anymore. Reference counting has a problem with circular links. In the above code foo.x points to bar, and bar.foo points to foo. This means that the reference counter never goes down, and the objects never get garbage collected. The destructors never gets called because of this.
The reason del foo doesn’t work is also simple to explain. I initially confused myself by thinking that del foo would call the destructor, but it only decreases the reference counter on the object (and removes the reference from the local scope). Since the count in the code above for Foo and Bar is 2 (one in the main program, one in the other instances), the count will only go down to 1 for the object.
Further info
After figuring this out, somebody (thanks Cris) pointed me to the documentation on __del__. Had I bothered to read it earlier I would have noticed the note there:
“del x” doesn’t directly call x.__del__() — the former decrements the reference count for x by one, and the latter is only called when x’s reference count reaches zero. Some common situations that may prevent the reference count of an object from going to zero include: circular references between objects
Something else I noticed in the documentation for __del__:
Circular references which are garbage are detected when the option cycle detector is enabled (it’s on by default), but can only be cleaned up if there are no Python-level __del__() methods involved.
Furher reading reveals:
A list of objects which the collector found to be unreachable but could not be freed (uncollectable objects). By default, this list contains only objects with __del__() methods.26.1Objects that have __del__() methods and are part of a reference cycle cause the entire reference cycle to be uncollectable, including objects not necessarily in the cycle but reachable only from it. Python doesn’t collect such cycles automatically because, in general, it isn’t possible for Python to guess a safe order in which to run the __del__() methods. […] It’s generally better to avoid the issue by not creating cycles containing objects with __del__() methods
This means that objects with cyclic references and __del__ methods will generate memory leaks in your Python program, unless the cyclic references are manually broken before the object is going to be deleted. Something to keep under consideration.
Program exit
You may wonder why Python doesn’t simply set all reference counts to 0 when exiting the program? As outlined in this post by the BDFL on __del__:
One final thing to ponder: if we have a __del__ method, should the interpreter guarantee that it is called when the program exits? (Like C++, which guarantees that destructors of global variables are called.) The only way to guarantee this is to go running around all modules and delete all their variables. But this means that __del__ method cannot trust that any global variables it might want to use still exist, since there is no way to know in what order variables are to be deleted.
Like the manual mentions: It is not guaranteed that __del__() methods are called for objects that still exist when the interpreter exits.
Exceptions inside destructors
On a side-note, as mentioned in the post by Van Rossom, exceptions raised in destructors are ignored:
def __del__(self):
raise Exception("Oopsy")
print "Bar: Bye"
Exception exceptions.Exception: Exception('Oopsy',) in > ignored
Bar: Hi
Foo: Hi
Foo: bye
(The warning is generated during compile-time, not run-time)
More side-notes
On another side-note: This is why it pays to learn software engineering students basic stuff like C programming and how garbage collectors and various algorithms work, even though some educators seem to think such information is not required anymore with today’s high-level languages.
Also, don’t rely on your IDE’s ability to display code-completions with a short description of the method and its parameters. Even though I probably wouldn’t have read the full documentation on __del__ and del anyway, I’ve often found that important notes and limitations are missed when not reading the full documentation of methods (deprecation notices, security concerns and nasty side effects in particular). If you want to have some fun with that, try to find a way to securely generate temporary files on a Unix system using Python or C using the manuals.
Update: A good way to prevent circular references seems to be the weakref module: weakref — Weak references. A quick introduction: Mindtrove: Python Weak References. (Thanks to zzzeek @ reddit for pointing out weakrefs)
Monday, July 7th, 2008
I just released v0.2 of pyBrainfuck.
PyBrainfuck is a speed-optimized Brainfuck interpreter written in Python.
Some other Python interpreters already exists for Brainfuck, but they are either obfuscated or awfully slow. PyBrainfuck has been optimized for speed by doing various preprocessing on the code such as pre-caching loop instructions, removing non-instructions, etc. PyBrainfuck also has configurable memory size, infinite loop protection and a somewhat spartan debugger.
PyBrainfuck can be used both as a stand-alone Brainfuck interpreter or as a python library. It can read from standard input or from a string (in library mode) and write to standard out or to a string buffer (in library mode).
Changes in this release:
- Improved exception throwing. Exceptions now include an error number.
- A bug was fixed in the jump instruction pre-processor where it would sometimes scan beyond the end-of-line of the code.
- A bug was fixed where a brainfuck program could increase the memory value beyond the byte boundary. It now wraps to 0 at 256 and to 255 at -1.
Update: Direct link to the interpreter code for those who are interested.
Saturday, June 28th, 2008
Okay. So what’s cool about Python? I can’t count the number of times I’ve had to show skeptics why Python is cool, what Python can do that their favorite language can’t do. So I’m writing a bunch of articles showing off Python’s Awesome.
All articles in this series:
(more…)
Friday, June 27th, 2008
I just released the first version of MiniOrganizer.
MiniOrganizer is a small no-nonsense personal digital organizer written in GTK2, featuring appointments, todos and notes. It uses iCalendar files as its native back-end, supports multiple alarms per appointment/todo and has recursive todos.
MiniOrganizer currently features:
- Appointments.
- Hierarchical Todos.
- Multiple alarms per appointment and todo.
- Alarm notification.
(Gnome) Panel docking.
You can visit its homepage at http://miniorganizer.electricmonk.nl.
Remember that this is only v0.1 – the first released version – so it will contain some bugs and other problems. Any feedback is greatly appreciated.
Wednesday, June 25th, 2008
Most of the Python programmers out there will know about IPython. Most of them will also know about the Python Debugger (PDB).
IPython has an advanced version of PDB (spectacularly named ‘ipdb’) which does the same for PDB as IPython does for the normal interactive Python interpreter. It adds tab completion, color syntax highlighting, etc. The -pdb switch to IPython gives us access to the debugger automatically in the event of uncaught exceptions:
def ham():
x = 5
raise NotImplementedError('Use the source, luke!')
ham()
We now run this code using ipython -pdb…
[todsah@jib]~$ ipython -pdb
In [1]: import test.py
<type 'exceptions.NotImplementedError'>: Use the source, luke!
> /home/todsah/test.py(5)ham()
4 x = 5
----> 5 raise NotImplementedError('Use the source, luke!')
6
ipdb> print x
5
And as you can see we get dropped at a nice ipdb> prompt which allows us to use the additional power of IPython to investigate the problem.
Like most decent debuggers, we can also use PDB to set breakpoints. In Python, we do that in the code, rather than via external means. To do this, we import the pdb module and tell it to drop us into the debugger when execution hits the pdb.set_trace() line.
import pdb
def ham():
x = 5
pdb.set_trace()
ham()
We run it, and lo and behold, we get dropped into the pdb debugger:
In [1]: import test.py
> /home/todsah/test.py(8)ham()
-> raise NotImplementedError('Use the source, luke!')
(Pdb) print x
5
Buuuuut… as you can see from the (Pdb) prompt, this is the normal PDB debugger, not IPython’s enhanced version. Wouldn’t it be neat to be able to use IPython’s debugger for breakpoint-induced debugging too? I spent some time looking around on the Interwebz, trying to find out how to do this, and to my surprise I couldn’t find anything? So I dove into the IPython source and discovered IPython.Debugger.Tracer:
from IPython.Debugger import Tracer; debug_here = Tracer()
def ham():
x = 5
debug_here()
raise NotImplementedError('Use the source, luke!')
ham()
Now when we execute this code, we’re dropped into the IPython enhanced IPDB debugger:
In [1]: import test
> /home/todsah/test.py(8)ham()
7 debug_here()
----> 8 raise NotImplementedError('Use the source, luke!')
9
ipdb> print x
5
We can use tab completion and all the other goodies IPython offers over standard Python now.
Tuesday, June 24th, 2008
George Carlin, an American Stand-up comedian (though I would rather classify him as a political speaker) died. Here’s an excerpt from one of his acts on American politics:
Everybody complains about politicians. Everybody says they suck. Well, where do people think these politicians come from? They don’t fall out of the sky. They don’t pass through a membrane from another reality. They come from American parents and American families, American homes, American schools, American churches, American businesses and American universities, and they are elected by American citizens. This is the best we can do folks. This is what we have to offer. It’s what our system produces: Garbage in, garbage out.
If you have selfish, ignorant citizens, you’re going to get selfish, ignorant leaders. Term limits ain’t going to do any good; you’re just going to end up with a brand new bunch of selfish, ignorant Americans. So, maybe, maybe, maybe, it’s not the politicians who suck. Maybe something else sucks around here – like, the public. Yeah, the public sucks. There’s a nice campaign slogan for somebody: ‘The Public Sucks. Fuck Hope’.
Because if it really is just the fault of these politicians, then where are all the other bright people of conscience? Where are all the bright, intelligent americans ready to step in and save the nation and lead the way? We don’t have people like that in this country. Everybody’s at the mall, scratching his ass and picking his nose and getting his creditcard out of his fanny-pack to buy a pair of sneakers with lights in them! So I have solved this little political dilemma for myself in a very simple way. On election day, I stay home! I don’t vote! Fuck ‘m!
Fuck ‘m! I don’t vote. Two reasons: First of all, it’s meaningless. This country was bought and sold and payed for a long time ago. The shit they shovel around every four years? (mimics masturbation) Doesn’t mean a fucking thing. And secondly I don’t vote because I believe if you vote, you have no right to complain. People like to twist that around, I know. “If you don’t vote you have no right to complain”, but where’s the logic in that? If you vote and you elect dishonest, incompetent people, they get into office and screw everything up, well, you are responsible for what they have done. You caused the problem, you voted them in. You have no right to complain. I, on the other hand, who did not vote? Who, in fact, did not even leave the house on election day, am in no way responsible for what these people have done and have every right to complain about the mess that you people created and that I have nothing to do with. I know that later on this year you’re gonna have another one of those really swell elections that you like so much. You enjoy yourselves, it’ll be a lot of fun, and I’m sure as soon as the election is over your country will improve immediately.
Now go and watch all his video’s on YouTube The man knew what he was talking about.
The text of all posts on this blog, unless specificly mentioned otherwise, are licensed under this license.
