Log <-

Archive for April, 2009

Templum v0.2.0: Simple PHP Templating

Sunday, April 26th, 2009

I just released v0.2.0 of Templum, a simple templating engine for PHP.

About

From the homepage:

Templum is an extremely lightweight, simple yet powerful and fast templating engine for PHP. It re-uses the power of PHP itself for rendering templates, but provides additional features making it easier to write templating code. Rendering templates using Templum is very fast; it approximates native PHP rendering speed for include() statements.

Changes

Changes in this release:

  • PHP 4 support added (patch by Pierre Jochem).
  • Bugfix (#1): {{$var}} at end-of-line eats newline following it.
  • Various examples added.
  • Added the ability to turn off automatic escaping using htmlentities().
  • Improved the error reporting.
  • The locale can now be changed after creating a Templum instance.
  • Userguide updated.

This release is backwards compatible with the previous version 0.1.0.

Install

You can install Templum v0.2.0 using PEAR:

pear install http://templum.electricmonk.nl/releases/templum-0.2.0.tgz

If you've got a previous version of Templum installed, you must first uninstall that one:

pear uninstall channel://__uri/templum-0.1.0

There's also a non-PEAR tar.gz which also contains examples and the API documentation and Userguide:

templum-src-0.2.0.tar.gz.

More information

Barry Schwartz: The paradox of choice

Monday, April 20th, 2009

Talks Barry Schwartz: The paradox of choice:

Psychologist Barry Schwartz takes aim at a central tenet of western societies: freedom of choice. In Schwartz's estimation, choice has made us not freer but more paralyzed, not happier but more dissatisfied.

Ubuntu sucks!

Sunday, April 19th, 2009

I used to be real pleased with Ubuntu, because it got a couple of things right that Debian didn't. But I've upgraded my Ubuntu install three times now, and every time I upgraded everything broke.

The last time I upgraded, everything even remotely having anything to do with sound broke. This was because the geniuses at Ubuntu decided to include the shitty PulseAudio sound architecture in Ubuntu way before it was ready to be included. (Yeah, I know, not really PulseAudio's fault, but I'm just trying to get the PulseAudio crowd pissed at the Ubuntu crowd in the hopes that they'll gun them down).

Last time I upgraded, from Hardy to Intrepid, all my sound stuff broke again, flash didn't work anymore, my wireless broke (I've kind of fixed it now, but now NetworkManager keeps dropping the wireless connection when I push too much data through it, and can't get up again without a reboot – piece of shit), my hotkeys and all my keybindings broke, my sessions weren't saved anymore, my sound applet refuses to address the right mixer channel, my ~/.xinitrc is being ignored, I can't rm -rf / anymore (my favorite past-time thing in Linux :-( ).

Each time I upgraded Ubuntu, I found myself doing a clean install a couple of days later because too many things had broken. And even after that, many of the broken things were still busted.

I've had it with Ubuntu. It's a piece of shit. I'm going back to Debian Stable.

Performance optimization: The first thing to do

Sunday, April 19th, 2009

In the last couple of years, I've done a lot of performance optimization. I've optimized raw C, Python and PHP code. I've optimized databases: tweaked settings, memory usage, caches, SQL code, the query analyzer, hardware and indexes. I've optimized templates for disk I/O, compilation and rendering. I've optimize various caches and all kinds of other stuff like VMWare configurations.

As I've done a lot of optimization, I've noticed a couple of things. One of these things is how persons in different roles look at optimization:

Programmers always want to optimize the code. The code isn't optimal, and a lot of speed can be gained from optimizing the code. This usually boils down to optimizing algorithms to be more efficient with either CPU cycles or memory. In contrast to the programmer, the system administrator always wants to tweak the configuration of either the Operating System, the application itself, or some piece of middle-ware in between. Meanwhile, managers always want to optimize the hardware. "Developer time is more expensive than hardware", they quip, so they decide to throw money at faster and more hardware, instead of letting the developer optimize the code.

What none of them realise is that they're all wrong. None of these approaches are any good. When it comes to optimizations, all of the people above are stuck in their own little world. Of course managers just love the fact that programmers "don't understand cost versus benefit", and a nice saying such as "Developer time is more expensive than hardware" has a really nice ring to it. Managers have a high-level view of the application's eco-system, and so they search for the solution in the cheapest component: hardware. Programmers, on the other hand, know the system from a very low-level point of view. And they naturally love the fact that managers don't understand technology. They are intimately familiar with the code of their application, or the database running behind it, and so they know a lot of its weak spots. Of course, they'll assume the optimization is best performed there. The system Systems administrators have limited options either way, so they stick to what they can influence: configuration.

An excellent example of this is a recent post on the JoelOnSoftware blog. I'll recap the main points I'd like to illustrate here:

One of the FogBugz developers complained that compiling was pretty slow (about 30 seconds). [...] He asked if it would be OK if someone spent a few weeks looking for ways to parallelize and speed it up, since we all have multiple CPU cores and plenty of memory. [...] I thought it might be a good idea to just try throwing money at the problem first, before we spent a lot of (expensive and scarce) developer time. [...] so I thought I'd experiment with replacing some of the hard drives around here with solid state, flash hard drives to see if that helped.

Suddenly everything was faster. Booting, launching apps… even Outlook is ready to use in about 1 second. This was a really great upgrade. But… compile time. Hmm. That wasn't much better. I got it down from 30 seconds to … 30 seconds. Our compiler is single threaded, and, I guess, a lot more CPU-bound than IO bound.

This is an excellent example of how a manager would try to solve optimization problems. At the start of the quote we see the typical way a developer would tackle the problem: parallelize and speed up. In other words: low-level optimizations.

Now it turns out Joel was wrong. Solid State disks didn't help at all, since their problem wasn't with disk I/O at all. But that doesn't mean the developer was right either! I like to see it as a kind of Schrödinger's Cat situation: both are wrong, until one is proven right. Why is that? Because they have no idea what the problem is!. All they're doing is guessing away at the problem in the hopes of finding out what exactly will solve it, without having any clue about the actual problem! We can see this quite clearly: after having dismissed disk I/O as the problem, they assume it must be because "our compiler is single threaded, and, I guess, a lot more CPU-bound than IO bound.". Again, they jump to conclusions without knowing what the problem is. So now they might not only waste a lot of time on solid state disks without fixing the problem, but they're about to spend weeks of developer time without knowing if that will fix the problem.

So, here is my point:

The most important thing about optimization is analysis.

You can't fix a problem by simply trying different solutions to see if they work. In order to fix a problem, you have to understand the problem first.

So, please, if you're a developer, don't assume saving a couple of CPU cycles here or there will solve the problem. And if you're a manager, don't assume some new hardware will solve the problem. Do some analysis first. Finding out if disk I/O, memory, CPU cycles or single threading is the problem is really not that hard if you spend a little time thinking about it and benchmarking various things. And in the end, you'll have a much better overview of the situation and the problem, and you'll be able to come up with specific solutions which will actually work.

And that's how you save money.

How Democracy works…

Wednesday, April 15th, 2009

Here's a good example of how democracy works:

NASA Christens Space Station Treadmill 'COLBERT'.

MySQL WEEK() function weirdness

Monday, April 13th, 2009

The other day I had to gather some statistics from a database, and the statistics needed to be grouped and filtered by weeknumbers. I ran into something a bit unexpected.

Let's say we have the follow table definition:

CREATE TABLE test (
  start DATETIME NOT NULL
);

We insert the following dates for testing purposes:

INSERT INTO test VALUES ('2008-12-27');
INSERT INTO test VALUES ('2008-12-28');
INSERT INTO test VALUES ('2008-12-29');
INSERT INTO test VALUES ('2008-12-30');
INSERT INTO test VALUES ('2008-12-31');
INSERT INTO test VALUES ('2009-01-01');
INSERT INTO test VALUES ('2009-01-02');
INSERT INTO test VALUES ('2009-01-03');
INSERT INTO test VALUES ('2009-01-04');
INSERT INTO test VALUES ('2009-01-05');

Those dates span the last week of last year, and the first week of the new year. Now, let's see what happens when we select the weeknumber from this data using MySQL's WEEK() function:

mysql> SELECT start, WEEK(start) FROM test;
+---------------------+-------------+
| start               | WEEK(start) |
+---------------------+-------------+
| 2008-12-27 00:00:00 |          51 |
| 2008-12-28 00:00:00 |          52 |
| 2008-12-29 00:00:00 |          52 |
| 2008-12-30 00:00:00 |          52 |
| 2008-12-31 00:00:00 |          52 |
| 2009-01-01 00:00:00 |           0 |
| 2009-01-02 00:00:00 |           0 |
| 2009-01-03 00:00:00 |           0 |
| 2009-01-04 00:00:00 |           1 |
| 2009-01-05 00:00:00 |           1 |
+---------------------+-------------+
10 rows in set (0.00 sec)

As you can see, we get four different weeks for a timespam of only ten days! Apparently, MySQL counts the first days of the year that do not belong to week 1 as week 0. This was certainly not what I expected, as I'm used to calendars that display the last days of the previous year and the first days of the new year as week 1.

MySQL's default WEEK() function could have caused serious data skew for me in this case. Fortunately, using the WEEK() function was too slow (as it had to calculate the weeknumber for each row in the result due to a WHERE WEEK(column) BETWEEN x AND y clause in my query), so we calculated the weeknumber using Unix timestamps ourselves. That's when we found the error. When using the WHERE clause mentioned, and the above data, we could have gotten:

mysql> SELECT * FROM test WHERE WEEK(start) BETWEEN 1 AND 52;
+---------------------+
| start               |
+---------------------+
| 2008-12-27 00:00:00 |
| 2008-12-28 00:00:00 |
| 2008-12-29 00:00:00 |
| 2008-12-30 00:00:00 |
| 2008-12-31 00:00:00 |
| 2009-01-04 00:00:00 |
| 2009-01-05 00:00:00 |
+---------------------+
7 rows in set (0.00 sec)

Only 7 rows are returned, while we should have gotten 10.

It turns out that MySQL's WEEK() function can operate in eight different ways. From the manual:

WEEK(date[,mode])

This function returns the week number for date. The two-argument form of WEEK() allows you to specify whether the week starts on Sunday or Monday and whether the return value should be in the range from 0 to 53 or from 1 to 53. If the mode argument is omitted, the value of the default_week_format system variable is used.

The following table describes how the mode argument works.

+--------------------------------------------------------------------+
| Mode | First day of week | Range | Week 1 is the first week ...    |
|------+-------------------+-------+---------------------------------|
| 0    | Sunday            | 0-53  | with a Sunday in this year      |
|------+-------------------+-------+---------------------------------|
| 1    | Monday            | 0-53  | with more than 3 days this year |
|------+-------------------+-------+---------------------------------|
| 2    | Sunday            | 1-53  | with a Sunday in this year      |
|------+-------------------+-------+---------------------------------|
| 3    | Monday            | 1-53  | with more than 3 days this year |
|------+-------------------+-------+---------------------------------|
| 4    | Sunday            | 0-53  | with more than 3 days this year |
|------+-------------------+-------+---------------------------------|
| 5    | Monday            | 0-53  | with a Monday in this year      |
|------+-------------------+-------+---------------------------------|
| 6    | Sunday            | 1-53  | with more than 3 days this year |
|------+-------------------+-------+---------------------------------|
| 7    | Monday            | 1-53  | with a Monday in this year      |
+--------------------------------------------------------------------+

So what we really wanted was the WEEK(column, 3); mode:

mysql> SELECT start, WEEK(start, 3) FROM test;
+---------------------+----------------+
| start               | WEEK(start, 3) |
+---------------------+----------------+
| 2008-12-27 00:00:00 |             52 |
| 2008-12-28 00:00:00 |             52 |
| 2008-12-29 00:00:00 |              1 |
| 2008-12-30 00:00:00 |              1 |
| 2008-12-31 00:00:00 |              1 |
| 2009-01-01 00:00:00 |              1 |
| 2009-01-02 00:00:00 |              1 |
| 2009-01-03 00:00:00 |              1 |
| 2009-01-04 00:00:00 |              1 |
| 2009-01-05 00:00:00 |              2 |
+---------------------+----------------+
10 rows in set (0.00 sec)

So, take care when using MySQL's WEEK() function, and always make sure to read the manual on all the functions you use at least once.

Easy PEAR Package creation

Sunday, April 12th, 2009

Here's a fairly simple way of creating PEAR package so you can distribute your application or library as a PEAR package.

Premise.

Let's assume you've written an application or library called 'MyApp'. The following is a possible directory structure layout:

myapp/
  |- MyApp.php
  \- MyAap/
       \- MyApp/MyLib.php

Install the PackageFileManager.

Since writing a package file by hand is way too much work, we first install the PEAR_PackageFileManager utility. This PEAR package includes a commandline utility which will guide you through the steps needed to create a proper package.xml file for your application or library.

# pear install pear install PEAR_PackageFileManager_Cli-0.3.0
Ignoring installed package pear/pear
No releases available for package "pear.php.net/install"
Cannot initialize 'channel://pear.php.net/install', invalid or missing package file
WARNING: channel "pear.php.net" has updated its protocols, use "channel-update pear.php.net" to update
Did not download optional dependencies: pear/PHP_CompatInfo, use --alldeps to download automatically
pear/PEAR_PackageFileManager can optionally use package "pear/PHP_CompatInfo" (version >= 1.4.0)
downloading PEAR_PackageFileManager_Cli-0.3.0.tgz ...
Starting to download PEAR_PackageFileManager_Cli-0.3.0.tgz (8,051 bytes)
.....done: 8,051 bytes
downloading PEAR_PackageFileManager-1.6.3.tgz ...
Starting to download PEAR_PackageFileManager-1.6.3.tgz (82,393 bytes)
...done: 82,393 bytes
Package "channel://pear.php.net/install" is not valid
install ok: channel://pear.php.net/PEAR_PackageFileManager-1.6.3
install ok: channel://pear.php.net/PEAR_PackageFileManager_Cli-0.3.0

If your PEAR is outdated, you will see something like:

pear/PEAR_PackageFileManager requires PEAR Installer (version >= 1.8.0alpha1), installed version is 1.7.1

In that case, you can upgrade your PEAR like so:

# pear channel-update pear.php.net
Updating channel "pear.php.net"
Update of Channel "pear.php.net" succeeded
# pear upgrade PEAR
...
upgrade ok: channel://pear.php.net/PEAR-1.8.1

You may also run into problems with alpha/beta state packages required for installation of the PackageFileManager CLI package. For instance, I got the following error: Failed to download pear/XML_Serializer within preferred state "stable", latest release is version 0.19.2, stability "beta", use "channel://pear.php.net/XML_Serializer-0.19.2" to install

In that case, you must manually install the beta packages:

# pear install channel://pear.php.net/XML_Serializer-0.19.2
downloading XML_Serializer-0.19.2.tgz ...
Starting to download XML_Serializer-0.19.2.tgz (39,703 bytes)
..........done: 39,703 bytes
downloading XML_Parser-1.3.2.tgz ...
Starting to download XML_Parser-1.3.2.tgz (16,260 bytes)
...done: 16,260 bytes
install ok: channel://pear.php.net/XML_Parser-1.3.2
install ok: channel://pear.php.net/XML_Serializer-0.19.2

Generate a package.xml file.

You can now use the PackageFileManager command-line utility to bootstrap a package.xml file for your PEAR package.

Change to the directory that is the root of your package:

$ cd myapp/

myapp$ ls
MyApp  MyApp.php

myapp$ pfm

PEAR Package File Manager Command Line Tool

Please enter the location of your package [.]*: .

Creating a new package file ...

Enter the base install directory*: .

Enter the name of the package [myapp]*: MyApp

Channel or URI based package? [c] (c,u)*: u

Enter the package URI*: http://www.example.com/download/PEAR/

Enter a 1 line summary*: My first PEAR example application.

Enter a description* (2 blank lines to finish):
MyApp is my first PEAR example application.
It rocks.

Enter the release version*: 0.1.0

Enter the API version [0.1.0]*: 

Choose a release stability [alpha] (alpha,beta,stable)*:  

Choose an API stability [alpha] (alpha,beta,stable)*: 

Enter any release notes* (2 blank lines to finish):
Initial release

Enter the minimum PHP version [5]*: 

Enter the minimum PEAR Installer version [1.4.0]*: 

Please choose a license from one of the following options

    1) Apache
    2) BSD Style
    3) LGPL
    4) MIT
    5) PHP

Please choose an option: 4

How many maintainers?*: 1

What type of maintainer is #1? [lead] (lead,developer,contributor,helper)*:     

Enter maintainer #1's name*: Ferry Boender

Enter maintainer #1's username*: fboender

Enter maintainer #1's email [fboender@php.net]*: ferry.boender@example.com

PEAR Package File Manager Command Line Tool

    1. Package name                 [MyApp]
    2. Channel/URI                  [URI: http://www.example.com/download/PEAR/]
    3. Summary                      [My first PEAR example application.]
    4. Description                  [MyApp is my first PEAR example applic...]
    5. Maintainers
    6. Version                      [Release: 0.1.0 API: 0.1.0]
    7. Stability                    [Release: alpha API: alpha]
    8. License                      [MIT]
    9. Notes                        [Initial release]
   10. Dependencies
   11. Tasks
   12. Regenerate contents
   13. Echo package file to stdout
   14. Save & Quit
   15. Quit without saving          (ctrl-c)

Please choose an option from the menu: 14

Analyzing MyApp/MyLib.php
Analyzing MyApp.php

This generates a package.xml for you, containing details for the PEAR package creation.

Create a PEAR package.

PEAR can use this package.xml file to create a PEAR package for you:

myapp$ pear package package.xml 
Analyzing myapp/MyApp.php
Analyzing myapp/MyApp/MyLib.php
Package MyApp-0.1.0.tgz done

Testing and (un)installing your package

Test the installation of the package:

myapp$ pear install ./MyApp-0.1.0.tgz
pear install ./MyApp-0.1.0.tgz
install ok: channel://__uri/MyApp-0.1.0

It appears to work. Make sure to check the PEAR directory to see if it placed the files in the right spot:

myapp$ ls /usr/share/php/
Archive  MyApp      OS    PEAR.php  Structures  adodb  doc   pearcmd.php  php-shell-cmd.php
Console  MyApp.php  PEAR  PHP       System.php  data   docs  peclcmd.php

The MyApp.php file, and the MyApp directory have been added to the PEAR library directory. If their PHP include_path is set correctly, users of your package should now be able to include it using:

include('MyApp.php');

You can uninstall the package using the pear uninstall command:

$ pear uninstall channel://__uri/MyApp-0.1.0
uninstall ok: channel://__uri/MyApp-0.1.0

Distribute your package

You can now put your PEAR package online, and point users to it by mentioning they can install it using PEAR with the command:

$ pear install http://www.example.com/download/PEAR/MyApp-0.1.0.tgz

Templum: Simple PHP Templating

Sunday, April 5th, 2009

At the company I work for (ZX), we needed a simple yet powerful templating language for PHP. I googled around a bit for something, but we couldn't really find anything. So I wrote Templum, and ZX was kind enough to allow me to release it as Open Source under the MIT License.

From the Templum homepage:

Templum is an extremely lightweight, simple yet powerful and fast templating engine for PHP. It re-uses the power of PHP itself for rendering templates, but provides additional features making it easier to write templating code. Rendering templates using Templum is very fast; it approximates native PHP rendering speed for include() statements.

You can install it using PEAR:

pear install http://templum.electricmonk.nl/releases/templum-0.1.0.tgz

For some examples, check the website.