technology from back to front

Archive for October, 2008

Erlang’s GC only runs when you’re running

Recently, as part of RabbitMQ server development, we ran into an interesting issue regarding Erlang’s per-process garbage collection. If a process is idle — not doing any work at all, simply waiting for an external event — then its garbage-collector will not run until it starts working again. The solution is to hibernate idle processes, which causes a very aggressive garbage-collection run and puts the process into a suspended state, from which it will wake when a message next arrives.

We were implementing a form of flow control in RabbitMQ server, essentially equivalent to XON/XOFF, and when a channel was paused, we found that its memory usage wasn’t decreasing in proportion to its decreased activity. As soon as we started hibernating processes, the memory usage dropped right back.


Where did all my space go?

Over the last little while, I’ve started to suffer from lack of space on the hard disk in my laptop, which is ridiculous, since there’s an 80GB disk in there and there is no way I have that much data I need to hang on to. I decided to do something about it last week. The main part of the problem was to figure out what was eating all the space: du tells you exactly what’s using how much, but it’s hard to get a feel for where your space has gone by scanning through pages of du output. So I built a program to help.

spaceviz is a small Python program that takes the output of du -ak, and builds you a picture and HTML client-side imagemap of your space usage

Running it against the output of du -ak / showed me very clearly where all the space had gone: not only did I have a few seasons of various TV shows on my disk (which I already knew were there), but I had 11 GB of unneeded gzipped RDF data left over from a project that finished earlier this year (that I had forgotten about). Instant win!

To run it for yourself, check out the mercurial repository, and run

make veryclean all ROOT=/

replacing the ROOT=/ with a definition of ROOT that points at the directory tree you want to generate usage data for. The makefile will take care of running du and for you. Edit the settings for WIDTH and HEIGHT in to change the dimensions of the generated picture.

The program runs not only on Linux without fuss, but also on OS X so long as you have the netpbm port installed to convert the python-generated PPM file to a browser-accessible (and much more efficiently compressed!) PNG.


Toy raytracer in Python

I spent a few days in Wellington recently with my 12-year-old brother. We somehow got to talking about ray tracing, and so we ran through a bit of linear algebra (vectors, normals, and dot products, basically) and built ourselves a raytracer in Python as a lark. We collaborated on the Vector and Point classes, and I filled in the rest of the program. There’s still a bug in the checkerboard code (see the odd horizontal lines in the reflection of the checkerboard in the yellow sphere?) but it’s not too awful for an afternoon’s idle hacking.

The code is contained within, and here’s the output it generates:

Raytracer output

Wouldn’t it be nice to have classes like Vector and Point as part of the standard distribution of Python? Even better, though, and perhaps less subject to bike-shed debates, would be the inclusion of a simple, small PNG-writing module like this one as part of standalone base Python!




You are currently browsing the LShift Ltd. blog archives for October, 2008.



2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us