Our Software: mercurial-server needs a long-term Debian sponsor

Thanks to the patient work of Jakub Wilk, mercurial-server 1.2-1 has hit the Debian “unstable” repository, where all being well it should make its way into testing, stable, Ubuntu and so forth. Jakub stepped in at the last minute when I discovered that the project’s previous sponsor, Steve Kemp, had resigned as a Debian developer in April.

I’d like to arrange a longer-term Debian sponsor for mercurial-server now, in advance of the next release, so that we can make sure we fix any issues with the package in advance and we’re ready when release time comes. If there are any Debian developers out there who use mercurial-server, please do get in touch - thanks!

Our Software: mercurial-server 1.2 released

Version 1.2 of mercurial-server is now available. This fixes a security problem, adds compatibility with Mercurial 1.9 and fixes incompatibilities with older versions of Python, adds MQ compatibility, and some other minor things.

Unfortunately it may not immediately enter Debian, because my former sponsor is no longer a Debian developer. If you’re an official Debian developer and you’d like to sponsor this package, please get in touch - thanks!

mercurial-server on mentors.debian.net.

Our Software: mercurial-server version 1.1 released

Version 1.1 of mercurial-server is now available. This permits one repository to be in a subdirectory of another, which makes subrepositories more convenient, as well as improvements to logging and documentation. I’ve also included the collection of scripts I use for testing under the “dev” subdirectory in the main repository, though I’d want to clean these up somewhat before including them in the release tarballs.

The Debian version also includes fixes for all the outstanding bugs against it in Debian; I’ve sent the files off to the sponsor, so hopefully that’ll enter Debian testing soon.

Our Software: SpringSource / VMWare Acquire Rabbit Technologies

SpringSource, a division of VMware, Inc. today announced the acquisition by VMware of Rabbit Technologies, Ltd, a company set up by LShift and partners Monadic and CohesiveFT.

Read the full story

Our Software: On the limits of concurrency: Worker Pools in Erlang

A worker pool is a very common pattern, and they exist in the standard libraries for many languages. The idea is simple: submit some sort of closure to a service which commits to running the closure in the future in some thread. Normally the work is shared out among many different threads and in the absence of anything fancier, one assumes a first-come-first-served queue of closures.

Erlang, with its light-weight process model is not a language which you would expect would require such an approach: processes are dirt cheap, and the scheduler maps processes onto threads when they are ready to be run — in many ways, the ErlangVM is a glorified implementation of a worker pool, only one that does pre-emption and other fancy features, in a very similar way to an OS kernel. However, we recently found in RabbitMQ a need for a worker pool. (more…)

Our Software: The fine art of holding a file descriptor

People tend to like certain software packages to be scalable. This can have a number of different meanings but mostly it means that as you throw more work at the program, it may require some more resources, in terms of memory or CPU, but it nevertheless just keeps on working. Strangely enough, it’s fairly difficult to achieve this with finite resources. With things like memory, the classical hierarchy applies: as you use up more and more faster memory, you start to spill to slower memory — i.e. spilling to disk. The assumption tends to be that one always has enough disk space.

Other resources are even more limited, and are harder to manage. One of these is file descriptors. (more…)

Our Software: Publishing your mercurial-server repositories to the Web

I got a couple of queries recently on how to make your mercurial-server repositories publically readable over HTTP. Happily this isn’t hard to do, and doesn’t really touch on mercurial-server itself. Here’s how we do it on our Debian systems; in what follows I assume that you have installed mercurial-server on hg.example.com, and that you’re not already using that machine as a web server for anything else. First install these packages; note that they tend to have a lot of stuff you don’t need marked as recommended, so don’t install those things:

apt-get --no-install-recommends install apache2 libapache2-mod-fcgid python-flup

Create the following four files:

/etc/mercurial-server/hgweb.config:

[collections]
/var/lib/mercurial-server/repos = /var/lib/mercurial-server/repos

/etc/mercurial-server/hgweb.hgrc:

[web]
style = gitweb
allow_archive = bz2 gz zip
baseurl = http://hg.example.com/
maxchanges = 200

/etc/mercurial-server/hgwebdir.fcgi:

#!/usr/bin/env python

from mercurial import demandimport; demandimport.enable()

import os
os.environ["HGENCODING"] = “UTF-8″
os.environ["HGRCPATH"] = “/etc/mercurial-server/hgweb.hgrc”

from mercurial.hgweb.hgwebdir_mod import hgwebdir
from mercurial.hgweb.request import wsgiapplication
from flup.server.fcgi import WSGIServer

def make_web_app():
    return hgwebdir(”/etc/mercurial-server/hgweb.config”)

WSGIServer(wsgiapplication(make_web_app)).run()

/etc/apache2/sites-available/hg:

<VirtualHost *>
    ServerName hg.example.com
    AddHandler fcgid-script .fcgi
    ScriptAlias / /etc/mercurial-server/hgwebdir.fcgi/
    ErrorLog /var/log/apache2/hg/error.log
    LogLevel warn
    CustomLog /var/log/apache2/hg/access.log combined
</VirtualHost>

Finally run these commands as root:

chmod +x /etc/mercurial-server/hgwebdir.fcgi
mkdir -p /var/log/apache2/hg
cd /etc/apache2/sites-enabled
rm 000-default
ln -s ../sites-available/hg
/etc/init.d/apache2 reload

Your files should now be served at http://hg.example.com/ . Sadly because of a design flaw in hgwebdir, there’s no easy way to get Apache to handle the static files it needs, but these are pretty small so there’s no harm in letting hgwebdir handle them. The “rm 000-default” thing seems pretty undesirable, but without it I can’t seem to get this recipe to work.

I’ve chosen FastCGI as the connector. This has the advantage that

• unlike CGI, it doesn’t fork a new handler on every request
• unlike mod_python, it keeps your Mercurial handler separate from your web server
• unlike SCGI, it will automatically start the service for you if it’s not already running, which is a massive convenience

I’m not aware of any other way of working that offers all three advantages.

As soon as a version of lighttpd with this bug fixed makes it into Debian, I’ll add my recipe for that.

Our Software: Memory matters - even in Erlang

Some time ago we got an interesting bug report for RabbitMQ. Surprisingly, unlike other complex bugs, this one is easy to describe: 

At some point basic.get suddenly starts being very slow - about 9 times slower!

(more…)

Our Software: RabbitMQ-shovel: Message Relocation Equipment

In several applications, it’s very useful to be able to take messages out of one RabbitMQ broker, and insert them into another. Many people on our mailing list have being asking for such a shovel, and we’ve recently been able to devote some time to writing one. This takes the form of a plugin for Rabbit, and whilst it hasn’t been through QA just yet, we’re announcing it so people who would like to play and even suggest further features for inclusion can do so sooner rather than later.

The shovel is written on top of the Erlang client. It supports both direct and network connections to nodes, SSL support, the ability to declare resources on nodes it connects to, basic round-robinrabbit balancing of both source and destination nodes, and allows you to configure many parameters controlling how messages are consumed from the source, and how they’re published to the destination. Multiple shovels can be specified, their statuses queried, and shovels can repeatedly reconnect to nodes in the event of failure.

The plugin is available from http://hg.rabbitmq.com/rabbitmq-shovel/, and is released under the MPL v1.1. There is a README included which contains full documentation. This is replicated below. (more…)

Our Software: Plugin exchange types for RabbitMQ

An obvious extension point for an AMQP broker is the addition of new types of exchange. An exchange type essentially represents an algorithm for dispatching messages to queues, usually based on the message’s routing key, given how the queues are bound to the exchange — it’s a message routing algorithm.

At a minimum, supporting new exchange types requires only some scaffolding to plug in to (an exchange type registry) and a hook for routing messages. However, this wouldn’t support some more interesting use cases, and in particular it didn’t support our motivating use case. Exchange types that want to keep their own state need to be initialised, and be notified about other lifecycle events. (more…)