apt-get --no-install-recommends install apache2 libapache2-mod-fcgid python-flup

Create the following four files:

/etc/mercurial-server/hgweb.config:

[collections]
/var/lib/mercurial-server/repos = /var/lib/mercurial-server/repos

/etc/mercurial-server/hgweb.hgrc:

[web]
style = gitweb
allow_archive = bz2 gz zip
baseurl = http://hg.example.com/
maxchanges = 200

/etc/mercurial-server/hgwebdir.fcgi:

#!/usr/bin/env python

from mercurial import demandimport; demandimport.enable()

import os
os.environ["HGENCODING"] = “UTF-8″
os.environ["HGRCPATH"] = “/etc/mercurial-server/hgweb.hgrc”

from mercurial.hgweb.hgwebdir_mod import hgwebdir
from mercurial.hgweb.request import wsgiapplication
from flup.server.fcgi import WSGIServer

def make_web_app():
    return hgwebdir(”/etc/mercurial-server/hgweb.config”)

WSGIServer(wsgiapplication(make_web_app)).run()

/etc/apache2/sites-available/hg:

<VirtualHost *>
    ServerName hg.example.com
    AddHandler fcgid-script .fcgi
    ScriptAlias / /etc/mercurial-server/hgwebdir.fcgi/
    ErrorLog /var/log/apache2/hg/error.log
    LogLevel warn
    CustomLog /var/log/apache2/hg/access.log combined
</VirtualHost>

Finally run these commands as root:

chmod +x /etc/mercurial-server/hgwebdir.fcgi
mkdir -p /var/log/apache2/hg
cd /etc/apache2/sites-enabled
rm 000-default
ln -s ../sites-available/hg
/etc/init.d/apache2 reload

Your files should now be served at http://hg.example.com/ . Sadly because of a design flaw in hgwebdir, there’s no easy way to get Apache to handle the static files it needs, but these are pretty small so there’s no harm in letting hgwebdir handle them. The “rm 000-default” thing seems pretty undesirable, but without it I can’t seem to get this recipe to work.

I’ve chosen FastCGI as the connector. This has the advantage that

• unlike CGI, it doesn’t fork a new handler on every request
• unlike mod_python, it keeps your Mercurial handler separate from your web server
• unlike SCGI, it will automatically start the service for you if it’s not already running, which is a massive convenience

As soon as a version of lighttpd with this bug fixed makes it into Debian, I’ll add my recipe for that.

It turns out, though, that there is an easy solution.

Using some other Add-Ins as examples (such as JetBrain’s ReSharper) the simplest strategy seems to be:

1. Install your AddIn into a standard application install directory (eg, the “Application Folder” pointed at [TARGETDIR] in your Visual Studio Setup project, or C:\Program Files\AppName if doing this manually)
2. Create a string value in HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\AutomationOptions\LookInFolders in the registry, with key [TARGETDIR] and value being something descriptive for your application. Note that [TARGETDIR] is a special substituted variable in Visual Studio Setup projects - if you’re doing this manually, you should instead use the actual path to the directory containing your .addin file (eg C:\Program Files\AppName)

When you start Visual Studio, your AddIn should be loaded. You can also verify this path by opening Tools -> Options, and looking at the list in Environment -> Add-in/Macros Security.

This solution also has the advantage of not placing application files in user directories - hence preventing the user accidentally removing them, and allowing Windows to apply normal Program File protections to the files too.

At some point basic.get suddenly starts being very slow - about 9 times slower!

Basic.get doesn’t do anything complex - it just pops a message from a queue. This behaviour was quite unexpected. Our initial tests confirmed that we have a problem when a queue contains thousands of elements:

queue_length: 90001  basic_get 3333 times took: 1421.250ms
queue_length: 83335  basic_get 3333 times took: 1576.664ms
queue_length: 60004  basic_get 3333 times took: 1403.086ms
queue_length: 53338  basic_get 3333 times took: 9659.434ms [ look at that! ]
queue_length: 50005  basic_get 3333 times took: 9885.598ms
queue_length: 46672  basic_get 3333 times took: 8562.136ms

Let me repeat that. Usually popping a message from a queue takes Xms. At some point, it slows down to 9*Xms.

It turned out that the problem is with the queue:len() function, which is executed during the basic.get. Actually, queue:len() calls only erlang:length() builtin. At some point it switches to the “slow” mode. 

Erlang:length() is a builtin that iterates through a linked list and counts it’s length. It’s complexity is O(N), where N is the length of the list. This function is implemented in the VM so it’s expected to be very, very fast.

The problem is not with erlang:length() being slow. It’s about being unpredictably slow. Let’s take a look at Erlang interpreter source code (erl_bif_guard.c:erts_gc_length_1). Here’s the main loop for erlang:length():

i=0
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}

It does nothing unusual - it just iterates through list elements. However, recompiling Erlang with some debugging information confirms that the problem is indeed here:

clock_gettime(CLOCK_REALTIME, &t0);
while (is_list(list)) {
    i++;
    list = CDR(list_val(list));
}
clock_gettime(CLOCK_REALTIME, &t1);
td_ms = TIMESPEC_NSEC_SUBTRACT(t1, t0) / 1000000.0;
if (i > 200000 || td_ms > 2.0) {
    fprintf(stderr, "gc_length_1(%p)=%i %.3fms\n\r", reg[live], i, td_ms);
}

gc_length_1(0x7f4dbfa7fc19)=499999 2.221ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.197ms
gc_length_1(0x7f4dbfa7fc19)=499999 2.208ms
(hibernation)
gc_length_1(0x7f4db0572049)=499999 13.793ms
gc_length_1(0x7f4db0572049)=499999 12.806ms
gc_length_1(0x7f4db0572049)=499999 12.531ms

This confirms Matthias’ initial guess - the slowdown starts after Erlang process hibernation.

For those who aren’t Erlang experts: Hibernation is an operation that compacts an Erlang process. It does aggressive garbage collection and reduces the memory footprint of a process to absolute minimum.

The intended result of hibernation is recovering free memory from the process. However its side effect is a new memory layout of objects allocated on the heap.

Ah, how could I have forgotten! The memory is nowadays slow! What happens, is that before hibernation list elements are aligned differently, more dense. Whereas after hibernation they are sparse. It’s easy to test it - let’s count the average distance between pointers to list elements:

gc_length_1(0x7f5c626fbc19)=499999 2.229ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.349ms avg=16.000 dev=0.023
gc_length_1(0x7f5c626fbc19)=499999 3.345ms avg=16.000 dev=0.023
(hibernation)
gc_length_1(0x7f5c61f7d049)=499999 13.800ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.726ms avg=136.000 dev=0.266
gc_length_1(0x7f5c61f7d049)=499999 12.367ms avg=136.000 dev=0.266

• Before hibernation list elements are aligned exactly one after another, values are somewhere else.
• After hibernation list elements are interleaved with values. 

This behavior does make sense. In most cases when you traverse the list, you actually do something with the values. After hibernation, when you access list item, the value will be already loaded to the CPU cache.

Knowing the mechanism, it’s easy to write a test case that reproduces the problem.

The average distance between pointers in my case is constant - the standard deviation is negligible. This information has a practical implication - we can “predict” where the next pointer will be. Let’s use that information to “fix” the Erlang VM by prefetching memory!

while (is_list(list)) {
    i++;
    list2 = CDR(list_val(list));
    __builtin_prefetch((char*)list2 + 128*((long)list2-(long)list));
    list = list2;
}

length: 300001  avg:0.888792ms dev:0.061587ms
length: 300001  avg:0.881030ms dev:0.040961ms
length: 300001  avg:0.875158ms dev:0.019436ms
hibernate
length: 300001  avg:14.861762ms dev:0.150635ms
length: 300001  avg:14.833733ms dev:0.017405ms
length: 300001  avg:14.884861ms dev:0.220119ms

length: 300001  avg:0.742822ms dev:0.029322ms
length: 300001  avg:0.739149ms dev:0.012897ms
length: 300001  avg:0.739465ms dev:0.014417ms
hibernate
length: 300001  avg:7.543693ms dev:0.284355ms
length: 300001  avg:7.342802ms dev:0.330158ms
length: 300001  avg:7.265960ms dev:0.053176ms

The test runs only a tiny bit faster for the “fast” case (dense conses) and twice as fast for the “slow” case (sparse conses).

Should this patch be merged into mainline Erlang? Not really. I have set the prefetch multiplier value to 128 and I don’t even know if it’s optimal. This was only an experiment. But it was fun to see how low-level system architecture can affect high-level applications.

The Java world has had a solution to this problem for a long time, in the form of Maven and Ivy. Remote servers store the binaries, and the build tool automatically downloads them on demand. WebGAC adds the core of this functionality to .NET, but without requiring you to switch build tools, or maintain a separate configuration file. Dependencies are specified just the same way as normal, but if you don’t have them when building your project, WebGAC will fetch them for you automatically.

WebGAC is available at http://github.com/paulj/webgac. Browse over there for more information and installation instructions, or continue reading here for more details.

The Remote Server

A WebGAC remote server is as simple as an Apache HTTP server running the DAV extensions. The DAV extensions provide the ability to query (in a structured form) for directory lists, and to upload new files remotely.

WebGAC in MSBuild and Visual Studio

To do its thing, WebGAC requires that a custom Import of targets is added to each project. This is as simple as adding the following block:

<Import Project="$(MSBuildExtensionsPath)\WebGAC\WebGAC.targets" />

This import adds all the necessary hooks for doing remote dependency resolution using your project’s normal references. Any missing assemblies will be automatically downloaded and inserted into your build.

The WebGAC Visual Studio add-in provides mechanisms for configuring the WebGAC extension (eg, setting up remote servers), browsing the remote servers and uploading binaries to the remote servers.

The imported MSBuild targets file also includes targets that allow the assembly produced by the current project to be uploaded to a WebGAC - thus allowing for the release of binaries either from developer machines or from a continuous integration environment.

The Gory Details

This section discusses how WebGAC actually does what it does. If you’re just interested in using WebGAC, feel free to skip it.

As a standard property within the MSBuild environment for a .NET project, the AssemblySearchPaths property is defined as:

 <AssemblySearchPaths>
      {CandidateAssemblyFiles};
      $(ReferencePath);
      {HintPathFromItem};
      {TargetFrameworkDirectory};
      {Registry:$(FrameworkRegistryBase),$(TargetFrameworkVersion),$(AssemblyFoldersSuffix)$(AssemblyFoldersExConditions)};
      {AssemblyFolders};
      {GAC};
      {RawFileName};
      $(OutputPath)
    </AssemblySearchPaths> 

The WebGAC targets import overrides this, by inserting a {WebGAC} target after the {GAC} element. It also overrides the ResolveAssemblyReferencesDependsOn variable (which defines the standard set of tasks to execute when preparing Assembly Resolution) to execute the target AddWebGACAssemblySearchPaths. This target will query it’s own cache (and any configured remote web servers) for the project’s references. If they are available via the WebGAC, it will download the files and insert the local cache directories into the search paths. MSBuild and Visual Studio will treat these new paths as if they came from any other standard part of the build system, and incorporate them transparently into the build. Visual Studio will even treat the assemblies as if they came from the GAC!

If WebGAC sounds like it could be useful to you on your .NET project, head over to http://github.com/paulj/webgac. There is even a pre-built installer to get you going quickly!

Firstly install Leiningen, this is a simple build tool for Clojure, follow the instructions on the web page and place the lein somewhere on your executable path.

Now use leiningen to create a new clojure project; I always name my projects after vegetables so we will call this one sprout. Execute this command at the command line:

% lein new sprout

sprout% lein deps
sprout% lein test

The first command will download some dependent libraries and the second command will run a default unit test that should fail like this:

     [null] Testing sprout.core-test
     [null] FAIL in (replace-me) (core_test.clj:6)
     [null] expected: false
     [null]   actual: false
     [null] Ran 1 tests containing 1 assertions.
     [null] 1 failures, 0 errors.
     [null] ——————–
     [null] Total:
     [null] Ran 1 tests containing 1 assertions.
     [null] 1 failures, 0 errors.

If this all goes successfully we have a project ready to work with. Working with Clojure is made a lot easier if you have a working REPL to interact with, create a file called clj in the project directory containing a script like this:

#!/bin/bash
    LIBS=`printf "%s:" lib/*.jar | sed s/:$//`
    if [ $# -eq 0 ]; then
         java -cp $LIBS:src clojure.main
    else
         java -cp $LIBS:src clojure.main $1 — $@
    fi

Make sure this script is executable and check that it works from the command line like this:

sprout%./clj
Clojure 1.1.0
user=> (+ 2 3)
5
user=> ^C
sprout%

Now edit the project.clj file to add dependencies for the Ring and Enlive libraries, the file should look like this:

(defproject sprout "1.0.0-SNAPSHOT"
  :description "Simple program to play with ring and enlive"
  :dependencies [[org.clojure/clojure "1.1.0"]
                 [org.clojure/clojure-contrib "1.1.0"]
                 [ring "0.1.1-SNAPSHOT"]
                 [enlive "1.0.0-SNAPSHOT"]])

Now use leiningen to download the dependencies:

sprout% lein deps

Now let us create a basic application that will serve some web pages, edit the file src/sprout/core.clj to look like this:

(ns sprout.core)

(defn router [req]
  (condp = (:uri req)
    "/error"
      {:status 404
       :headers {"Content-Type" "text/html"}
       :body    (str "Error")}
    "/info"
      {:status 200
       :headers {"Content-Type" "text/html"}
       :body    (str "This is Ring on " (:server-name req))
      }
    {:status 200
     :headers {"Content-Type" "text/html"}
     :body    (str "Welcome")}))

(defn app [req]
  (router req))

This contains two functions, the router function generates different pages dependent on the requested uri and the app function that forwards a request to the router. We can test this code with a unit test, edit the file test/sprout/core_test.clj and make it look like this:

(ns sprout.core-test
  (:use [sprout.core] :reload-all)
  (:use [clojure.test]))

(deftest not-found-error
  (let [req {:uri "/error"}
        resp (app req)]
    (is (= 404 (:status resp)))))

(deftest default-page
  (let [req {:uri "/index.html"}
        resp (app req)]
    (is (= 200 (:status resp)))))

(deftest info-page
  (let [req {:uri "/info"
             :server-name "test"}
        resp (app req)]
    (testing "Info page shown"
      (is (= 200 (:status resp)))
      (is (= "This is Ring on test" (:body resp))))))

You can run this test with lein test at the command line.

So far this is pure functional Clojure with no real web pages in site, just a map being passed into a function and a different map being returned, the maps happen to be keyed with keys that look a bit like http requests and responses. The Ring library wraps these functions with adapters that talk to real web containers we will use Jetty to demonstrate real web pages.

Create a new file in src/sprout called run.clj and edit it to appear like this:

(ns sprout.run
  (use [ring.adapter.jetty])
  (use [sprout.core]))

(run-jetty app {:port 8080})

Now from the command line do this:

sprout%./clj src/sprout/run.clj
2010-02-27 14:08:03.602::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2010-02-27 14:08:03.604::INFO:  jetty-6.1.14
2010-02-27 14:08:03.727::INFO:  Started SocketConnector@0.0.0.0:8080

Using a browser visit the urls http://localhost:8080, http://localhost:8080/info and http://localhost:8080/error. All being well you should see a welcome page, an info page and a 404 error.

Ring provides a further level of wrapper functions that enable html processing pipelines to be built using simple functions written in Clojure. The pages you have seen so far haven’t looked very impressive, they are not even valid HTML! We could embed the HTML in the code or write a library to generate HTML in Clojure but instead we will use Enlive.

Enlive is a selector based templating library, this differs from templating languages such as JSP or ASP in that the raw html remains untouched, functions are written that use css selectors to replace and augment the existing html. Create a file in the src/sprout directory called enlive.clj and edit it to appear like this:

(ns sprout.enlive
  (:use [net.cgrand.enlive-html]))

(deftemplate t1 "templates/t1.html" []
  [:h1]   (content "heading"))

(defn enlive-template []
  (apply str(t1)))

The deftemplate declaration creates a template called t1, that takes no parameters and substitutes the content of any h1 tags with the word heading. The function enlive-template just applies the template to generate HTML.

Now add a new mapping to the router defined in src/sprout/core.clj to use this function, the file should now look like this:

(ns sprout.core
  (:use sprout.enlive))

(defn router [req]
  (condp = (:uri req)
    "/error"
      {:status 404
       :headers {"Content-Type" "text/html"}
       :body    (str "Error")}
    "/info"
      {:status 200
       :headers {"Content-Type" "text/html"}
       :body    (str "This is Ring on " (:server-name req))
      }
    "/enlive"
      {:status 200
       :headers {"Content-Type" "text/html"}
       :body (enlive-template)
      }
    {:status 200
     :headers {"Content-Type" "text/html"}
     :body    (str "Welcome")}))

(defn app [req]
  (router req))

Running the run script again:

prout%./clj src/sprout/run.clj
2010-02-27 15:10:35.476::INFO:  Logging to STDERR via org.mortbay.log.StdErrLog
2010-02-27 15:10:35.479::INFO:  jetty-6.1.14
2010-02-27 15:10:35.528::INFO:  Started SocketConnector@0.0.0.0:8080

Now go to http://localhost:8080/enlive and an HTML page should be visible.

This has been an extremely basic introduction to using Ring and Enlive with Clojure to serve up web pages. I’ll be expanding on this example at a later date to dive deeper into the functionality of Ring, Enlive and Clojure.

The shovel is written on top of the Erlang client. It supports both direct and network connections to nodes, SSL support, the ability to declare resources on nodes it connects to, basic round-robinrabbit balancing of both source and destination nodes, and allows you to configure many parameters controlling how messages are consumed from the source, and how they’re published to the destination. Multiple shovels can be specified, their statuses queried, and shovels can repeatedly reconnect to nodes in the event of failure.

The plugin is available from http://hg.rabbitmq.com/rabbitmq-shovel/, and is released under the MPL v1.1. There is a README included which contains full documentation. This is replicated below.

RabbitMQ-shovel

Introduction

This is a plug-in for RabbitMQ that shovels messages from a queue on one broker to an exchange on another broker. The two brokers may be the same. The plug-in allows several shovels to be specified at the same time. Each shovel may have a number of source and destination brokers specified, and one of each is chosen whenever the shovel attempts to make a connection: this permits simple round-rabbit load balancing.

Resources can be declared upon connection to both the source and destination brokers, and parameters can be specified for both the reception and publishing of messages.

Requirements

Currently, you must build the server from source, under branch bug16653. You must also have checked out the rabbitmq-public-umbrella hg repository, and have the rabbitmq-erlang-client built. From scratch, the following commands should build RabbitMQ with the shovel plug-in:

hg clone http://hg.rabbitmq.com/rabbitmq-public-umbrella
cd rabbitmq-public-umbrella
hg clone http://hg.rabbitmq.com/rabbitmq-codegen
hg clone http://hg.rabbitmq.com/rabbitmq-erlang-client
hg clone http://hg.rabbitmq.com/rabbitmq-server
hg clone http://hg.rabbitmq.com/rabbitmq-shovel
cd rabbitmq-server
hg up -C bug16653
make -j
mkdir -p plugins
cd plugins
ln -s ../../rabbitmq-erlang-client
ln -s ../../rabbitmq-shovel
cd ../../rabbitmq-erlang-client
make
cd ../rabbitmq-shovel
make
cd ../rabbitmq-server
./scripts/rabbitmq-activate-plugins
make cleandb run

Configuration

The RabbitMQ configuration file specifies the shovel configurations. This exists by default, in /etc/rabbitmq/rabbitmq.config under Linux systems, %RABBITMQ_BASE%\rabbitmq.config under Windows or somewhere else under OS X. This file configures both RabbitMQ-server and all the plugins installed in it. It is an Erlang-syntax file of the form:

[{section1, [section1-config]},
 {section2, [section2-config]},
 …
 {sectionN, [sectionN-config]}
].

thus a list of tuples, where the left element of each tuple names the applications being configured. Don’t forget the last element of the list doesn’t have a trailing comma, and don’t forget the full-stop is needed after closing the list. Hence if you configure RabbitMQ-server and the RabbitMQ-shovel, then the configuration file may have a structure like this:

[{rabbit,        [configuration-for-RabbitMQ-server]},
 {rabbit-shovel, [configuration-for-RabbitMQ-shovel]}
].

A full example of the shovel configuration is:

{rabbit_shovel,
  [{shovels,
    [{my_first_shovel,
      [{sources,      [{brokers,
                          ["amqp://fred:secret@host1.domain/my_vhost",
                           "amqp://john:secret@host2.domain/my_vhost"
                          ]},
                       {declarations,
                          ['queue.declare',
                           {'queue.bind',
                                  [{exchange, <<"my_exchange">>},
                                   {queue,    <<>>}]}
                          ]}]},
       {destinations, [{broker, "amqp://"},
                       {declarations,
                          [{'exchange.declare',
                                  [{exchange, <<"my_exchange">>},
                                   {type, <<"direct">>},
                                   durable]}
                          ]}]},
       {queue, <<>>},
       {qos, 10},
       {auto_ack, false},
       {tx_size, 0},
       {delivery_mode, keep},
       {publish_fields, [{exchange, <<"my_exchange">>},
                         {routing_key, <<"from_shovel">>}]},
       {reconnect, 5}
      ]}
     ]
   }]
}

Firstly, all shovels are named. Here we have one shovel, called ‘my_first_shovel’. We can have multiple shovels if you wish. Every shovel must have all sub-fields specified: sources, destinations, qos, auto_ack, delivery_mode, publish_fields, reconnect.

Sources and Destinations

Sources and destinations specify respectively where messages are fetched from and delivered too. One of ‘broker’ and ‘brokers’ must be specified, and ‘broker’ is simply shorthand for when only one broker needs specifying. Using ‘brokers’ allows a list of brokers to be specified: whenever the connection to a broker is lost, another one is chosen at random from the list and a connection attempt is made to that. The syntax for broker URIs is:

amqp://username:password@host:port/vhost

If username or password are omitted, the default values of guest and guest are used. If the vhost is omitted, the default value of / is used. If the host is omitted, then the plugin uses the “direct” connection internally rather than a network connection: this means it connects to the RabbitMQ-server node on which it is running without going through the network stack. This is much more efficient. If port is omitted then the default value is used (5672 or 5671 if SSL is used).

SSL is implemented, for which additional parameters are needed:

amqps://username:password@host:port/vhost?cacertfile=/path/to/cacert.pem&certfile=/path/to/certfile.pem&keyfile=/path/to/keyfile.pem&verify=verifyOption&fail_if_no_peer_cert=failOption

(note, this is a single line)

All five parameters (3 paths: cacertfile, certfile and keyfile; 2 options: verify, fail_if_no_peer_cert) must be specified. See the SSL guide at http://www.rabbitmq.com/ssl.html#configure-erlang for details of SSL in RabbitMQ in general and specifically for the Erlang client (on which the shovel is built).

Note that SSL cannot be used with the direct connection (i.e. a host must be specified when using SSL), and that it is preferable to use the non-SSL direct connection when connecting to the same node that’s running the shovel.

Resource Declarations

Both sources and destinations can have an optional ‘declarations’ clause. The value of this is a list, consisting of AMQP Methods. If default values are sufficient, then the method name alone can be specified - e.g. ‘queue.declare’. If parameters need to be set then the method should be given as a tuple, with the right hand side a proplist specifying which fields need altering from their default values. E.g:

    {'exchange.declare',[{exchange, <<"my_exchange">>},
                         {type, <<"direct">>},
                         durable]},

One very useful feature here is the Most-Recently-Declared-Queue feature, in which RabbitMQ remembers the name of the most recently declared queue. This means that you can declare a private queue, and then bind it to exchanges without ever needing to know its name.

queue :: binary

This parameter specifies the name of the queue on the source brokers to consume from. This queue must exist. Use the resource declarations to create the queue (or ensure it exists) first. Note again that the Most-Recently-Declared-Queue feature can be used here, thus an anonymous queue can be used: use <<>> to indicate the Most-Recently-Declared-Queue.

qos :: non-negative-integer

The shovel consumes from a queue. The QoS controls how many messages are sent to the shovel in advance of the message the shovel is currently processing.

auto_ack :: boolean

Setting this to ‘true’ turns on the no_ack flag when subscribing to the source queue.

tx_size :: non-negative-integer

When set to 0, transactions are not used. Other values make publishes transactional, with a commit every N messages. In lieu of the auto-ack option, when transactions are not used, messages are acknowledged to the source immediately after every publish. When transactions are used, acks are only issued to the source on receipt of the commit-ok message from the destination. This can thus be used to guarantee that messages are only acknowledged (and thus forgotten about by the source broker) when they are guaranteed to have been received by the destination broker.

delivery_mode :: ‘keep’ | 0 | 2

This affects the delivery_mode field when publishing to the destination. A value of ‘keep’ means that the same delivery_mode should be used as when the message was originally published to the source broker. 0 and 2 override the original setting.

publish_fields

This is a list of tuples which override fields in the publish method when publishing to the destination. This can be used to direct messages to a particular exchange on the destination, for example, or change the routing key. By default, the routing key of the message as it is received by the shovel is passed through, but this can be overridden as necessary.

reconnect :: non-negative-integer

When an error occurs, the shovel will disconnect from both the source and destination broker immediately. This will force uncommitted transactions at the destination to be rolled back, and delivered but unacknowledged messages from the source to be requeued. The shovel will then try connecting again. If this is unsuccessful, then it’s not a good idea for the shovel to very quickly and repeatedly try to reconnect. The value specified here is the number of seconds to wait between each connection attempt.

Note that if set to 0, the shovel will never try to reconnect: it’ll stop after the first error.

Obtaining shovel statuses

From the broker Erlang prompt, call rabbit_shovel_status:status(). This will return a list, with one row for each configured shovel. Each row has three fields: the shovel name, the shovel status, and the timestamp (a local calendar time of {{YYYY,MM,DD},{HH,MM,SS}}). There are 3 possible statuses:

• ’starting’: The shovel is starting up, connecting and creating resources.

• ‘running’: The shovel is up and running, shovelling messages.

• {’terminated’, Reason}: Something’s gone wrong. The Reason should give a further indication of where the fault lies.

This works well very well for simple form validation but sometimes you may want to perform a more complex series of validations and reflect the failures back to the user in a similar manner to the field level validation. For example, a particular combination of fields may result in breaking a business rule for that particular user and the client has asked you to highlight all of these fields with a similar message. In this situation additional validation when the form is submitted may be the simplest solution, so we need to carry out some process like this:

1. Perform validation of business object obtained from data binding.
2. Interpret validation results.
3. Flag multiple fields as being incorrect.

To perform the last step you need to know how to programmatically force binding errors. I googled and binged for ages trying to find this information so here is the simple recipe.

1. Make sure the fields that the fields that need to show binding errors have a mode of TwoWay or OneWayToSource in their binding expression.
2. Make sure the fields that need to show binding errors have the ValidatesOnDataErrors set to true in their binding expressions.
3. Force the validation errors when the form is submitted like this

ValidationError validationError = new ValidationError(emptyValidationRule,
         textBox.GetBindingExpression(TextBox.TextProperty), "Error message", null);
Validation.MarkInvalid(textBox.GetBindingExpression(TextBox.TextProperty), validationError);

I am using an emptyValidationRule just to satisfy the type signature here. It is never actually used because my business validation is carried out elsewhere. The emptyValidationRule is a simple implementation of a ValidationRule that always fails.

To remove a validation error from a field

Validation.ClearInvalid(textBox.GetBindingExpression(TextBox.TextProperty));

This approach can also help if you need to validate that a ComboBox value is actually selected when no default value has been provided. Validation rules only fire automatically if data binding actually occurs, that means the ComboBox must actually be used. You could force databinding to occur like this:

comboBox.GetBindingExpression(ComboBox.SelectedItemProperty).UpdateTarget();

when the ComboBox loses focus. However, if the user just uses the mouse to press the form submit button, the combobox will never gain or lose focus so no validation will take place. To work around this you can validate in the way I have shown above or perform UpdateTarget operations on all of the ComboBox controls in the form when submit is pressed.

At a minimum, supporting new exchange types requires only some scaffolding to plug in to (an exchange type registry) and a hook for routing messages. However, this wouldn’t support some more interesting use cases, and in particular it didn’t support our motivating use case. Exchange types that want to keep their own state need to be initialised, and be notified about other lifecycle events.

The branch bug22169 default branch of RabbitMQ supports plugin exchange types, by providing a behaviour for exchange type modules to implement, and an exchange type registry to map a module to a type (i.e., what the client supplies in the type field of exchange.declare).

The behaviour requires exported hooks for validating exchange declarations, creating exchanges, recovering durable exchanges, publishing to an exchange (this is where the routing comes in), maintaining bindings, and deleting an exchange. RabbitMQ continues to maintain the database of exchanges and bindings, and calls the hooks after it’s done its own bookkeeping.

For simplicity, the hooks are not called atomically with the bookkeeping; so, it is possible for instance to publish to a new exchange for which the hook has not completed. However, provided there are no asynchronous operations in the hook implementation, the hook will have completed by the time the OK message is sent to the client. This is no more racey than AMQP itself, with consistency at the channel level where operations follow a single thread of control.

Here’s an example that simply io:formats things as they happen:

-module(rabbit_exchange_type_debug).
-include("rabbit.hrl").

-behaviour(rabbit_exchange_type).

-export([description/0, publish/2]).
-export([validate/1, create/1, recover/2, delete/2, add_binding/2, remove_bindings/2]).
-export([register_debug_types/0]).
-include(”rabbit_exchange_type_spec.hrl”).

-rabbit_boot_step({debug_exchange_types, 
                   [{description, "debugging exchange types"},
                    {mfa, {?MODULE, register_debug_types, []}},
                    {requires, rabbit_exchange_type_registry},
                    {enables, exchange_recovery}]}).

description() ->
    [{name, <<"debug">>},
     {description, <<"Debugging exchange">>}].

backing_module(#exchange{ type = Type }) ->
    %% Presume that Type is EITHER one of the standard types –
    %% i.e,. that we have been registered this module as direct,
    %% topic, fanout or match — or, for testing purposes, it’s
    %% registered (as in the boot steps above) as debug_direct, etc.
    Type1 = case atom_to_list(Type) of
                “x-debug-” ++ T -> T; 
                              T -> T
            end,
    list_to_existing_atom(”rabbit_exchange_type_” ++ Type1).

publish(Exchange, Delivery) ->
    io:format(”Publish ~p to ~p~n”, [Delivery, Exchange]),
    Module = backing_module(Exchange),
    Module:publish(Exchange, Delivery).

validate(X) ->
    io:format(”Validate ~p~n”, [X]),
    (backing_module(X)):validate(X).

create(X) ->
    io:format(”Create ~p~n”, [X]),
    (backing_module(X)):create(X).

recover(X, Bs) ->
    io:format(”Recover ~p with bindings ~p~n”, [X, Bs]),
    (backing_module(X)):recover(X, Bs).

delete(X, Bs) ->
    io:format(”Delete ~p with bindings ~p~n”, [X, Bs]),
    (backing_module(X)):delete(X, Bs).

add_binding(X, B) ->
    io:format(”Add binding ~p to ~p~n”, [B, X]),
    (backing_module(X)):add_binding(X, B).

remove_bindings(X, Bs) ->
    io:format(”Delete bindings ~p from ~p~n”, [Bs, X]),
    (backing_module(X)):remove_bindings(X, B).

register_debug_types() ->
    lists:foreach(
      fun (T) ->
              rabbit_exchange_type_registry:register(T, ?MODULE)
      end,
      [<<"x-debug-direct">>,
       <<"x-debug-topic">>,
       <<"x-debug-fanout">>,
       <<"x-debug-headers">>]).

Bit by important bit:

-behaviour(rabbit_exchange_type).
-export([description/0, publish/2]).
-export([validate/1, create/1, recover/2, delete/2, add_binding/2, remove_bindings/2]).

rabbit_exchange_type specifies these exported callbacks.

-include("rabbit_exchange_type_spec.hrl").

This include has the specs for each of the exported functions, if you’re using specs.

-rabbit_boot_step({debug_exchange_types, 
                   [{description, "debugging exchange types"},
                    {mfa, {?MODULE, register_debug_types, []}},
                    {requires, rabbit_exchange_type_registry},
                    {enables, exchange_recovery}]}).

This uses the new boot sequence mechanism to register the exchange type during boot. The “enables” and “requires” say that the function given as mfa above must be run after the exchange type registry is available, but before any exchanges are recovered.

publish(Exchange, Delivery) ->
    io:format("Publish ~p to ~p~n", [Delivery, Exchange]),
    Module = backing_module(Exchange),
    Module:publish(Exchange, Delivery).

This exchange type simply delegates to a “backing” exchange type.

register_debug_types() ->
    lists:foreach(
      fun (T) ->
              rabbit_exchange_type:register(T, ?MODULE)
      end,
      [<<"x-debug-direct">>,
       <<"x-debug-topic">>,
       <<"x-debug-fanout">>,
       <<"x-debug-headers">>]).

rabbit_exchange_type_registry maintains a registry of type to module; because of this indirection, we can register this module as many different types, then check the declared type of the exchange in our hook to see which type we’re expected to be. Note that the AMQP specification requires the “x-” prefix for non-standard exchange types.

This should reach default branch soon after RabbitMQ 1.7.1 is released. This is in RabbitMQ’s default branch, and should be in the next release (1.7.2).

You can drop modules straight into src, but they are better packaged as plugins — follow the drill at the plugin development guide (and note that your plugin may only need to be a library application).

[EDITS: Updated sample code to keep up with name changes, and correct use of list_to_atom to list_to_existing_atom] [Further EDITS: slight API change, now in default branch]

LVM is Linux’s “Logical Volume Manager”. You take a bunch of physical disks and partitions - the “physical volumes” - and bundle them together to make a “volume group”. Then you split up the “volume group” into partitions - or “logical volumes” again in whatever way you like. It works by breaking the disk up into 4MB chunks called “extents”, and maintaining a map between the logical and physical indexes of extents. The advantage is that you don’t have to care about the physical layout when you create the logical layout - you can stick together several physical disks into one big virtual disk, for example, or grow whatever logical volume you like without worrying about what might be using the disk sectors in front of it. You can also do things like move a logical volume from one disk to another while it’s still in use, which is the very exciting subject of this post.

I’ve been setting up LVM on every machine I’ve installed Linux on for a while now, on the assumption that this sort of flexibility would come in handy one day, but I’ve never actually used it until now. At install time, I set up a 100MB physical partition at the start of the disk for the “/boot” directory - I’ll talk about this later when I discuss GRUB - then dedicate the rest of the disk to LVM. You can do all this in the Ubuntu installer. When I needed to extend my disk space, here’s what I did:

• switched my machine off (not sure if I have SATA hot-swap, and thought it best not to find out the hard way)
• fitted the new disk
• brought it back up; the new disk arrived as /dev/sdb.
• Applied the same partitioning to the new disk - /dev/sdb1 is 100MB and /dev/sdb5 is the rest of the disk.
• “pvcreate /dev/sdb5″ makes (most of) the disk into a “physical partition” that LVM can use
• “vgextend lvmvolume /dev/sdb5″ adds the new physical partition to the volume group. Now my volume group is three times bigger!
• I needed to make use of some of that extra space right away to get some work done, so next came “lvextend -L +50G /dev/lvmvolume/root” to give my root partition an extra 50 gig.
• Now the partition is bigger than the filesystem, so I tell the filesystem to make use of the extra space: “resize2fs /dev/lvmvolume/root”. “resize2fs” is happy to make the filesystem bigger even while it’s mounted and in use, but you have to unmount it to make it smaller, so I tend to resize upwards in small increments as the extra space is needed.
• And that’s it, I had the space I needed. I started to use it right away.
• But, I want to take the old disk out otherwise they will start to pile up, so there’s more to do. To tell LVM to stop using the old disk, I issued “pvmove /dev/sda1″. Over the course of several hours, this copied my root filesystem - which, in case I hadn’t mentioned this enough, was mounted and in use - from the old disk to the new.
• I was disturbed to see that pvmove finshed with an error; I’m afraid I didn’t keep a copy of the error. However, issuing it a second time said there was no work to do, and “pvdisplay” showed that /dev/sda1 was no longer in use.
• “vgreduce -a” removed /dev/sda1 from the list of physical volumes in the volume group, and
• “pvremove” marked it as no longer a physical volume.

These days I call my volume group “vg” rather than “lvmvolume”. I suspect that the system that needs more than one volume group is rare, so a short name is handy.

That’s the LVM part of things moved. However, there’s still “/boot” to take care of. To explain what’s going on there, I need to talk a little about how a modern Linux system boots; I got this from this mailing list post and discussions on the #grub IRC channel.

If the BIOS decides that it’s going to boot from a hard drive, it loads a very small program from the very first sector of the disk (known as the MBR). That program then loads the next things it needs “core.img” which is in sectors 1 to 62 (which are always unused). It finds two things in there:

• GRUB modules - these tell it how to do things like read partition tables, or read Linux filesystems
• basic config information - which tells it where “/boot” may be found

In other words “core.img” contains exactly enough for GRUB to find and read the filesystem with “/boot” in it. From there, it can read any other modules it may need. It can also find the “grub.cfg” configuration file that tells it what to do next. “grub.cfg” will usually instruct it to display a menu; when you select a system to boot, “grub.cfg” will tell it where to find the kernel image it’s going to boot. It loads that image into memory and runs it, with parameters from “grub.cfg” that tell it where to look for the root filesystem it’s eventually going to use.

What I’ve described here is GRUB 2, and as it happens GRUB 2 is happy to read /boot from an LVM-based filesystem, given the right modules. However, the old GRUB couldn’t do that, and since Ubuntu still prefers to use the old GRUB at install time and upgrade later, I prefer to put it on a separate filesystem. Possibly I could have changed that while moving to the new disk, but I didn’t want to change too much at once, and it only takes up 0.1% of the disk.

So, after this transition I’ve got most of my filesystem onto the new disk, but the boot process still relies on the old disk - only the old disk has the MBR, or core.img, or the /boot partition. Copying over /boot is easy - “mkfs.ext3 /dev/sdb1″ to create it, then mount it and copy over the files, after which I unmounted both, changed “/etc/fstab” to point at the new drive (to avoid problems when devices move about, /etc/fstab names the partition by UUID - you can get the UUID of all your partitions with “blkid”) and re-mounted it now using the new one.

After this, I went wrong. I had hoped that “grub-install –recheck /dev/sdb” would be enough to get the new drive booting OK, but it turns out that this doesn’t re-write “grub.cfg”, and that file still contained references to an old UUID. I’m not sure what the Right Way to fix this file was; I just used “grub-mkconfig > /boot/grub/grub.cfg” to rewrite it, but I suspect there’s some higher-level tool I should have called.

That’s it - with that done, I could remove the drive from the machine. However, I plugged it in one last time - in order to zero it out. Not so much for security, though this is also a consideration, since though there is some theoretical possibility of recovering data from such a drive, in practice zeroing will make recovering any data vastly more expensive. But the main reason for zeroing the drive is to unambiguously mark the drive as not in use, so that neither I nor anyone else hesitates to use the drive for another purpose. Otherwise, it’s easy to acquire stacks of “limbo” drives which have been replaced, but which you’re keeping because you can’t quite remember if there’s anything on them you might want. Zeroing the drive out eliminates that concern.

This could have been easier - the LVM stage was as straightforward as I could ask, but the GRUB side of things was harder, at least in part thanks to a lack of documentation. Still, I’m now happily using my new disk, and I was very happy not to have to leave the machine out of use while it copied half a gigabyte of data from the old drive to the new.

Download source code, building, and running

The following example is comprised of a .cm file for building the program, and the .sml file itself. The complete sources:

• test.cm
• test.sml

Running the following command compiles the project:

ml-build test.cm Testprog.main

The ml-build output is a heap file, with a file extension dependent on your architecture and operating system. For me, right now, it produces test.x86-darwin. To run the program:

sml @SMLload=test.x86-darwin

substituting the name of your ml-build-produced heap file as necessary.

On Ubuntu, you will need to have run apt-get install smlnj libcml-smlnj libcmlutil-smlnj to ensure both SML/NJ and CML are present on your system.

The build control file

The test.cm file contains

Group is
    $cml/basis.cm
    $cml/cml.cm
    $cml-lib/smlnj-lib.cm
    test.sml

which instructs the build system to use the CML variants of the basis and the standard SML/NJ library, as well as the core CML library itself and the source code of our program. For more information about the SML CM build control system, see here.

The example source code

Turning to test.sml now, we first declare the ML structure (module) we’ll be constructing. The structure name is also part of one of the command-line arguments to ml-build above, telling it which function to use as the main function for the program.

structure Testprog = struct

Next, we bring the contents of the TextIO module into scope. This is necessary in order to use the print function with CML; if we use the standard version of print, the output is unreliable. The special CML variant is needed. We also declare a local alias SU for the global SockUtil structure.

open TextIO
structure SU = SockUtil

ML programs end up being written upside down, in a sense, because function definitions need to precede their use (unless mutually-recursive definitions are used). For this reason, the next chunk is connMain, the function called in a new lightweight thread when an inbound TCP connection has been accepted. Here, it simply prints out a countdown from 10 over the course of the next five seconds or so, before closing the socket. Multiple connections end up running connMain in independent threads of control, leading automatically to the natural and obvious interleaving of outputs on concurrent connections.

fun connMain s =
    let fun count 0 = SU.sendStr (s, "Bye!\r\n")
          | count n = (SU.sendStr (s, "Hello " ^ (Int.toString n) ^ "\r\n");
                       CML.sync (CML.timeOutEvt (Time.fromReal 0.5));
                       count (n - 1))
    in
        count 10;
        print "Closing the connection.\n";
        Socket.close s
    end

The function that depends on connMain is the accept loop, which repeatedly accepts a connection and spawns a connection thread for it.

fun acceptLoop server_sock =
    let val (s, _) = Socket.accept server_sock
    in
        print "Accepted a connection.\n";
        CML.spawn (fn () => connMain(s));
        acceptLoop server_sock
    end

The next function is the primordial CML thread, responsible for creating the TCP server socket and entering the accept loop. We set SO_REUSEADDR on the socket, listen on port 8989 with a connection backlog of 5, and enter the accept loop.

fun cml_main (program_name, arglist) =
    let val s = INetSock.TCP.socket()
    in
        Socket.Ctl.setREUSEADDR (s, true);
        Socket.bind(s, INetSock.any 8989);
        Socket.listen(s, 5);
        print "Entering accept loop...\n";
        acceptLoop s
    end

Finally, the function we told ml-build to use as the main entry point of the program. The only thing we do here is disable SIGPIPE (otherwise we get rudely killed if a remote client’s socket closes!) and start CML’s scheduler running with a primordial thread function. When the scheduler decides that everything is over and the program is complete, it returns control to us. (The lone end closes the struct definition way back at the top of the file.)

fun main (program_name, arglist) =
    (UnixSignals.setHandler (UnixSignals.sigPIPE, UnixSignals.IGNORE);
     RunCML.doit (fn () => cml_main(program_name, arglist), NONE);
     OS.Process.success)

end