technology from back to front

Archive for the ‘RabbitMQ’ Category

Controlling fast producers in a Rabbit-as-a-Service

Offering a RabbitMQ server on the public internet poses a few challenges. Today we’ll look at the problem of controlling fast producers.

Read more…

Frank Shearar

Visualising RabbitMQ Topologies

I’m currently documenting an application that uses RabbitMQ extensively. I want to show the routing topology within the broker, but I want the bulk generated automatically because there are a lot of entities to deal with. Given that I can export the broker definitions into JSON, it seemed like it would be fairly straightforward to generate something using Graphviz.

Read more…

Lee Coomber

Secure communication in the Cloud

The internet does reliability end to end. That is, when a node A sends a message to node B, the message travels through many other nodes. These intervening nodes make a best effort at delivery, but if they don’t succeed, they just forget about it. B must confirm receipt, and A must try again if it doesn’t.

Communicating securely requires the same approach. We should not guarantee each step is secure, and hence the whole path is secure, but design a system in which only the design of A and B matters.

The more we do our computing in the cloud, the more true this becomes. Simply put, the security of our communications depends on too many people, and the chances of failure multiply.

For example, we might use a messaging service, provided in the cloud. A and B could authenticate to the message server, and ensure privacy by connecting using SSL. The server could implement access controls, to ensure other users of the service won’t receive messages they are not entitled to receive. In this arrangement, you are making some assumptions:

  • The authorisation code in the server is correct. I.e. bugs don’t cause it to deliver messages to the wrong recipients.
  • The code which configures authorisation in each of the clients is correct. I.e. bugs in the client don’t cause deliveries to the wrong recipients.
  • The server is never remotely hacked.
  • The server is physically secure.
  • The administrators of the server never looking at you messages, and never pass them on to anyone else.
  • The administrators of the server never provide access to someone unauthorised.
  • The backups of the server are securely stored, and a never used for anything other than restoring the service. They are never passed on to someone unauthorised.
  • Backup media is securely disposed of.
  • The servers disks are securely disposed of at the end of their life.
  • The SSL private keys of A, B are never transmitted to someone unauthorised

Actually, this list only cover’s privacy. If I covered authenticity, my list would be twice as long.

Alternatively, A could encrypt messages using B’s private encryption key, and sign them with its private signing key. Then we are assuming:

  • The private keys of A and B are never transmitted to someone unauthorised.

Now you can perhaps see that if you are not encrypting and signing messages before sending them via a message server, you need a good excuse.

Unfortunately good excuses abound:

  • What standard encryption scheme or software should I use?
  • What key management software should I use?

For example, GPG requires you to fork a process for each message you sign and encrypt, and when it decrypts and validates messages, it only tells you the message was from someone you trust, but not who, so it assumes you trust everyone equally.

I’ve recently designed a solution which uses end to end encryption of messages, and a message server to minimise the attack surface of the solution, and ensure privacy while transferring data over the internet. The attack surface is minimised to the extent that the nodes are behind firewalls that allow no incoming connections. Because messages must have valid signatures to be accepted, an attacker cannot use specific payloads in messages to attack the system. They are limited to attacking the code which parses the envelope. This code is extremely simple.

To do this efficiently, we have designed our own cryptographic suite, based on the NSA Suite B specification. Because this specification does not specify an encoding, we have designed our own envelope for messages. The resulting library also includes a sophisticated, extensible trust model. We plan to publish the result as open source.

We see lots of sites that use RabbitMQ, and other message servers, and would benefit from the security this architecture provides. Either these sites allow clients to interact with services via a message server, or use cloud based messaging services. Using cloud based messaging services has significant cost benefits over hosting your own message server, and in this architecture, the decision to do this has no security implications.


SpringSource / VMWare Acquire Rabbit Technologies

SpringSource, a division of VMware, Inc. today announced the acquisition by VMware of Rabbit Technologies, Ltd, a company set up by LShift and partners Monadic and CohesiveFT.

Read the full story


On the limits of concurrency: Worker Pools in Erlang

A worker pool is a very common pattern, and they exist in the standard libraries for many languages. The idea is simple: submit some sort of closure to a service which commits to running the closure in the future in some thread. Normally the work is shared out among many different threads and in the absence of anything fancier, one assumes a first-come-first-served queue of closures.

Erlang, with its light-weight process model is not a language which you would expect would require such an approach: processes are dirt cheap, and the scheduler maps processes onto threads when they are ready to be run — in many ways, the ErlangVM is a glorified implementation of a worker pool, only one that does pre-emption and other fancy features, in a very similar way to an OS kernel. However, we recently found in RabbitMQ a need for a worker pool. Read more…


The fine art of holding a file descriptor

People tend to like certain software packages to be scalable. This can have a number of different meanings but mostly it means that as you throw more work at the program, it may require some more resources, in terms of memory or CPU, but it nevertheless just keeps on working. Strangely enough, it’s fairly difficult to achieve this with finite resources. With things like memory, the classical hierarchy applies: as you use up more and more faster memory, you start to spill to slower memory — i.e. spilling to disk. The assumption tends to be that one always has enough disk space.

Other resources are even more limited, and are harder to manage. One of these is file descriptors. Read more…


Memory matters – even in Erlang

Some time ago we got an interesting bug report for RabbitMQ. Surprisingly, unlike other complex bugs, this one is easy to describe: 

At some point basic.get suddenly starts being very slow – about 9 times slower!

Read more…


RabbitMQ-shovel: Message Relocation Equipment

In several applications, it’s very useful to be able to take messages out of one RabbitMQ broker, and insert them into another. Many people on our mailing list have being asking for such a shovel, and we’ve recently been able to devote some time to writing one. This takes the form of a plugin for Rabbit, and whilst it hasn’t been through QA just yet, we’re announcing it so people who would like to play and even suggest further features for inclusion can do so sooner rather than later.

The shovel is written on top of the Erlang client. It supports both direct and network connections to nodes, SSL support, the ability to declare resources on nodes it connects to, basic round-robinrabbit balancing of both source and destination nodes, and allows you to configure many parameters controlling how messages are consumed from the source, and how they’re published to the destination. Multiple shovels can be specified, their statuses queried, and shovels can repeatedly reconnect to nodes in the event of failure.

The plugin is available from, and is released under the MPL v1.1. There is a README included which contains full documentation. This is replicated below. Read more…


Plugin exchange types for RabbitMQ

An obvious extension point for an AMQP broker is the addition of new types of exchange. An exchange type essentially represents an algorithm for dispatching messages to queues, usually based on the message’s routing key, given how the queues are bound to the exchange — it’s a message routing algorithm.

At a minimum, supporting new exchange types requires only some scaffolding to plug in to (an exchange type registry) and a hook for routing messages. However, this wouldn’t support some more interesting use cases, and in particular it didn’t support our [motivating use case]( Exchange types that want to keep their own state need to be initialised, and be notified about other lifecycle events.
Read more…


Merry Christmas: Toke — Tokyo Cabinet driver for Erlang

Tokyo Cabinet is a rather excellent key-value store, with the ability to write to disk in a sane way (i.e. not just repeatedly dumping the same data over and over again), operate in bounded memory, and go really fast. I like it a lot, and there’s a likelihood that there’ll be a RabbitMQ plugin fairly soon that’ll use Tokyo Cabinet to improve the new persister yet further. Toke is an Erlang linked-in driver that allows you to use Tokyo Cabinet from Erlang. Read more…




You are currently browsing the archives for the RabbitMQ category.



2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us