technology from back to front

Archive for the ‘Erlang’ Category

SpringSource / VMWare Acquire Rabbit Technologies

SpringSource, a division of VMware, Inc. today announced the acquisition by VMware of Rabbit Technologies, Ltd, a company set up by LShift and partners Monadic and CohesiveFT.

Read the full story

by
mike
on
14/04/10

On the limits of concurrency: Worker Pools in Erlang

A worker pool is a very common pattern, and they exist in the standard libraries for many languages. The idea is simple: submit some sort of closure to a service which commits to running the closure in the future in some thread. Normally the work is shared out among many different threads and in the absence of anything fancier, one assumes a first-come-first-served queue of closures.

Erlang, with its light-weight process model is not a language which you would expect would require such an approach: processes are dirt cheap, and the scheduler maps processes onto threads when they are ready to be run — in many ways, the ErlangVM is a glorified implementation of a worker pool, only one that does pre-emption and other fancy features, in a very similar way to an OS kernel. However, we recently found in RabbitMQ a need for a worker pool. Read more…

by
matthew
on
29/03/10

The fine art of holding a file descriptor

People tend to like certain software packages to be scalable. This can have a number of different meanings but mostly it means that as you throw more work at the program, it may require some more resources, in terms of memory or CPU, but it nevertheless just keeps on working. Strangely enough, it’s fairly difficult to achieve this with finite resources. With things like memory, the classical hierarchy applies: as you use up more and more faster memory, you start to spill to slower memory — i.e. spilling to disk. The assumption tends to be that one always has enough disk space.

Other resources are even more limited, and are harder to manage. One of these is file descriptors. Read more…

by
matthew
on
23/03/10

Memory matters – even in Erlang

Some time ago we got an interesting bug report for RabbitMQ. Surprisingly, unlike other complex bugs, this one is easy to describe: 

At some point basic.get suddenly starts being very slow – about 9 times slower!

Read more…

by
marek
on
28/02/10

RabbitMQ-shovel: Message Relocation Equipment

In several applications, it’s very useful to be able to take messages out of one RabbitMQ broker, and insert them into another. Many people on our mailing list have being asking for such a shovel, and we’ve recently been able to devote some time to writing one. This takes the form of a plugin for Rabbit, and whilst it hasn’t been through QA just yet, we’re announcing it so people who would like to play and even suggest further features for inclusion can do so sooner rather than later.

The shovel is written on top of the Erlang client. It supports both direct and network connections to nodes, SSL support, the ability to declare resources on nodes it connects to, basic round-robinrabbit balancing of both source and destination nodes, and allows you to configure many parameters controlling how messages are consumed from the source, and how they’re published to the destination. Multiple shovels can be specified, their statuses queried, and shovels can repeatedly reconnect to nodes in the event of failure.

The plugin is available from http://hg.rabbitmq.com/rabbitmq-shovel/, and is released under the MPL v1.1. There is a README included which contains full documentation. This is replicated below. Read more…

by
matthew
on
01/02/10

Plugin exchange types for RabbitMQ

An obvious extension point for an AMQP broker is the addition of new types of exchange. An exchange type essentially represents an algorithm for dispatching messages to queues, usually based on the message’s routing key, given how the queues are bound to the exchange — it’s a message routing algorithm.

At a minimum, supporting new exchange types requires only some scaffolding to plug in to (an exchange type registry) and a hook for routing messages. However, this wouldn’t support some more interesting use cases, and in particular it didn’t support our [motivating use case](http://oceanobservatories.org/spaces/download/attachments/19432960/ooi-amqp-api-20091228.pdf?version=1). Exchange types that want to keep their own state need to be initialised, and be notified about other lifecycle events.
Read more…

by
mikeb
on
22/01/10

Merry Christmas: Toke — Tokyo Cabinet driver for Erlang

Tokyo Cabinet is a rather excellent key-value store, with the ability to write to disk in a sane way (i.e. not just repeatedly dumping the same data over and over again), operate in bounded memory, and go really fast. I like it a lot, and there’s a likelihood that there’ll be a RabbitMQ plugin fairly soon that’ll use Tokyo Cabinet to improve the new persister yet further. Toke is an Erlang linked-in driver that allows you to use Tokyo Cabinet from Erlang. Read more…

by
matthew
on
21/12/09

RabbitMQ at the Skills Matter Functional Programming Exchange

Today I was lucky enough to give a talk at the Skills Matter Functional Programming Exchange. I talked about resource management in RabbitMQ and how we’re improving this in upcoming versions of RabbitMQ. All the sessions were videotaped and it would seem that a podcast will be going up shortly. In the mean time you can have a look at the slides if you want to.

The attendance was really good and the talks well received. There was a good range of talks, from some very practical and pragmatic such as my own, to slightly more theoretical talks. It was great to see Haskell, Erlang and F# being discussed outside of a purely academic setting and great to see so many companies and organisations getting really interested in functional programming and coming along to see how other people were making the most of it.

The Park Bench session was also good fun, with a good range of questions and experience being demonstrated by all. A good, fun atmosphere, and I’m sure all enjoyed the day.

by
matthew
on
07/12/09

Cranial Surgery: Giving Rabbit more Memory

Many users of Rabbit have been asking us about how Rabbit copes with many large messages in queues, to the extent that the total size of these messages exhausts the available physical memory (RAM). As things stand at the moment, the answer is not very well. Although we have a persistence mechanism, that is not quite an answer either because whilst it does ensure that messages are written to disk, it does not remove messages from RAM. So, we’ve been looking at writing a disk-based queue so that should RAM become tight, we can start to push messages out to disk and collect them later from there.

However, there is this thing called swap, and it seems wise to test how Rabbit copes when we just allow it to expand into swap. Read more…

by
matthew
on
02/04/09

Erlang/OTP’s global module

Erlang/OTP’s global module helps with atomic assignment of names for processes in a distributed Erlang cluster. It makes sure that only a single process at a time holds any given name, across all connected nodes. Unlike the local name registration function, names aren’t limited to being atoms: with global, they can be any term at all.

To see global‘s conflict-resolution in action, we need to register a name on two nodes not initially connected, and then make them aware of each other. The system will pick one registration to survive, and will terminate the other registration.

First, register the name “a” on each of two nodes (started with erl -sname one and erl -sname two, respectively). On node one:

Eshell V5.6.2  (abort with ^G)
(one@walk)1> global:register_name(a, self()).
yes
(one@walk)2> global:whereis_name(a).
<0.37.0>

We see that the name was registered successfully (the call to register_name returned yes), and that when looked up, a pid (the pid of the shell process) is returned, as we would expect. Now, the same on node two:

Eshell V5.6.2  (abort with ^G)
(two@walk)1> global:register_name(a, self()).
yes
(two@walk)2> global:whereis_name(a).
<0.37.0>

Again, we see it succeeding. Note that each node has successfully registered the “global” name “a”. This is because they are unaware of each other. Once they’re connected, Erlang/OTP will automatically resolve the situation. By default, it does this by terminating one of the two contending processes.

Let’s see what happens. Connect the two nodes together, by pinging one from the other — here, pinging node two from node one:

(one@walk)3> net_adm:ping(two@walk).
pong
(one@walk)4> 
=INFO REPORT==== 13-Feb-2009::03:05:22 ===
global: Name conflict terminating {a,<5744.37.0>}

(one@walk)4> global:whereis_name(a).
<0.37.0>
(one@walk)5> 

See that the termination of one of the contenders is reported with a message in the system log. It was the registration on node two that was terminated, and the registration on node one that survived. Here’s what we see on node two:

** exception error: killed
(two@walk)3> global:whereis_name(a).
<5768.37.0>
(two@walk)4> node(global:whereis_name(a)).   
one@walk

Node two’s registered process has been killed. When we then ask about the registration for the name “a”, we see a pid from node one.

Finally, we’ll try registering the name for a second time:

(two@walk)5> global:register_name(a, self()).
no
(two@walk)6> 

It answers no because there’s already a registration that it knows about in the system. The same no answer would have been returned if we’d tried the same thing on node one instead.

by
tonyg
on
13/02/09

Search

Categories

You are currently browsing the archives for the Erlang category.

Feeds

Archives

2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us