I’m currently documenting an application that uses RabbitMQ extensively. I want to show the routing topology within the broker, but I want the bulk generated automatically because there are a lot of entities to deal with. Given that I can export the broker definitions into JSON, it seemed like it would be fairly straightforward to generate something using Graphviz.
The internet does reliability end to end. That is, when a node A sends a message to node B, the message travels through many other nodes. These intervening nodes make a best effort at delivery, but if they don’t succeed, they just forget about it. B must confirm receipt, and A must try again if it doesn’t.
Communicating securely requires the same approach. We should not guarantee each step is secure, and hence the whole path is secure, but design a system in which only the design of A and B matters.
The more we do our computing in the cloud, the more true this becomes. Simply put, the security of our communications depends on too many people, and the chances of failure multiply.
For example, we might use a messaging service, provided in the cloud. A and B could authenticate to the message server, and ensure privacy by connecting using SSL. The server could implement access controls, to ensure other users of the service won’t receive messages they are not entitled to receive. In this arrangement, you are making some assumptions:
Alternatively, A could encrypt messages using B’s private encryption key, and sign them with its private signing key. Then we are assuming:
Unfortunately good excuses abound:
I’ve recently designed a solution which uses end to end encryption of messages, and a message server to minimise the attack surface of the solution, and ensure privacy while transferring data over the internet. The attack surface is minimised to the extent that the nodes are behind firewalls that allow no incoming connections. Because messages must have valid signatures to be accepted, an attacker cannot use specific payloads in messages to attack the system. They are limited to attacking the code which parses the envelope. This code is extremely simple.
To do this efficiently, we have designed our own cryptographic suite, based on the NSA Suite B specification. Because this specification does not specify an encoding, we have designed our own envelope for messages. The resulting library also includes a sophisticated, extensible trust model. We plan to publish the result as open source.
We see lots of sites that use RabbitMQ, and other message servers, and would benefit from the security this architecture provides. Either these sites allow clients to interact with services via a message server, or use cloud based messaging services. Using cloud based messaging services has significant cost benefits over hosting your own message server, and in this architecture, the decision to do this has no security implications.
A few years back, needing decent authentication and key management for Mercurial repositories, we wrote mercurial-server. Recently, we were in the position of wanting to use this with one of our clients, but this time it would need to run under Windows. This was a bit of a problem as it was designed for a Unix-type environment. On the other hand, it’s mostly written in Python, and we could use Cygwin, so how bad could it be?
As it turns out, not too awful. I needed to make a few small modifications, and we’ve now managed to deploy a working mercurial-server under Windows. (more…)
Thanks to the patient work of Jakub Wilk, mercurial-server 1.2-1 has hit the Debian “unstable” repository, where all being well it should make its way into testing, stable, Ubuntu and so forth. Jakub stepped in at the last minute when I discovered that the project’s previous sponsor, Steve Kemp, had resigned as a Debian developer in April.
I’d like to arrange a longer-term Debian sponsor for mercurial-server now, in advance of the next release, so that we can make sure we fix any issues with the package in advance and we’re ready when release time comes. If there are any Debian developers out there who use mercurial-server, please do get in touch – thanks!
Version 1.2 of mercurial-server is now available. This fixes a security problem, adds compatibility with Mercurial 1.9 and fixes incompatibilities with older versions of Python, adds MQ compatibility, and some other minor things.
Unfortunately it may not immediately enter Debian, because my former sponsor is no longer a Debian developer. If you’re an official Debian developer and you’d like to sponsor this package, please get in touch – thanks!
Version 1.1 of mercurial-server is now available. This permits one repository to be in a subdirectory of another, which makes subrepositories more convenient, as well as improvements to logging and documentation. I’ve also included the collection of scripts I use for testing under the “dev” subdirectory in the main repository, though I’d want to clean these up somewhat before including them in the release tarballs.
The Debian version also includes fixes for all the outstanding bugs against it in Debian; I’ve sent the files off to the sponsor, so hopefully that’ll enter Debian testing soon.
SpringSource, a division of VMware, Inc. today announced the acquisition by VMware of Rabbit Technologies, Ltd, a company set up by LShift and partners Monadic and CohesiveFT.
A worker pool is a very common pattern, and they exist in the standard libraries for many languages. The idea is simple: submit some sort of closure to a service which commits to running the closure in the future in some thread. Normally the work is shared out among many different threads and in the absence of anything fancier, one assumes a first-come-first-served queue of closures.
Erlang, with its light-weight process model is not a language which you would expect would require such an approach: processes are dirt cheap, and the scheduler maps processes onto threads when they are ready to be run — in many ways, the ErlangVM is a glorified implementation of a worker pool, only one that does pre-emption and other fancy features, in a very similar way to an OS kernel. However, we recently found in RabbitMQ a need for a worker pool. (more…)
People tend to like certain software packages to be scalable. This can have a number of different meanings but mostly it means that as you throw more work at the program, it may require some more resources, in terms of memory or CPU, but it nevertheless just keeps on working. Strangely enough, it’s fairly difficult to achieve this with finite resources. With things like memory, the classical hierarchy applies: as you use up more and more faster memory, you start to spill to slower memory — i.e. spilling to disk. The assumption tends to be that one always has enough disk space.
Other resources are even more limited, and are harder to manage. One of these is file descriptors. (more…)
I got a couple of queries recently on how to make your mercurial-server repositories publically readable over HTTP. Happily this isn’t hard to do, and doesn’t really touch on mercurial-server itself. Here’s how we do it on our Debian systems; in what follows I assume that you have installed mercurial-server on hg.example.com, and that you’re not already using that machine as a web server for anything else. First install these packages; note that they tend to have a lot of stuff you don’t need marked as recommended, so don’t install those things:
apt-get --no-install-recommends install apache2 libapache2-mod-fcgid python-flupCreate the following four files:
[collections] /var/lib/mercurial-server/repos = /var/lib/mercurial-server/repos/etc/mercurial-server/hgweb.hgrc:
[web] style = gitweb allow_archive = bz2 gz zip baseurl = http://hg.example.com/ maxchanges = 200/etc/mercurial-server/hgwebdir.fcgi:
from mercurial import demandimport; demandimport.enable()
import os os.environ["HGENCODING"] = "UTF-8" os.environ["HGRCPATH"] = "/etc/mercurial-server/hgweb.hgrc"
from mercurial.hgweb.hgwebdir_mod import hgwebdir from mercurial.hgweb.request import wsgiapplication from flup.server.fcgi import WSGIServer
def makewebapp(): return hgwebdir("/etc/mercurial-server/hgweb.config")
<VirtualHost *> ServerName hg.example.com AddHandler fcgid-script .fcgi ScriptAlias / /etc/mercurial-server/hgwebdir.fcgi/ ErrorLog /var/log/apache2/hg/error.log LogLevel warn CustomLog /var/log/apache2/hg/access.log combined </VirtualHost>Finally run these commands as root:
chmod +x /etc/mercurial-server/hgwebdir.fcgi mkdir -p /var/log/apache2/hg cd /etc/apache2/sites-enabled rm 000-default ln -s ../sites-available/hg /etc/init.d/apache2 reloadYour files should now be served at http://hg.example.com/ . Sadly because of a design flaw in hgwebdir, there’s no easy way to get Apache to handle the static files it needs, but these are pretty small so there’s no harm in letting hgwebdir handle them. The “rm 000-default” thing seems pretty undesirable, but without it I can’t seem to get this recipe to work.
I’ve chosen FastCGI as the connector. This has the advantage that
You are currently browsing the archives for the Our Software category.