Streamlining HTTP

HTTP/1.1 is a lovely protocol. Text-based, sophisticated, flexible. It does tend toward the verbose though. What if we wanted to use HTTP's semantics in a very high-speed messaging situation? How could we mitigate the overhead of all those headers? In this post, we invent a simple alternative syntax for HTTP that drastically reduces the wasted bandwidth. For the specific example of the OPRA feed, the computed bandwidth requirement of the experimental syntax is only 11% higher than the raw data itself — nearly 300% better than ordinary HTTP.

Jeff Lindsay on Web Hooks

From Jason Salas‘s interview with Jeff Lindsay, the guy who invented the term web hooks: “For example, the Facebook Platform, although pretty complicated and full of their own technology, is still at the core based on web hooks. They call out to a user-defined external web application and integrate that with their application. That’s quite…

Come to LShift!

For some reason we are busier than ever at the moment, and we’re even more keen than usual to recruit. There are recruitment details on our website, but I wanted to put it into my own words, as something I can send to my friends when I’m telling them what we’re about and why you…

EvServer, part2: Rabbit and Comet

Few days ago I introduced EvServer. In this post I'll present a simple EvServer example. EvServer is a normal WSGI server, but with one additional feature. Instead of blocking in your WSGI application you yield a file descriptor to the server. On descriptor activity the server will continue your WSGI app till it yields again. I'll show how to wait for AMQP messages inside the WSGI application and how to push them up to the browser. If you can't wait till the end of the post, please feel free to view the online demo(outdated) of the code described below.

Erlang/OTP’s global module

Erlang/OTP’s global module helps with atomic assignment of names for processes in a distributed Erlang cluster. It makes sure that only a single process at a time holds any given name, across all connected nodes. Unlike the local name registration function, names aren’t limited to being atoms: with global, they can be any term at…

Fast ACID persistence manager for Jackrabbit

I tend to think of Oracles BDB java edition as the solution to all storage performance problems: The least performance sacrifice you can possibly make to get ACID storage in java. So when as our automated configuration import for Magnolia got slower and slower, it was naturally what I turned to. Its particularly suited in…

EvServer, Introduction: The tale of a forgotten feature

Long long time ago there was a WSGI spec. This document described a lot of interesting stuff. Between other very important paragraphs you could find a hidden gem: [...] applications will usually return an iterator (often a generator-iterator) that produces the output in a block-by-block fashion. These blocks may be broken to coincide with mulitpart boundaries (for "server push"), or just before time-consuming tasks (such as reading another block of an on-disk file). [...] It means that all WSGI conforming servers should be able to send multipart http responses. WSGI clock application theoretically could be written like that: def clock_demo(environ, start_response): start_response("200 OK", [('Content-type','text/plain')]) for i in range(100): yield "%sn" % (datetime.datetime.now(),) time.sleep(1) The problem is that way of programming just doesn't work well. It's not scalable, requires a lot of threads and can eat a lot of resources. That's why the feature has been forgotten. Until May 2008, when Christopher Stawarz reminded us this feature and proposed an enhancement to it. He suggested, that instead of blocking, like time.sleep(1), inside the code WSGI application should return a file descriptor to server. When an event happens on this descriptor, the WSGI app will be continued. Here's equivalent of the previous code, but using the extension. With appropriate server this could be scalable and work as expected: def clock_demo(environ, start_response): start_response("200 OK", [('Content-type','text/plain')]) sd = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) try: for i in range(100): yield environ['x-wsgiorg.fdevent.readable'](sd, 1.0) yield "%sn" % (datetime.datetime.now(),) except GeneratorExit: pass sd.close() So I created a server that supports it: EvServer the Asynchronous Python WSGI Server