Posts filed under 'Our Software'
In a previous post I explored some of the options for supporting RFC4627 (JSON) Unicode-in-strings well when mapping to Erlang terms. In the end, I settled on keeping the interface almost unchanged: the only change is that binaries returned from rfc4627:decode are to be interpreted as UTF-8 encoded text now, whereas before their interpretation was less well defined.
The new module is Erlang-RFC4627 version 1.1.0, and is available as a tarball, a debian package, or by browsing online here. You can also get the code using mercurial:
hg clone http://hg.opensource.lshift.net/erlang-rfc4627/
Here are some examples using the new module. First, let’s explore the autodetection of which encoding is being used. In the following example, we see UTF-16, both big- and little-endian, as well as ill-formed and well-formed examples of UTF-8 being passed through the autodetector. (It also supports UTF-32 big- and little-endian.)
Eshell V5.5.5 (abort with ^G)
1> rfc4627:unicode_decode([34,0,228,0,34,0]).
{'utf-16le',"\"ä\""}
2> rfc4627:unicode_decode([0,34,0,228,0,34]).
{'utf-16be',"\"ä\""}
3> rfc4627:unicode_decode([34,228,34]).
** exited: {ucs,{bad_utf8_character_code}} **
4> rfc4627:unicode_decode([34,195,164,34]).
{'utf-8',"\"ä\""}
5>
Now let’s look at decoding some UTF-8 encoded JSON text into Erlang terms, and vice versa.
5> rfc4627:decode([34,194,128,34]).
{ok,<<194,128>>,[]}
6> rfc4627:encode(<<194,128>>).
[34,194,128,34]
7> rfc4627:encode_noauto(<<194,128>>).
[34,128,34]
8> rfc4627:unicode_encode({’utf-32le’,
rfc4627:encode_noauto(<<194,128>>)}).
[34,0,0,0,128,0,0,0,34,0,0,0]
9> rfc4627:encode_noauto({obj, [{[27700], 123}]}).
[123,34,27700,34,58,49,50,51,125]
10> rfc4627:encode({obj, [{[27700], 123}]}).
“{\”æ°´\”:123}”
11>
Notice, on that final example, that Erlang is printing the final UTF-8 encoded JSON text as if it were Latin-1. This is nothing to worry about: the numbers in the returned list/string are the correct UTF-8 encoding for Unicode code point 27700.
October 3rd, 2007
tonyg
Some seven months ago, I built simple Erlang modules for generic SMTP and POP3 services. The idea is that the programmer should instantiate a service, providing callbacks for user authentication and for service-specific operations like handling deliveries, and scanning and locking mailboxes. Originally, I was planning on providing SMTP-to-AMQP and AMQP-to-POP3 gateways as part of RabbitMQ, but I haven’t had the time to seriously pursue this yet.
A snapshot of the code is available as a zip file, or you can browse the code online or retrieve it using darcs:
darcs get http://www.lshift.net/~tonyg/erlang-smtp/
The current status of the code is:
- SMTP deliveries from Thunderbird work
- POP3 retrieval from Thunderbird works, but isn’t very solid, because I haven’t implemented the stateful part of mailboxes yet.
- The SMTP implementation is somewhat loosely based on RFC 2821. It’s what you might generously call minimally conformant (improving this situation is tedious but not difficult). It doesn’t address RFC 2822 in any serious way (yet)
- The POP3 implementation is based on RFC 1939.
- SMTP AUTH is not yet implemented (but is not difficult)
- I can’t recall the details (seven months!), but I think I might have skimped on something relating to POP3 UIDL.
- Neither module has pluggable callbacks: the SMTP delivery-handler is currently
io:format, and the POP3 mailbox and user authentication database are similarly hard-coded.
Patches, bugfixes, contributions, comments and feedback are all very welcome!
Update: a new post summarises changes since this post, including pluggable callbacks etc.
September 20th, 2007
tonyg
There are a few tools for building javadoc-like documentation for .NET code available out there on the ‘net. Unfortunately, the major contenders (e.g. NDoc, Sandcastle) suffer from a few flaws: they are variously not free (gratis), not free (libre), not cross-platform, not maintained, and/or not easy-to-use. Therefore:
Presenting: NDocProc.
Download a zip snapshot from here, or use darcs to check out the repository:
darcs get http://www.lshift.net/~tonyg/ndocproc/
Update: New release available.
September 13th, 2007
tonyg
I am guest blogging here on behalf of CohesiveFT. We work with the excellent LShift team on our joint venture, RabbitMQ.
I’m here to invite you to a Birds of a Feather session this coming Thursday, August 30th, at 8pm, in central London. It is FREE and will last for 45 minutes starting at 8pm, followed by the traditional breakout discussions over a beer.
Please do take a look at RabbitMQ if you have not yet done so. It’s a commercial open source product, available under the MPL 1.1 and implementing the Advanced Message Queue Protocol. AMQP is a new way to do business messaging (ie: “what goes in, must come out“). What’s really cool is that like HTTP it is a protocol instead of a language specific API. This should make interoperability between platforms much easier and less painful (business readers: “systems integration projects take less time and success can be
predicted more accurately”). For more information, please see my list of links
here.
What is the BOF about - and why come? It’s an informal session about RabbitMQ and AMQP, and how they apply within popular environments such as Spring, Mule, Ruby,
AJAX, and other messaging protocols such as FIX.
“Informal” means we’ll be encouraging a conversation between people interested in any of these things. We want to hear from you, and from each other, rather than pushing slideware at people.
Come if you want to:
You can find out details of the BOF here. Ideally we ask you to register via the web site, but late arrivals are very welcome - if you turn up, we shall get you in. The BOF is offered as part of the popular EJUG series of tech talks and as a
tie-in with the most excellent No Fluff Just Stuff conference.
If you cannot come but want to know more about any of these things then you can email us at info@rabbitmq.com.
Thank-you very much - and we hope to see you on Thursday :-)
Posted by Chris on behalf of Alexis Richardson, CohesiveFT.
August 28th, 2007
chris
Our jukebox (mentioned previously) received an update yesterday.
To download the code,
There is a little bit of documentation available, and you can browse the code.
June 21st, 2007
tonyg
About a month ago, I wrote an implementation of RFC 4627, the JSON RFC, for Erlang. I also implemented JSON-RPC over HTTP, in the form of mod_jsonrpc, a plugin for Erlang’s built-in inets httpd. This makes accessing Erlang services from in-browser Javascript very comfortable and easy indeed.
Continue Reading February 17th, 2007
tonyg
We’re proud to announce that the project we’ve been working on for the past few months, RabbitMQ, has been released. RabbitMQ is an AMQP server written using Erlang/OTP. Check it out at http://www.rabbitmq.com/ - or you can go straight to the downloads page for sources and binaries.
February 1st, 2007
tonyg
It’s hard to believe only a week has passed since we’ve integrated the project blog into Trac - with the Timeline view now displaying both source code changes and blog entries, it now became really convenient to just keep Trac open and refresh it every time we want to get an update on the state of the project.
I still found myself switching between Trac and Bugzilla all the time, and that can be very frustrating - so frustrating it can ruin even days like this, when the weather is so fine. Something had to be done - it was time to integrate our project’s bug-list from Bugzilla into Trac.
Continue Reading November 27th, 2006
Tom Berger
Trac is the best thing that happened to humanity since the cultivation of chik peas. Finally software project management has a centralized hub almost anyone can use to make the process more effective. It is beautifully designed and implemented. Extending it is easy and there’s a strong community of developers and users. I decided to see how difficult it would be to extend Trac to grab feeds from our project blog and display them in the timeline view.
Continue Reading November 20th, 2006
Tom Berger
Sometime around the beginning of July I rewrote our internal jukebox
in Erlang. It’s taken me four months to get a round tuit, but new
stock has just arrived: here’s the code for our AJAX jukebox
web-application, as a tarball. (There’s
also a mercurial repository:
hg clone http://hg.opensource.lshift.net/erlang-jukebox/.)
Click on the image for a screenshot.
To run it, you will need Erlang,
Yaws (the Erlang webserver), a modern browser, mpg123, ogg123 (from
vorbis-tools), and some MP3 or OGG files to listen to.
I’ve made a start on a bit of documentation
and design rationale. Here are a few highlights for the curious:
You point the jukebox at one or more root URLs, which it then
spiders, collecting URLs for MP3 and OGG files, which it puts into a
simple flat-file database. Just expose, say, your iTunes folder via Apache, point the Jukebox at it,
and you’re away.
It relies on mpg123 and ogg123’s support for playing HTTP-streamed
MP3 and OGG files, respectively, rather than retrieving or playing
the media itself.
The user interface is completely written in HTML+Javascript, using
prototype for its event
binding and XMLHttpRequest
support.
The server side of the application communicates with the user
interface solely via JSON-RPC.
Erlang made a great platform for the server side of the
application. Its support for clean, simple concurrency let me design
the program in a very natural way.
As part of the development of the program, I built a few stand-alone
modules that others might be interested in reusing:
[Update: fixed an issue with json.js, tweaked the use of screen real-estate, and now seems to work with Safari, IE6, and Opera. I’ve changed the tarball link above to point to the new version.]
[Update: fixed a couple of links that had broken over time as the darcs repository evolved.]
[Update: moved from darcs to mercurial, and altered the links to reflect the change.]
November 6th, 2006
tonyg
Next Posts
Previous Posts
