March, 2009: LShift FPGA club

A bunch of us at LShift recently discovered a shared interest in FPGAs. These devices are reconfigurable hardware: chips that can be programmed to act like any arrangement of digital logic gates, including designs as large as general purpose processors. High-end FPGAs still cost thousands of dollars, but low-end development boards are relatively affordable, and due to the march of Moore’s law, they are now capable enough for potentially interesting applications. This means that the same kind of economic structure that allows open source software development to thrive now applies to the world of hardware design. Students, enthusiasts and professional engineers publish their projects on the Internet, and communities have formed around sites such as OpenCores.

Obstacles remain for those from a software background who wish to learn about FPGAs. Many of these are cultural: There is an overlap between the concepts used in hardware design and those of software construction, but often those superficial similarities conceal significant differences in the basic approach. And some areas that are considered basic knowledge for electronic engineers will be unfamiliar to most software developers.

But there’s nothing we like more than jumping into a previously unfamiliar area. So we have acquired an FPGA development board to play with. This board has a Xilinx FPGA, together with other bits and pieces including a 32MB DRAM chip, various Flash chips, an Ethernet port, a couple of serial ports, and a small LCD display. One of the attractions of this board is that Xilinx makes a full version of their ISE development environment free to download, including the Linux version. It’s not open source, but in the world of FPGA development environments, this is as good as it currently gets; FPGA vendors consider some details of their devices to be confidential, and so steps in the FPGA programming process necessarily involve proprietary tools.

We are still finding our feet with VHDL and Verilog, and understanding what is and isn’t feasible in the scope of an evenings-and-weekends hardware design project. The Xilinx development environment is not as polished or easy to get started with as modern software development environments. But we have already made some modest progress:

March, 2009: Yahoo doesn’t know what an email address is

Many websites refuse to accept email addresses of the form myusername+sometext@gmail.com, despite the fact that the +sometext is perfectly legitimate¹ and is an advertised feature gmail offers for creating pseudo-single-use email addresses from a base email address.

My guess is that the developers of these sites think, because they’re either lazy or incompetent, that email addresses have more restrictions than they in fact have. It’s reasonable (and fairly easy) these days to check the syntax of the DNS part of an email address, because few people use non-DNS or non-SMTP transfer methods anymore, but the mailbox part is extremely flexible and hard to check accurately. A sane thing to do is just trust the user, and send a test mail to validate the address.

I picked on Yahoo in the title of this post: Yahoo are by no means the only offender, but I just signed up for a yahoo account, so they’re for me the most recent. Their signup form also refused to provide any guidance about why they were rejecting the form submission: I had to use my previous experience of sites wrongly rejecting valid email addresses to guess what the problem might be. Fail.

Footnote 1: According to my best reading of the relevant RFCs, anyway. See the definition of dot-atom in section 3.2.4 of RFC 2822, referenced in this context by section 3.4.1.

March, 2009: OpenAMQ’s JMS client with RabbitMQ server

OpenAMQ has released their JMS client for using JMS with AMQP-supporting brokers. This afternoon I experimented with getting it running with RabbitMQ.

After a simple, small patch to the JMS client code, to make it work with the AMQP 0-8 spec that RabbitMQ implements (rather than the 0-9 spec that OpenAMQ implements), the basic examples shipped with the JMS client library seemed to work fine. The devil is no doubt in the details, but no problems leapt out at me.

To get it going, I checked it out using Git (git clone git://github.com/pieterh/openamq-jms.git). Compilation was as simple as running ant. Kudos to the OpenAMQ team for making the build process so smooth! (Not to mention writing a great piece of software :-) )

The changes to make it work with AMQP 0-8 were:

• retrieving the 0-8 specification XML

• changing the JMS client library’s build.xml file to point to the downloaded file in its generate.spec variable

• changing one line of code in src/org/openamq/client/AMQSession.java: in 0-8, the final null argument to BasicConsumeBody.createAMQFrame must be omitted

• re-running the ant build

After this, and creating a /test virtual-host using RabbitMQ’s rabbitmqctl program, the OpenAMQ JMS client examples worked fine, as far as I could tell.

rabbitmqctl add_vhost /test
rabbitmqctl set_permissions -p /test guest '.*' '.*' '.*'

You can download the patch file I applied to try it yourself. Note that you’ll need to put the correct location to your downloaded amqp0-8.xml file into build.xml.

March, 2009: LShift at QCon!

LShift will be attending QCon London!

Please come over and meet us at stand 20 during the conference, from March 11th to 13th.

I will also be presenting Etherpad clone at Skillsmatter stand (booth number 10). This will happen in break between sessions on Wednesday at 4:45 pm.

March, 2009: Reverse HTTP == Remote CGI

I’ve been working recently on Reverse HTTP, an approach to making HTTP easier to use as the distributed object system that it is. My work is similar to the work of Lentczner and Preston, but is independently invented and technically a bit different: one, I’m using plain vanilla HTTP as a transport, and two, I’m focussing a little more on the enrollment, registration, queueing and management aspects of the system. My draft spec is here (though as I’m still polishing, please excuse its roughness), and you can play with some demos or download and play with an implementation of the spec.

Comments welcome!

March, 2009: Evserver, part3: Simplified Etherpad clone

Realtime collaboration

I hate to write using markup languages.

The problem with markups is that when I see a typo in a rendered output, I have to click through the text and search for exact place with the mistake. I have the same feeling about editing Wikipedia, documentation on code.google.com, Trac, Blogger, Wordpress and so on.

But I hate writing in WYSIWYG editors even more. Almost all graphical editors generate crappy output: badly closed html tags, broken styles, stripped white space. Considering this problems I usually try to stay with markups.

Next problem is that I’m the only person that can fix mistakes in my texts. My friends tell me about typos, but I have to fix them by hand. I tried to share texts on google docs, but the collaboration doesn’t work well enough.


A few months ago I saw an online real-time editor Etherpad. That’s quite a cool toy. It solves the problem of sharing the text with my friends, but it doesn’t support any markups - it’s just a plaintext editor.

But I know how to create Comet applications easily using EvServer and Django. I realized that I could build a simplified Etherpad clone, which supports a markup language!


The Etherpad clone

I decided to spend a very limited time on this project. Actually I wanted to do everything in my spare time in a week, that is about 6 afternoons.

Features I wanted, ordered by importance:

• must support editing by many users in real time - like Etherpad
  
• must generate rendered markup with reasonable latency
• must support all major browsers (though IE and Konqueror are not a must)
• must be dead simple
  

• should be able to scale up (for reasons I’ll describe below)
• should show who created what - like Etherpad. In the end I dropped this requirement due to the limited time.
  

With such hard time constraints I was ready to make some technological decisions:

• Python on the server side
  
• EvServer as a server
• Django
  
• Haproxy as a loadbalancer
  
• use EvServer’s Comet transports
• RabbitMq as a message broker
• MemcacheDB as a database
• Memcache for temporary storage
• Support reStructuredText as a markup language

The hardest part of the project is synchronizing and merging updates from many clients in real time and fixing collisions and propagating changes to users. Fortunately Neil Fraser from Google solved this problem and published it as a very nice project diff-match-patch.

It seems that the only job I have to do is to glue this parts together.


Scalability?

A few days after Etherpad was launched I saw this dialog:

It seems that Etherpad failed to scale. I don’t expect my clone to be popular, but it would be nice to scale better, even if it’s only in theory.

So I built the application with scalability in mind. The major hub in the application is the message broker - RabbitMQ. As long as this AMQP server scales - my application will also do. In the end RabbitMQ ought to be scalable - it was build exactly for that.

To synchronize user changes between browsers with low latency EvServer application shouldn’t do any jobs requiring high CPU usage. Managing and comparing changes from browsers is quite fast. The tough part is to generate the rendered output from the markup. Rendering reStructuredText markup to html can take up to few seconds of processor time! I decided that this job should be done by external process that can afford greater latency. Except for that, the message flow is pretty standard for a real-time web application:

Burning the time

Once I decided what I wanted to build I was ready to start programming. I haven’t got a detailed plan, but I accomplished the task almost on time. This is roughly the time I used and how the effort broke down:

• 1st evening: creating initial html view
• 2nd evening: figuring out javascript hacks for the textarea
• 3rd evening: created basic message flow - the latency counter started to work
• waiting at the airport and flight: integrating the diff-merge-patch library, synchronization starts to work
• 4th evening: fixed message routing in amqp, integrated the persistent database
• 5th evening: javascript fixes for browsers
• all Sunday: making it work on IE and final fixes
  

The application


Here you can play with the final application. It’s not especially impressive, but it works. I established my goals and I proved that using proper tools it’s possible to build simplified Etherpad clone in just few days. I must admit that creating this app was a great fun.

I even start to think that the application could be useful. Maybe I should consider adding more popular markup languages like Markdown, Wiki or Trac.