technology from back to front

Posts Tagged ‘feedshub’

Grouping and collapsing in WireIt

I have recently been modifying the WireIt code to allow collapsing of multiple containers down into 1 composite container.
A quick summary of WireIt (from their site)

WireIt is an open-source javascript library to create web wirable interfaces for dataflow applications, visual programming languages, graphical modeling, or graph editors.

I got started on this when I was following on from Jonathan Lister’s work on using WireIt to create a non-technical user interface for Rabbit Streams. It was soon apparent that if you wanted to use this in practice you would end up with huge graphs with many containers and many wires going all over the place. Once above about 10 containers or so with a couple of wires per container it gets hard to see what is happening. To solve this problem some kind of abstraction is needed, in this case collapsing multiple containers down into 1 composite container.

After only a few containers things get hairyShrink down the 3 groups of containers for a cleaner graph

A collapsed group of containers essentially boils down to a single container representing a sub graph. So I started by looking at the JSON object produced when you save a complete WireIt graph and also how that JSON is then expanded into the complete graph on loading. Once identified it was relatively easy to implement those function for only a sub set of the whole graph.

The next challenge was maintaining the wires across a collapse/expand. The first step was to show the terminals from grouped containers on the composite container. Naming conflicts had to be handled (in the case of two duplicate internal terminals from different containers) which led to the creation of a map from external Terminals to internal Containers and their terminals. With this map I could iterate over the wires attached to terminals in the group and create copies of those wires going to the external terminals. Then on expansion an wires connected to the composite container were again mapped back to their counter parts. The same was done for fields.

In combination with the ability to manually select which fields and terminals are visible and manually set their names you can easily represent encapsulation.

I have made a fork of the WireIt GitHub repository which can be found here. I also have a small readme and example of the editor here.

James Uther

Untangling the BBC’s data feeds

Recently, Alan Ogilvie from A&Mi at the BBC announced that they were developing a “Feeds Hub”, and outlined their ambitions for it.

He also mentioned LShift, RabbitMQ and open source, and I would like to explain, from our point of view, what this project is and how we’re working with the BBC.

What is a “Feeds Hub”?

Alan describes the central problem they want to solve:

The number of new projects across the BBC starting to use feeds in creative ways is growing very quickly – just think of spaghetti… on a massive scale.

So what do we do? What are the options?

We could go down the route of gathering together a centralised ‘Feed Usage’ committee with members across the BBC, to ‘federate’ feeds so that they are all produced in the same way but, in practice, this never truly works and is likely to stifle creativity. Often it is quite difficult to convince people to work together when they have already experienced the freedom of doing what they want – often they are concerned that their projects will be delayed. Not all feeds sources that we use or want to use are under our control, things like Twitter, Flickr, blogs, etc. Federation will never solve all our problems anyway – for example, it can’t help when a source feed is turned off, it doesn’t monitor failures.

The idea is, then, is to bring the spaghetti under control; not by mandating things be done a certain way, but by overlaying a bunch of management and monitoring tools that would otherwise be ad-hoc or not exist.

We also want to enable people to discover, reuse and adapt existing feeds, rather than reinvent them. Again, not by enforcement, but by making it easier to do so than to not.

And we’re not just talking about RSS — there are (at the BBC and in general) many different protocols and formats flying about.

Technically-speaking, this adds up to a couple of pieces of kit: a platform for relaying feeds through, that supports routing, transformation and distribution by a number of different means; and, a user interface for discovering, creating, managing and monitoring these feeds.

How are LShift involved?

In short: LShift are developing the core technology, helping the BBC shepherd the various strands of the project along, and helping engage with developers to build the open source aspect of the project (about which more in a bit).

LShift are the progenitors of RabbitMQ, a message broker implementing AMQP. Over the last few years we’ve been thinking about and experimenting with different applications of messaging (and not just AMQP); for example, Rabbiter, which puts a Twitter-like spin on XMPP.

In the meantime, RabbitMQ itself has gained client libraries, gateways, adapters, and a smart, active community, to the point where it’s no longer just an AMQP message broker — it’s becoming more like a universal messaging adapter.

So we were very enthused when we heard that the BBC wanted a feeds hub, because it seemed to bring together lots of what we’d been thinking about abstractly, as well as new ideas and problems to solve, and give it all a concrete purpose.

When and how will it be open source?

We’re working on a prototype, and our plan is to make the source public as soon as it’s fit for consumption. We hope this will be in the next month.

In the meantime, I may talk about some of the core technical ideas, and our plans, here on our blog; and, of course, you can follow LShift on Twitter and the Radiolabs blog.






2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us