technology from back to front

Archive for October, 2009

Python quirks

I’ve been using Python for a while. Recently I have noted some nuances, wonders and counter-intuitive things I ran into. The list grew surprisingly fast.

Read more…

by
marek
on
29/10/09

Scripting SQL Server Databases

I recently had a need to move a SQL Server database around several development environments. I would normally use SQL Server Management Studio to generate creation and drop scripts, but as the database was changing frequently, I wanted a way of automating this process. Being able to script the structure and data separately was also useful, as the latter changed more frequently and independently than the former.

We use PowerShell for the bulk of the admin tasks around the project, so this, in conjunction with the SMO (SQL Server Management Objects) API was a natural starting point. The basic script was easy, but the problem came in working out dependencies so that the generated scripts could be executed without barfing on constraint problems. For our project I went with the quick and dirty solution of disabling integrity checks whilst performing the operations, i.e.:

EXEC sp_MSForEachTable 'ALTER TABLE ? NOCHECK CONSTRAINT ALL'
[SQL statements here]
EXEC sp_MSForEachTable 'ALTER TABLE ? CHECK CONSTRAINT ALL'

Whilst this achieved my aim, it was a bit of a cop-out. I later managed to get something closer to what I had originally envisaged (essentially “mysqldump” but the ability to script only the data if requested). This is how I got there..

Read more…

by
Lee Coomber
on
28/10/09

HTTP Routing with RabbitMQ and Trapeze

After building Hookout a little while back, I’ve been considering other things you could do to funnel clients through a server without them necessarily being reachable, or having an entire address space of their own. Hookout was working within the constraints of the reverse http protocol, where clients could speak only http. I wanted to explore a space where this wasn’t necessarily a requirement. At the same time, Rabbit gained a plugin mechanism.

The combination of these is Trapeze. Trapeze provides HTTP request routing via AMQP. Requests are received by a RabbitMQ plugin, which then forwards these requests via AMQP to listening applications. Applications then respond via a private reply queue, and this is then forwarded back onto the web caller. Whilst this idea isn’t entirely novel (after writing Trapeze, I found http://somic.org/2008/12/18/introducing-rabbitbal/), it is an interesting area to explore, and provides a very different use case for Rabbit versus the “I want 100% guarantees of message delivery at the expense of speed” case.

Trapeze is available at http://github.com/paulj/trapeze, along with a Ruby application runner at http://github.com/paulj/trapeze-rb. Installation instructions are provided at http://github.com/paulj/trapeze/blob/master/README.rdoc.

Once installed, one can start playing with some of the interesting features of the routing algorithm. Some of the more interesting points are:

  • When applications are started, a routing key is provided that the client uses to subscribe to the “trapeze” exchange. When incoming requests arrive, a routing key is generated in the form: <verb>.<host parts>.<port>./.<path parts>. An AMQP topic exchange is used, allowing RabbitMQ to handle all of the logic involved in selecting the appropriate listener based on their declared subscription. The use of topic key syntax provides the ability for arbitrarily complex routing keys to be used, starting with simple subscriptions:

    • “*.localhost.55672./.#” to receive everything sent to http://localhost:55672
    • “*.#.example.com.*./.#” to receive everything sent to example.com and its subdomains, on any port

    Through to more complex subscriptions, such as:

    • “post.#./.listener” to receive only post requests to a /listener path, but on any server/port combination
  • Trapeze utilises AMQP Mandatory routing to ensure that if no application has registered a listener that matches a given incoming request, a basic.return will be received, and result in a 404 being returned to the client. This single flag prevents the need for Trapeze itself to have any understanding of the connected clients – it can rely entirely on the broker to inform it whether or not a connector is attached that can handle a given request.
  • To create the binding from the trapeze exchange, a named auto-delete queue is created. If multiple applications are started using the same queue name, queue load-balancing takes effect, allowing for requests to be balanced between a number of consumers. The use of QOS even opens the possibility of balancing to consumers able to process requests at different rates.

Trapeze has already spurred some interesting reflections about changes that could be made to Rabbit to do interesting things in these kinds of cases. I’m quite interested to hear if anyone else has any suggestions that could make Trapeze more useful or interesting. Also, if you’d like to write an adapter for a different language (for example, Python WSGI or even a Tomcat connector), then do get in touch. And please do check out the source on Github.

by
paulj
on
26/10/09

simple build tool

I started using Maven at a company where we had 60+ Java projects all
with their own individual Ant build file. Each build file was different and
each project was structured completely differently. Porting the most active projects to
Maven made this situation a lot saner, test code was always in the same location, the same
commands achieved the same things on different projects and we could
start to manage our library dependencies sensibly. However, some things just always seemed
crazy about Maven - having to explain that Maven really did need to ‘download the whole
internet’ just to clean compiled code out of its target directory soon gets painful!

Since I started working on some Scala projects at LShift I experienced
further annoyances with Maven so I thought I would examine an alternative to Maven written in
Scala – the simple build tool (sbt) http://code.google.com/p/simple-build-tool.

The first thing it did nicely was new project creation, this is much
simpler than Maven, no multiple command line options or selections from very long lists,
just a simple interactive script with questions that are easy to answer.

[bert] things%sbt
Project does not exist, create new project? (y/N/s) : y
Name: parsnip
Organization []: lshift
Version [1.0]:
Scala version [2.7.5]:
sbt version [0.5.5]:
:: retrieving :: sbt#boot
confs: [default]
2 artifacts copied, 0 already retrieved (9831kB/619ms) :: retrieving :: sbt#boot
confs: [default]
3 artifacts copied, 0 already retrieved (3395kB/39ms)
[success] Successfully initialized directory structure.
[info] Building project parsnip 1.0 using sbt.DefaultProject
[info]    with sbt 0.5.5 and Scala 2.7.5
[info] No actions specified, interactive session started. Execute 'help' for more information.

This creates a project structure for you that is closely modelled on the standard directory layout used by Maven. There are two additional directories lib and project. The lib
directory provides a standard place to keep any dependencies that aren’t available in a Maven repository. The project directory is where the project configuration is kept.

Secondly, there is no XML required to configure the project, the configuration is carried out by a properties
file and a Scala file. Dependencies are managed by adding the values to the Scala file that defines the project configuration; for example:

import sbt._

class ParsnipProject(info: ProjectInfo) extends DefaultProject(info) {
  val derby     = "org.apache.derby" % "derby" % "10.4.1.3"
  val scalatest = "org.scala-tools.testing" % "scalatest" % "0.9.5" % "test"
}

This uses a simple DSL to replace your Maven or Ivy configuration file.

Thirdly, testing is easy! Maven can be a bit hit or miss running the
three major Scala testing frameworks and it doesn’t report the actual
error when a test fails – you need to dig through the test output. If
a test fails with sbt the standard output you would get from the test runner is
displayed in the console. Additionally, you can make sbt monitor your project
continuously and re-test as code changes so you are informed of broken tests as soon
as they break.

Fourthly, it won’t ‘download the internet’ unless you explicitly tell
it. The command, sbt update, downloads dependencies but you can do sbt
clean without having to download the internet.

[bert] things%sbt clean
[info] Building project parsnip 1.0 using sbt.DefaultProject
[info]    with sbt 0.5.5 and Scala 2.7.5
[info]
[info] == clean ==
[info] Deleting directory /Users/timc/tmp/things/target
[info] == clean ==
[success] Successful.
[info]
[info] Total time: 0 s
[success] Build completed successfully.
[info]
[info] Total build time: 1 s

So for Scala development sbt seems to be as good if not better than Maven. Extending sbt is also
simpler, custom tasks and plugins are simple Scala functions or classes with no additional XML required.

by
tim
on
12/10/09

Search

Categories

You are currently browsing the LShift Ltd. blog archives for October, 2009.

Feeds

Archives

2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us