Rant: “Nearby art”: using the V&A API and geolocation

A little while back, I was informed that the V&A had an API. To be honest, my first response to this was “why on earth?”. There’s been a few similar APIs coming out recently from organisations, with some sort of “build it and they’ll come” expectations i.e. expecting that all they have to do is provide the API and all us developers will automagically build them shiny apps for free. If you’re TfL, then this kinda works, but it’s not so true for a lot of places.

Having had this initial reaction, I still decided to dig through the documentation a bit, and spotted an interesting nugget - they’ll let you do geospatial searches. I’d been tinkering around with the idea of playing with this, especially for use with my shiny new Android phone, and I had an a idea for a little app to show you “nearby art” i.e. search with the V&A’s API for the nearest bit of art.

I did this mostly in Javascript, doing XMLHttpRequest’s for JSON chunks of the API. There’s also a block of Python code that needs to run on a server, but that’s entirely to get around the issues of XMLHttpRequest only allowing same-server requests. It first uses navigator.geoLocation (official spec, easier documentation) to get the user’s location, then does two V&A queries - the first to get a list of local objects, and the second to get more info on the first object.

One thing you have to be careful about is that this can break in various ways. The most obvious is a lack of navigator.geoLocation (any version of IE, and all not-latest versions of most other browsers), and another is if the user denies access to their location data. This does make navigator.geoLocation unsuitable for general use currently, but it’s a useful source of data when there is support.

The full app is over here and the source is here.

Rant: Yahoo doesn’t know what an email address is

Many websites refuse to accept email addresses of the form myusername+sometext@gmail.com, despite the fact that the +sometext is perfectly legitimate¹ and is an advertised feature gmail offers for creating pseudo-single-use email addresses from a base email address.

My guess is that the developers of these sites think, because they’re either lazy or incompetent, that email addresses have more restrictions than they in fact have. It’s reasonable (and fairly easy) these days to check the syntax of the DNS part of an email address, because few people use non-DNS or non-SMTP transfer methods anymore, but the mailbox part is extremely flexible and hard to check accurately. A sane thing to do is just trust the user, and send a test mail to validate the address.

I picked on Yahoo in the title of this post: Yahoo are by no means the only offender, but I just signed up for a yahoo account, so they’re for me the most recent. Their signup form also refused to provide any guidance about why they were rejecting the form submission: I had to use my previous experience of sites wrongly rejecting valid email addresses to guess what the problem might be. Fail.

Footnote 1: According to my best reading of the relevant RFCs, anyway. See the definition of dot-atom in section 3.2.4 of RFC 2822, referenced in this context by section 3.4.1.

Rant: Adventures with the Fisher Price My First Firewall

I’m writing this blog entry for therapeutic reasons. Everything you need to know is in the link below. Readers are invited to share the worst anti-features they have found in network devices by posting a comment.

I had a strange problem sending email from a host. I first discovered that trac couldn’t send messages via a remote smtp server. It would just hang indefinitely. So I decided it was better to set up exim on the local box, and have trac send mail using that - at least it wouldn’t hang.

Unfortunately, exim wouldn’t send messages either.

At this stage, we were using the same smtp server - exim was configured to use it as a smart host.

We discounted any firewall problems immediately, because we could establish a connection. We didn’t immediately notice that we didn’t get an initial message from the server. When we did, we assumed it was because the server wasn’t sending it for some reason, and started checking on things like DNS.

This got us nowhere.

Then I noticed that if I typed HELO into the connection I did get a response. Eventually I noticed I could type anything into the connection, and get the initial 220 back from exim.

At this point, I decided I would use tshark to check on what the smtp server was doing, and discovered that actually, it was sending the 220, and resending it a good few times too, it just never turned up at the end.

This turned my attention to the Zyxel firewall we were using.

It turns out that a ‘feature’ of the firewall designed to prevent spam prevented as receiving anything from the server on the connection until we had sent something on the connection. This feature is particularly ridiculous, since most spam mail clients don’t bother to try and synchronize with the server, so only spam would get through while legitimate clients would not.

We gather a firmware upgrade has solved this problem, but letting a firewall release into the wild without checking you could send email through it is a spectacular screw up - enough to convince me never to buy from this brand again, anyway.

Thanks Simon, for dubbing this product the ‘Fisher Price My First Firewall’.

Thanks Lucas Beeler for blogging about it here.

Thanks Zyxel for wrecking my day.

Rant: Smalltalk vs. Javascript; Diff and Diff3 for Squeak Smalltalk

Many of my recent posts here have discussed the diff and diff3 code I wrote in Javascript. A couple of weekends ago I sat down and translated the code into Squeak Smalltalk. The experience of writing the “same code” for the two different environments let me compare them fairly directly.

To sum up, Smalltalk was much more pleasant than working with Javascript, and produced higher-quality code (in my opinion) in less time. It was nice to be reminded that there are some programming languages and environments that are actually pleasant to use.

The biggest win was Smalltalk’s collection objects. Where stock Javascript limits you to the non-polymorphic

for (var index = 0; index < someArray.length; index++) {
  var item = someArray[index];
  /* do something with item, and/or index */
}

Smalltalk permits

someCollection do: [:item | "do something with item"].

or, alternatively

someCollection withIndexDo:
    [:item :index | "do something with item and index"].

Smalltalk collections are properly object-oriented, meaning that the code above is fully polymorphic. The Javascript equivalent only works with the built-in, not-even-proper-object Arrays.

Of course, I could use one of the many, many, many, many Javascript support libraries that are out there; the nice thing about Smalltalk is that I don’t have to find and configure an ill-fitting third-party bolt-on collections library, and that because the standard library is simple yet rich, I don’t have to worry about potential incompatibilities between third-party libraries, such as can occur in Javascript if you’re mixing and matching code from several sources.

Other points that occurred to me as I was working:

• Smalltalk has simple, sane syntax; Javascript… doesn’t. (The number of times I get caught out by the semantics of this alone…!)
• Smalltalk has simple, sane scoping rules; Javascript doesn’t. (O, for lexical scope!)
• Smalltalk’s uniform, integrated development tools (including automated refactorings and an excellent object explorer) helped keep the code clean and object-oriented.
• The built-in SUnit test runner let me develop unit tests alongside the code.

The end result of a couple of hours’ hacking is an implementation of Hunt-McIlroy text diff (that works over arbitrary SequenceableCollections, and has room for alternative diff implementations) and a diff3 merge engine, with a few unit tests. You can read a fileout of the code, or use Monticello to load the DiffMerge module from my public Monticello repository. [Update: Use the DiffMerge Monticello repository on SqueakSource.]

If Monticello didn’t already exist, it’d be a very straightforward matter indeed to build a DVCS for Smalltalk from here. I wonder if Spoon could use something along these lines?

It also occurred to me it’d be a great thing to use OMeta/JS to support the use of

<script type="text/smalltalk">"<![CDATA["
  (document getElementById: 'someId') innerHTML: '<p>Hello, world!</p>'
"]]>”</script>

by compiling it to Javascript at load-time (or off-line). Smalltalk would make a much better language for AJAX client-side programming.

Rant: E4X: Not as awful as I thought

Long, long ago, I complained about various warts and infelicities in E4X, the ECMAScript extensions for generating and pattern-matching XML documents. It turns out that two of my complaints were not well-founded: sequence-splicing is supported, and programmatic construction of tags is possible.

Firstly (and I’m amazed I didn’t realise this at the time, as I was using it elsewhere), it’s not a problem at all to splice in a sequence of items, in the manner of Scheme’s unquote-splicing; here’s a working solution to the problem I set myself:

function buildItems() {
  return <>
           <item>Hello</item>
           <item>World!</item>
         </>;
}
var doc = <mydocument>{buildItems()}</mydocument>;

You can even use real Arrays (which is what I tried and failed to do earlier), by guerilla-patching Array.prototype:

Array.prototype.toXMLList = function () {
    var x = <container/>;
    for (var i = 0; i < this.length; i++) {
        x.appendChild(this[i]);
    }
    return x.children();
}
function buildItems() {
    return [<item>Hello</item>,
            <item>World!</item>].toXMLList();
}
var doc = <mydocument>{buildItems()}</mydocument>;

Programmatic construction of tags is done by use of the syntax for plain old unquote, in an unusual position: inside the tag’s angle-brackets:

var tagName = "p";
var doc = <{tagName}>test</{tagName}>;

So in summary, my original expectation that E4X should turn out to be very quasiquote-like wasn’t so far off the mark. It’s enough to get the basics done (ignoring for the minute the problems with namespace prefixes), but it’s still a bit of a bolt-on afterthought; it would have been nice to see it better integrated with the rest of the language.

Rant: .NET is an endless supply of fascinating puzzles

In C, size_t is unsigned. In Java, there are no unsigned fixed-width pseudointegral types, so it can perhaps be forgiven for having an array’s length field be signed. In .NET, however, which has unsigned ints, an array’s length field is also signed. What could it possibly mean to have a length less than zero?

Rant: Closing over context still not easy in mainstream languages, Film at 11

I find it fascinating that after so many decades of support for closures, we’re still stuck in a C-style mentality of passing function-pointers that take an explicit context argument rather than a proper closure object. Witness the design of .NET’s Type.FindInterfaces method:

public virtual Type[] FindInterfaces (TypeFilter filter,
                                      Object filterCriteria);

The TypeFilter argument is a delegate. The Object argument is context that the delegate may require! This is pretty much exactly the old-school C-style way of implementing closures:

/* Yes, pretty crude translation, I know */
TypeArray find_interfaces(int (*type_filter)(Type*, void*),
                          void *argument);

Smalltalk (and Lisps) would do it in the natural way, with a block (a closure):

someType selectInterfaces: [:interface | ... ]

Lisp 1.5, complete with support for lexical closures, appeared in 1959. It’s 2007. That’s forty-eight years.

Rant: No CV?

Although we’ve been very pleased to welcome Felix and Simon in the last few months, and we’re very happy about the return of Sam Jones, we’re still on the lookout for fresh blood.

We really haven’t had much luck with recruiting strangers recently. Simon and Felix both came to us via personal introductions.

We’ve been sifting sadly though the CVs sent by recruiters. There have been a few exceptions, but these nearly always fail to raise a flutter of interest. Typically there’s no sense of the person behind the application, just a very narrow range of experience and interest. We really rely on the slow but steady trickle of CVs we get from people who actually know of LShift, and have perhaps even looked at this website and have actively chosen to apply. This kind of application has a much better chance of success here. We’ve had a few close runs in the past few months. Unfortunately, one of the interesting applicants just disappeared into the ether mid-way through the recruitment process (we’re not that bad surely?). We met someone else we’d hire tomorrow - sadly for us it turned out not to be such good timing for the applicant (yet, i hope). We’ve just heard from another interesting prospect too, but we’d still love to see more applicants like these, and a little bit more luck.

If you know open source and commercial projects, if you’re not afraid of computer science and you know how to complete a project then please cheer us up and get in touch. If you fit the bill but the job described here doesn’t appeal then we’d love to know why.

Also, we’d be very interested to know where all the CVs from women in this industry go - we haven’t seen one for ages.

Rant: Why does everything on the web require registration?

Some sites or services, quite reasonably, need to know who I am (and that I really am that person, to some acceptable level of verifiability). It’s usually because they hold data on my behalf, and neither me nor they want anyone else getting at that data.

But why does InfoQ require me to register to download a free PDF? They say, … we’re happy to offer a free version for download, to get this knowledge in as many peoples hands as possible; however, I had to complete a long form (to which I mostly invented answers), then verify my email address, then navigate back to the page. I hardly see how that is compatible with the stated aim.

After running into that, I almost despaired when I went to download the official MySQL JDBC driver. Almost: there’s actually a link below the register (or log in) options saying “No thanks, just take me to the downloads!”.

Yay for MySQL.com!

Rant: E4X and the DOM

Reading through tonyg’s recent post I came across something i haven’t yet seen in use - inline XML within Javascript code. E4X, it seems, has landed. It is now available by default in Firefox and Rhino - other implementation will surely follow.

E4X, shorthand for ECMAScript for XML is a nice language extension to Javascript adding native XML support. It adds XML types, a notation for literal XML and some basic operations. Previously, if you wanted to use XML in your Javascript code, you had two choices. Since XML has a textual representation, you could work with strings. This approach, however, is extremely error-prone, and is of limited use if you intend to do anything more sophisticted than just generating XML. The other approach is to use the XML DOM, which exposes the full power of XML using a consistent model, but is too verbose and so rather unpleasant to use.

Example: XML using strings / innerHTML

// Short, but notice how I forgot to close the paragraph
// Also, this is non-standard, and only works in HTML
myElement.innerHTML = '<p><b>Hello</b> <i>World</i>';

Example: XML using the DOM

// That must be one of the longest hello world
// examples I've ever written
var paragraph = document.createElement('p');
var bold = document.createElement('b');
var hello = document.createTextNode('Hello');
bold.appendChild(hello);
var italic = document.createElement('i');
var world = document.createTextNode('World');
italic.appendChild(world);
var space = document.createTextNode(' ');
paragraph.appendChild(bold);
paragraph.appendChild(space);
paragraph.appendChild(world);
myElement.appendhChild(paragraph);

As it happens, I am working on something that requires quite a lot of DOM manipulation within the browser, and tired of constructing XML using the DOM API I set to give the new E4X capabilities of Firefox 1.5 a try. The dissapointing reality, I soon found out, is that while E4X is very much present, it can’t be used for accessing or creating DOM elements. So if you plan on parsing some XML data, or generating XML from your program you can use E4X, but DOM manipulation, arguably the most important activity involving XML in a browser is not served by this new extension at all.

Example: How E4X could be used with the DOM

// This is structured XML, notice how there are no quotes
var p_xml =  <p><b>Hello</b> <i>World</i><p>;
// But unfortunately you can't do that
var p_element = document.createElement(p_xml);
myElement.appendChild(p_element)

Javascript is a complete, general-purpose language, but in practice, it is being used exclusively as an extension for host environments. In Firefox, for example, it is used for adding program logic to the browser’s display formats - HTML, XUL and SVG. These formats can be expressed in text, but in order to manipulate them you need to access them using the DOM. For HTML, firefox adopted the nasty innerHTML non-standard extension, which allows the user to access the contents of a node as text. Fortunately, this extension doesn’t work with non-HTML elements. E4X could have been the perfect replacement - a compromise between using the dumb textual representation and the structured, but counter-intuitive DOM.

Why doesn’t Firefox provide a way to construct and manipulate DOM elements using E4X? It’s hard to blame the mozilla developers, given that the ECMA standard does not include any mention of the DOM or how to interact with it. Any extension they would have come up with would end being the next generation innerHTML non-standard.

This failure of the E4X standard, together with tonyg’s previous critique of E4X, as well as other rumours from the Javascript development arena have me wondering whether the standartisation efforts by ECMA have greatly benefited the language and its active community.