technology from back to front

Archive for the ‘Tools’ Category

Automating pre-deployment sanity checks with Grunt

Grunt is a great tool for building, running and deploying ‘Single Page Apps’. I have a single grunt command to build and deploy to S3 for production, but recently I added some extra functionality to make deployment safer and even easier:

  • Abort if you are not on master branch
  • Abort if there are any uncommitted local changes
  • Abort if not up to date with the origin repo
  • Create a file revision.txt containing the deployed git revision hash, so we can GET it from the server and be sure of which revision is live
  • Automatically create a tag with the date and time.

I found a few existing pieces to implement some of these, but not all of them, and I ended up with a set of custom Grunt tasks, which I present here in the hope that they are useful to others. They could perhaps be packaged up into a Grunt plugin.

With no further ado, here is the stripped down Gruntfile, just showing the parts relevant to this post, though the deploy-prod task definition leaves in the other task names for context in the overall flow.

module.exports = function(grunt) {

  // Load all grunt tasks matching the `grunt-*` pattern

    // Lots of other Grunty things
    // ...

    // Executing the 'gitinfo' command populates grunt.config.gitinfo with useful git information
    // (see for details) plus results of our custom git commands.
    gitinfo: {
      commands: {
        'status': ['status', '--porcelain'],
        'origin-SHA': ['rev-parse', '--verify', 'origin']

    gittag: {
      prod: {
        options: {
          tag: 'prod-<%="ddmmyy-HHMM") %>'

    shell: {
      gitfetch: {
        command: 'git fetch'
      saverevision: {
        // Save the current git revision to a file that we can GET from the server, so we can
        // be sure exactly which version is live.
        command: 'echo <%= gitinfo.local.branch.current.SHA %> > revision.txt',
        options: {
          execOptions: {
            cwd: 'dist'

  grunt.registerTask('check-branch', 'Check we are on required git branch', function(requiredBranch) {

    if (arguments.length === 0) {
      requiredBranch = 'master';

    var currentBranch = grunt.config('');

    if (currentBranch !== requiredBranch) {
      grunt.log.error('Current branch is ' + currentBranch + ' - need to be on ' + requiredBranch);
      return false;

  grunt.registerTask('check-no-local-changes', 'Check there are no uncommitted changes', function() {

    var status = grunt.config('gitinfo.status');

    if (status != '') {
      grunt.log.error('There are uncommitted local modifications.');
      return false;

  grunt.registerTask('check-up-to-date', 'Check code is up to date with remote repo', function() {

    var localSha = grunt.config('gitinfo.local.branch.current.SHA');
    var originSha = grunt.config('gitinfo.origin-SHA');

    if (localSha != originSha) {
      grunt.log.error('There are changes in the origin repo that you don\'t have.');
      return false;

  // Some of these tasks are of course ommitted above, to keep the code sample focussed.
  grunt.registerTask('deploy-prod', ['build','prod-deploy-checks','gittag:prod','aws_s3:prod']);

  grunt.registerTask('prod-deploy-checks', ['gitinfo','check-branch:master','check-no-local-changes','shell:gitfetch','check-up-to-date']);

We rely on a few node modules:

  • grunt-git which provides canned tasks for performing a few common git activities. We use it for tagging here.
  • grunt-gitinfo which sets up a config hash with handy data from git, and allows adding custom items easily. This helps us to query the current state of things.
  • grunt-gitshell which lets us run arbitrary command line tasks. We use it to git fetch (not supported by grunt-git, though we could probably have abused gitinfo to do it) and to save the current revision to file. I hope that the command I use for that is cross-platform, even to Windows, but it’s only tested on Mac so far.

Hence I ended up with the following added to package.json:

    "grunt-git": "~0.2.14",
    "grunt-gitinfo": "~0.1.6",
    "grunt-shell": "~0.7.0"
Sam Carr

Things I wish I’d known about Google Docs

I have had cause to write a lot of Google Docs recently, which leaves me furnished with a stock of interesting observations that others might find helpful. With no further ado…

It doesn’t auto-number headers

I typically want my business-like docs to have numbered headings, so an H3 might be “2.4.1. Architecture considerations”. Word can just do this automatically and keep them up to date with the changing structure of your doc. Google Docs can’t, though there is a free add-on called “Table of contents” which performs a dual duty here:

  • It shows the structure of your documents headers in a sidebar, which is incredibly handy for reviewing that structure and for navigating the doc (click to jump).
  • It can optionally renumber the headers, though it only does this when explicitly invoked via a button, which you have to remember to do after inserting new headings or restructuring. The numbering is just inserted as ordinary text in the doc as part of each header so it’s crude and non-semantic.

Rather surprisingly, the add-on can be very slow indeed to do its thing – even clicking on a link often took 3+ seconds to actually jump to the location in a 27 page doc. This is hard to fathom, but most docs are fairly short and it behaves acceptably well. Add-ons are trivially easy to install – just go to the Add-ons menu in your doc – so I would recommend everyone to dive in. Once you have used this particular add-on once, it’s two clicks to turn it on for any doc from the menu.

Printing is a lame experience

In Safari when you hit cmd-P to print, nothing happens. This leaves you a little bewildered, so you try again, and then you try invoking the menu item with the mouse rather than using the keyboard shortcut. A few seconds after the initial attempt, you might notice a little icon swoop up to the downloads button in the Safari toolbar – and when you click up there to check you’ll find each of your print attempts have caused it to download a PDF of the doc, after a multi-second wait in each case, naturally. Then you curse, open the PDF in Preview and print it from there.

I suspect it’s a lot better in Chrome, but for my money there’s no excusing such a poor experience in Safari. At the very least it should give feedback to show that it’s received your request to print and is working on it, and then make it clear what it’s actually done.

You can’t have mixed orientation pages

I wanted to include a landscape format diagram on its own page. Tough – all pages in the doc must be the same orientation.

Pasting from a Google Spreadsheet doesn’t maintain formatting

This is a trivial little thing, but annoying: if I paste a table (of estimates breakdowns, say) from a Google Spreadsheet into a Google Doc, it drops some of the text alignment formatting – so cells that were left-aligned become right-aligned.

Really it’s a shame I can’t embed a Spreadsheet directly in the doc, especially where I just want to get totals added up for me.

It doesn’t have a concept of appendices

Then again, I always found Word rather wanting in handling appendices nicely.

Drawings don’t support gradients

I was shocked and dismayed (again) to see no gradients in Google Drawings. The whole story of these apps seems to be excruciating simplicity, which is great in a way, but the reluctance to gradually increase the feature set puzzles me when they’re genuinely trying to compete with Word.

In one case I resorted to rolling my own gradients by duplicating and offsetting a shape repeatedly with very low opacity (so the opacities gradually stack up), then grouping the results. You only want to try this in extreme circumstances where it’s really important to you.

Basically, it’s pretty awesome

All of those irritations aside, it’s still my go-to tool for bashing out docs, partly because I don’t have Word and am not in a hurry to acquire it. Learn the keyboard shortcuts, use the Table of contents add-on, and you can be quite effective. I suppose the simplicity may even help to concentrate on the content and structure.

That said, an online editor that had the same cloud storage, collaboration and a much improved feature set, would be a big draw. Frankly it’s probably out there if only I look, but Google have done just enough to grab and retain the market.

Sam Carr

Getting back into front-end web development

I’ve been working on a small SPA (Single Page Application) – just HTML, CSS and JavaScript statically served and doing its thing entirely in the browser. I learned a great deal throughout the project, but here are some of the things that strike me as most valuable.

Get a good workflow going

I used Grunt to setup a nice build system that is mostly a joy during development. It took a while to evolve my Gruntfile, but now when I edit a file, the results are immediately refreshed in the browser (I don’t even have to hit cmd-R). I can deploy to S3 test, staging and live sites with a single command that takes about 3 seconds. My SASS files are compiled down to minified CSS, my JS is minified with source maps etc.

The hardest part of using Grunt is figuring out how to configure it and its many contrib plugins. I could have done with a definitive reference or perhaps I could have used Yeoman to give me an out of the box solution. However I recognised that I was always going to have to figure out the guts of Grunt so I think I really was better off bespoking it from the start. I’m glad I did as now I have a tight setup that does precisely what I want and that I understand completely.

Now it seems there is a new kid on the scene, Gulp – nicely introduced in this tutorial blog post. I will definitely be looking closely at that for my next project, with the piping approach looking like the key step beyond Grunt, along with nicer syntax. I’d also look at Browserify, for a nicer way to piece together the JS bits.

Learn JavaScript properly

To the uninitiated, JavaScript is fairly surprising in many subtle ways, and though I can grok the prototype-based inheritance fairly easily, the scoping rules caught me out repeatedly. This was especially the case as I tried to create JQuery plugins with private methods and state. Eventually a simple old article by grand-daddy of JavaScript writing Douglas Crockford gave me the vital clues I was missing.

Really I should just read his book, and I would recommend that anyone else doesn’t just attempt to learn JavaScript as they go, but takes some time to pro-actively figure out the core concepts – it will pay off in short order.

jQuery is non-negotiable

And the award for most indispensable library goes to: jQuery. Seriously, it should be baked into the browsers or the ECMAScript standard. The nicest thing about it is I can pretty much just guess at the API and be right most of the time, though reading the docs usually reveals new conveniences that go further than I even imagined.

Browser quirks can be a living nightmare

JavaScript itself is fairly reliable, especially with judicious use of mature libraries like jQuery that paper over the cross-browser DOM cracks. CSS in complicated scenarios is where it all seems to go wrong however.

It’s amazing how broken/different some browsers are. Here are just a few highlights, though every day brought tens of new oddities and associated workarounds.

  • Mobile Safari on iOS 7 reports the viewport height inconsistently (depending on how you access it) leading to bad layout and horrible JavaScript workarounds.
  • Use of -webkit-overflow-scrolling:touch causes the hardware accelerated renderer to kick in, resulting in various flickers, flashes and flitches with content not rendering.
  • IE 10 on Windows 8 shows back/forward overlays at the left/right of the screen when your mouse moves near them, obscuring links in those locations.
  • Chrome running on Retina Macs suffers from strange graphical glitches when running CSS Animations, but is fine with CSS Transitions. However other browsers/platforms really need CSS Animations to get smooth, hardware accelerated movement. In my case it was necessary to implement both approaches and select using browser detection.
Sam Carr

Grunt uglify file specs

I struggled a bit finding relevant examples of Gruntfile configuration for Uglify, so having solved a few specific problems myself, here’s what I came up with.

This is just a snippet from the whole Gruntfile of course, and contains half-decent comments already, though I’ll provide some extra explanations below to point out the most interesting bits.

// Variables used internally within this config.
conf: {
  app: 'app',
  dist: 'dist',
  // Just our own custom scripts.
  script_files: ['scripts/*.js'],
  // All scripts that should be minified into final result.
  // Ordering is important as it determines order in the minified output and hence load order at runtime.
  // We don't include jquery (though we could) as it's better to get it from Google where possible.
  minify_js_files: [
      '<%= conf.script_files %>',

uglify: {
  options: {
    banner: '/*! <%= %> <%="yyyy-mm-dd") %> */\n',
    sourceMap: '<%= conf.dist %>/scripts/',
    sourceMapRoot: '/scripts',
    sourceMappingURL: '/scripts/',
    sourceMapPrefix: 2
  // For dev, effectively just concatenate all the JS into one file but perform no real minification.
  // This means that the HTML is the same for dev and prod (it just loads the single .js file) but
  // debugging in the browser works properly in the dev environment. It should work even when fully
  // minified, given the source maps, but practice shows that it doesn't.
  dev: {
    options: {
      report: false,
      mangle: false,
      compress: false,
      beautify: true
    files: [{
      expand: true,
      cwd: '<%= %>',
      src: '<%= conf.minify_js_files %>',
      dest: '<%= conf.dist %>/scripts/main.min.js',
      // Because we want all individual sources to go into a single dest file, we need to use this
      // rename function to ensure all srcs get the same dest, otherwise each would get a separate
      // dest created by concatenting the src path onto dest path.
      rename: function(dest, src) { return dest; }
  prod: {
    options: {
      banner: '/*! <%= %> <%="yyyy-mm-dd") %> */\n',
      report: 'min',
      mangle: true,
      compress: true
    files: '<%= %>'

Use of a rename function for configuring file srcs and dests

I was really struggling to come up with src/dest configuration for Uglify that pushed all of my source files into a single minified dest file. To be fair, this is trivially easy in the common case, as you can simply use files: { ‘dest_path’: ['file_1', 'file2'] }.

However I have my list of source files in <%= conf.minify_js_files %> and the paths therein do not include the root app/ directory, because this works out for the best in various other Grunt tasks (not shown) where I use cwd in the files block to supply that root dir. Unfortunately, without my custom rename function, a separate dest is calculated for each src file, by concatenating the src path with the dest, so instead of one minified JS file we get lots of individual files sprayed over all sorts of unintended locations. The trivial rename function I’ve used overrides those calculated dest locations to our originally intended single dest. Where different srcs have the same dest, the grunt-contrib-uglify plugin has the good sense to assume you want to merge their output. And hence we get the result we want. To be clear, this is only complicated because I want to use cwd in the file config rather than using the simpler approach.

Re-using files blocks in multiple places

You can share common options amongst multiple targets by putting them at the top level of the task and overriding/extending as required in the specific targets. However you can’t do this when specifying files. In my case I want to specify the same files for both dev and prod Uglify targets, so I specify them in full for dev then use Grunt’s templating facility to refer to them from prod with files: ‘<%= %>’.

Theoretically I could have put the definition in the conf block at the top, but it’s specific to Uglify and only used there so I prefer it to be local to the Uglify task. It seems obvious now that I can refer back to it like this, but at the time I struggled to see how to achieve it. I think I had a blind spot for the generic nature of the templating mechanism, having only used it in a rigid way for config previously, and still being very new to Gruntfiles.

Uglify may break JS debugging

I found that my minified JS files could not be successfully debugged in any browsers. I could see the original un-minified code thanks to the source maps and I could set breakpoints, but they usually wouldn’t trigger, or if I did break (e.g. with JS ‘debugger’ command in my code) it was impossible to get variable values. All very frustrating.

Whilst I’m developing I use Grunt’s watch task to serve my files and to auto-process modifications on the fly, so in that mode I turn off all the actual minification features of Uglify and effectively just concatenate the files together into one. Because it’s still the same single file with the same name as in production, I can use identical static HTML to include my JS script. The source maps are still there and allow me to see the individual files in the browser debugger.

Sam Carr

Documenting an HTTP API with Swagger

I recently tried out Swagger, for documenting an HTTP API. The big win with Swagger is that it provides a sweet HTML UI to browse your API docs and experiment with sending requests and viewing responses, which is a great experience for other developers that are trying to get to grips with your API. Try out their demo of the Swagger UI, for a simple petstore example.

Swagger petstore example - screenshot

Swagger is effectively three things ‘architecturally’:

  • A specification for the JSON files, which contain your API documentation in the abstract
  • Various “generator” tools and libraries (many third party) for producing those JSON files in the first place, either statically or dynamically
  • Swagger UI, a static HTML/JS app which consumes those JSON files and presents the nice UI.

Ideally you use a generator that extracts the API documentation for free, right from your API code. Many such generators are available for various languages and HTTP/REST frameworks, so you may have to do very little to get a free lunch here. However typically you’d expect to use extra annotations to further document the API with the useful human-facing semantic information which isn’t present in the raw code, and further annotations may be required to clarify object serialisation etc. Still, this is a pretty good facsimile of the promised land, where documentation stays in sync with the code and never goes stale!

In our specific case we were working with an app written in Go, so could potentially have used the go-restful library for our REST services, which has Swagger support built into it. However we were already committed to another library that didn’t have that support and being new to Swagger we couldn’t be sure if it was worth switching libraries or wiring up our own swagger integration. We decided to prototype a Swagger solution by hand-crafting the JSON files in the first instance, to see if we (and our users) liked the results. This showed up a particular challenge that is worth covering here.

You can’t do URL hierarchies with static file serving

A typical REST API will have URL hierarchies such as /users (that lists users) /users/fred-smith (details for a specific user) and indeed the Swagger JSON file URLs consumed by Swagger UI are assumed to be in this sort of hierarchy. Swagger UI consumes Swagger JSON files via HTTP: you give it the URL of the main JSON “resource listing” file which provides URLs for the subordinate “API declaration” files. If that resource listing is served from /main, it expects the API declarations to be at /main/user, /main/product etc. and this is hardcoded into the way it constructs URLs. Unfortunately if we want to provide these JSON files by simply serving them via Nginx, straight from disk with no smarts, we’re out of luck as your average filesystem cannot have both a file “main” and a directory “main” in the same parent directory. You just can’t do it, so you can’t serve up a hierarchy like that from static files.

Obviously you could configure your web server more intricately, mapping individual URLs to individual files to construct the hierarchy. This isn’t appealing however, especially as Swagger UI itself can be served statically (it’s just static HTML, JS, CSS etc.) and we are simply including our JSON files within its directory structure. Three simple lines of config in Nginx should be enough, to serve up swagger-ui and our included JSON files:

location /api-docs {
    alias /opt/swagger-ui;

The root problem here is that Swagger UI is extremely simplistic about how it interprets paths in the top-level resource listing JSON. It assumes that the paths to the individual API declaration files can simply be concatenated to the resource listing path, as if they are laid out in a pure hierarchy as sub-resources. If the resource listing is at /api-doc.json and it references a path “users.json” then Swagger UI concatenates these and looks for the API declaration at /api-doc.jsonusers.json. This looks especially bad if you have a .json extension and no leading / on the path. By fixing those two problems we get a bit closer but it’s still looking for /api-doc/users and as mentioned above, we can’t have both a file and a directory named “api-doc” in the filesystem so we are stuck. As an aside, losing the file extension is worth doing regardless, as Swagger UI uses the full name as the title for each section of the docs and you really want “users” rather than “users.json” as your heading.

The trick to win the day here is to use a path like “/../users” in the resource listing. Then the concatenated path is /api-doc/../users which is ultimately resolved to just /users. That being the case, we can put our JSON files “api-doc” and “users” in the same directory (even though Swagger likes to consider them as hierarchical) and they will link together correctly. If you do want the API declaration files to be down a level, you could use “/../apis/users” and put them in an “apis” directory one level deeper than the resource listing file. The key here is that we don’t have to have a file and directory with the same name.

Sam Carr

Managing multiple GitHub repositories

We all know GitHub is a fantastic way to collaborate on software. It has a fairly basic issue tracker, but it serves most purposes well enough. But if your code base is spread across several repositories, it can be difficult to know how you’re doing, from a management perspective. Octoherder can help with that.

Read more…

Frank Shearar

Enhancing peer review through GitHub

You love GitHub. Of course you do. You love peer review. You especially love sending a pull request back asking for nits to be picked. So when your submitter claims to have addressed your concerns, how do you check? You could walk the commits. You could diff the entire pull request against master. If only you could diff the HEAD of the pull request against the original state of the pull request, letting you check just the new set of commits…

With github-differ you can!

Simply add this tiny extension[1] to your Chrome, and it will decorate each commit in GitHub’s Commits tab. Pick any two commits, and the extension will redirect you to a page showing the comparison of those two commits! Job done!

[1]The JavaScript involved is so small that it should be trivial to port this to FireFox’s GreaseMonkey framework.

Frank Shearar

Comments are important

Nat Pryce wrote a fun little library the other day called code-words. It rips your source into words, and turns the words into a wordcloud. In short, a visual representation of the most common words in your source, and using font size to indicate the more common terms. The aim is to give an introduction into the concepts the code speaks about. And in playing around with it I noticed something interesting.

Read more…

Frank Shearar

Three approaches to ambiguous grammars

We have many tools in our parsing toolbox. Today let’s look at how three different parsing techniques handle ambiguity caused by choice.

Read more…

Frank Shearar

Just In Time Development

Since the dark ages of yesteryear Squeak has had a very interesting button in its Debugger – “create”. Today we’re going to teach it a new trick.

Read more…

Frank Shearar



You are currently browsing the archives for the Tools category.



2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us