technology from back to front

Archive for the ‘Go’ Category

Documenting an HTTP API with Swagger

I recently tried out Swagger, for documenting an HTTP API. The big win with Swagger is that it provides a sweet HTML UI to browse your API docs and experiment with sending requests and viewing responses, which is a great experience for other developers that are trying to get to grips with your API. Try out their demo of the Swagger UI, for a simple petstore example.

Swagger petstore example - screenshot

Swagger is effectively three things ‘architecturally’:

  • A specification for the JSON files, which contain your API documentation in the abstract
  • Various “generator” tools and libraries (many third party) for producing those JSON files in the first place, either statically or dynamically
  • Swagger UI, a static HTML/JS app which consumes those JSON files and presents the nice UI.

Ideally you use a generator that extracts the API documentation for free, right from your API code. Many such generators are available for various languages and HTTP/REST frameworks, so you may have to do very little to get a free lunch here. However typically you’d expect to use extra annotations to further document the API with the useful human-facing semantic information which isn’t present in the raw code, and further annotations may be required to clarify object serialisation etc. Still, this is a pretty good facsimile of the promised land, where documentation stays in sync with the code and never goes stale!

In our specific case we were working with an app written in Go, so could potentially have used the go-restful library for our REST services, which has Swagger support built into it. However we were already committed to another library that didn’t have that support and being new to Swagger we couldn’t be sure if it was worth switching libraries or wiring up our own swagger integration. We decided to prototype a Swagger solution by hand-crafting the JSON files in the first instance, to see if we (and our users) liked the results. This showed up a particular challenge that is worth covering here.

You can’t do URL hierarchies with static file serving

A typical REST API will have URL hierarchies such as /users (that lists users) /users/fred-smith (details for a specific user) and indeed the Swagger JSON file URLs consumed by Swagger UI are assumed to be in this sort of hierarchy. Swagger UI consumes Swagger JSON files via HTTP: you give it the URL of the main JSON “resource listing” file which provides URLs for the subordinate “API declaration” files. If that resource listing is served from /main, it expects the API declarations to be at /main/user, /main/product etc. and this is hardcoded into the way it constructs URLs. Unfortunately if we want to provide these JSON files by simply serving them via Nginx, straight from disk with no smarts, we’re out of luck as your average filesystem cannot have both a file “main” and a directory “main” in the same parent directory. You just can’t do it, so you can’t serve up a hierarchy like that from static files.

Obviously you could configure your web server more intricately, mapping individual URLs to individual files to construct the hierarchy. This isn’t appealing however, especially as Swagger UI itself can be served statically (it’s just static HTML, JS, CSS etc.) and we are simply including our JSON files within its directory structure. Three simple lines of config in Nginx should be enough, to serve up swagger-ui and our included JSON files:

location /api-docs {
    alias /opt/swagger-ui;
}

The root problem here is that Swagger UI is extremely simplistic about how it interprets paths in the top-level resource listing JSON. It assumes that the paths to the individual API declaration files can simply be concatenated to the resource listing path, as if they are laid out in a pure hierarchy as sub-resources. If the resource listing is at /api-doc.json and it references a path “users.json” then Swagger UI concatenates these and looks for the API declaration at /api-doc.jsonusers.json. This looks especially bad if you have a .json extension and no leading / on the path. By fixing those two problems we get a bit closer but it’s still looking for /api-doc/users and as mentioned above, we can’t have both a file and a directory named “api-doc” in the filesystem so we are stuck. As an aside, losing the file extension is worth doing regardless, as Swagger UI uses the full name as the title for each section of the docs and you really want “users” rather than “users.json” as your heading.

The trick to win the day here is to use a path like “/../users” in the resource listing. Then the concatenated path is /api-doc/../users which is ultimately resolved to just /users. That being the case, we can put our JSON files “api-doc” and “users” in the same directory (even though Swagger likes to consider them as hierarchical) and they will link together correctly. If you do want the API declaration files to be down a level, you could use “/../apis/users” and put them in an “apis” directory one level deeper than the resource listing file. The key here is that we don’t have to have a file and directory with the same name.

by
Sam Carr
on
27/11/13

Fudging generics in Go with AST rewriting

One possible workaround for a lack of generics is code generation. Let’s look at Go’s AST manipulation to make a Maybe Int out of a Maybe a.

Read more…

by
Frank Shearar
on
30/10/13

Going m(on)ad with parser combinators

It’s about time someone started talking about Go again around here, so I picked up the old editor, and (painlessly!) installed Go. Maybe 5 minutes later I had the world’s faster compiler, a test framework, a coverage analyzer and a bunch of stuff besides available on my machine. But what to do? Hello World is so done, so I thought I’d grab my copy of Hutton & Meijer and implement a basic parser combinator.

Read more…

by
Frank Shearar
on
25/10/13

Some reflective testing with gocheck

Last time
I wrote about custom gocheck checkers and wrote a checker that checked if a
slice of int contained a specific int – it would be nice to have a generic
contains checker and using the go reflection we can write one.

Read more…

by
tim
on
30/09/11

Testing with gocheck – custom checkers

This post follows on from my previous post about gocheck. gocheck uses a checker abstraction to test arbitrary properties in tests, in the previous post for example we used the Equals and Panic checkers to test our code. It is very simple to write your own checkers and I will walk through some more complicated checker in this post.
Read more…

by
tim
on
31/08/11

Testing go programs with gocheck

The Go programming language comes with a simple built in test framework testing – this is usable and functional but is lacking in features that you might find in other languages test frameworks. A more fully featured testing framework called gocheck has been developed by Gustavo Niemeyer, this blog post walks through developing an extremely simple go program using gocheck.

The code we will try and produce is a package for editing two dimensional images such as icons. To start with lets write a minimal implementation of the library and a test. We will use a struct that holds a two dimensional array of integers and the size of the image and add a New function so that we can hide the underlying array from consumers of the library.

package image

type Image struct {
  M, N int
  content [][]int
}

func New(M, N int) *Image {
  c := make([][]int, M, M)
  for i := 0; i < M; i++ {
    c[i] = make([]int, N, N)
  }
  return &Image{N, M, c}
}

For the first test I would really like not to be able to create images with negative sizes and would like the library to panic if that happens. gocheck allows panics to be tested for so this is our first basic test.

package image

import (
  . "launchpad.net/gocheck"
  "testing"
  "os"
)

// This plumbs gocheck into testing
func Test(t *testing.T) {
  TestingT(t)
}

// This is a fixure used by a suite of tests
type S struct{}
var _ = Suite(&S{})

// This is a test
func (s *S) TestNew(c *C) {
  c.Assert(func(){New(-1, -2)}, Panics, os.NewError("Bang"))
}

Now I run the test!

make test
gotest
rm -f _test/test.a
8g  -o _gotest_.8 image.go  image_test.go
rm -f _test/test.a
gopack grc _test/test.a _gotest_.8

----------------------------------------------------------------------
PANIC: image_test.go:16: test.S.TestNew

... Panic: runtime error: makeslice: len out of range (PC=0x8056ABA)

/home/tim/Tools/go/src/pkg/runtime/proc.c:1041
  in runtime.panic
/home/tim/Tools/go/src/pkg/runtime/runtime.c:116
  in runtime.panicstring
/home/tim/Tools/go/src/pkg/runtime/slice.c:22
  in runtime.makeslice
image.go:9
  in test.New
image_test.go:17
  in test.S.TestNew
OOPS: 0 passed, 1 PANICKED
--- FAIL: image.Test (0.01 seconds)
FAIL
gotest: "./8.out" failed: exit status 1
make: *** [test] Error 2

It has failed, a panic occurred but not the panic I expected, so lets modify the New function to produce a specific panic if negative sizes are passed in.

const newError = "An image dimension cannot be negative (M = %d  N = %d)"

func New(M, N int) *Image {
  if M < 0 || N < 0 {
    panic(fmt.Errorf(newError, M, N))
  }

  c := make([][]int, M, M)
  for i := 0; i < M; i++ {
    c[i] = make([]int, N, N)
  }
  return &Image{N, M, c}
}

Running the tests again produces a pass.

make test
gotest
rm -f _test/test.a
8g  -o _gotest_.8 image.go  image_test.go
rm -f _test/test.a
gopack grc _test/test.a _gotest_.8
OK: 1 passed
PASS

Now lets provide a function to allow values to be read from the image, to add a small wrinkle let us make the origin of the image be at (1, 1) as opposed to (0, 0). Additionally in go style let us return an error with the result in a similar way to go maps. We use two auxiliary methods and a function to check that coordinates are valid.

func check(val, limit int) bool {
  return val > 0 && val <= limit
}

func (i Image) checkX(x int) bool {
  return check(x, i.M)
}

func (i Image) checkY(y int) bool {
  return check(y, i.N)
}

func (i Image) ValueAt(x, y int) (value int, ok bool)   {
  if i.checkX(x) && i.checkY(y) {
    return i.content[x - 1][y - 1], true
  }
  return 0, false
}

The accompanying test looks like this:

func (s *S) TestNewShouldPanicWithNegatives(c *C) {
  c.Assert(func() { New(-1, -2) }, Panics, fmt.Errorf(newError, -1, -2))
}

func (s *S) TestNew(c *C) {
  c.Assert(New(1, 2), NotNil)
}


func (s *S) TestValueAt(c *C) {
  i := New(1, 2)
  v, ok := i.ValueAt(-1, 2)
  c.Assert(v, Equals, 0)
  c.Assert(ok, Equals, false)

  v, ok = i.ValueAt(1, 1)
  c.Assert(v, Equals, 0)
  c.Assert(ok, Equals, true)
}

Running this gives us another pass.

make test
gotest
rm -f _test/test.a
8g  -o _gotest_.8 image.go  image_test.go
rm -f _test/test.a
gopack grc _test/test.a _gotest_.8
OK: 3 passed
PASS

Now lets add a function to put data into the image pixel, by pixel, this function also returns a value to indicate whether the pixel value was set, if you attempt to set a pixel outside of the image it will return false.

func (i* Image) SetValueAt(x, y, v int) (ok bool) {
  if i.checkX(x) && i.checkY(y) {
    i.content[x - 1][y - 1] = v
    return true
  }
  return false
}

With an accompanying test:

func (s *S) TestSetValueAt(c *C) {
  i := New(1, 2)
  c.Assert(i.SetValueAt(-1, -2, 3), Equals, false)
  c.Assert(i.SetValueAt(1, 2, 3), Equals, true)

  v, ok := i.ValueAt(1, 2)
  c.Assert(v, Equals, 3)
  c.Assert(ok, Equals, true)
}

Which will of course pass.

make test
gotest
rm -f _test/test.a
8g  -o _gotest_.8 image.go  image_test.go
rm -f _test/test.a
gopack grc _test/test.a _gotest_.8

----------------------------------------------------------------------
FAIL: image_test.go:36: test.S.TestSetValueAt

image_test.go:39:
    c.Assert(i.SetValueAt(1, 2, 3), Equals, true)
... obtained bool = false
... expected bool = true

OOPS: 3 passed, 1 FAILED
--- FAIL: image.Test (0.01 seconds)
FAIL
gotest: "./8.out" failed: exit status 1
make: *** [test] Error 2

And I have been overconfident, aarghhhhh! why doesn’t it pass? Much pondering later I realised that I was having problems with my alphabet in my New function since I swap N for M when I construct the struct. The correct New function looks like this:

func New(M, N int) *Image {
  if M < 0 || N < 0 {
    panic(fmt.Errorf(newError, M, N))
  }

  c := make([][]int, M, M)
  for i := 0; i < M; i++ {
    c[i] = make([]int, N, N)
  }
  return &Image{M, N, c}
}

Running the tests again:

make test
gotest
rm -f _test/test.a
8g  -o _gotest_.8 image.go  image_test.go
rm -f _test/test.a
gopack grc _test/test.a _gotest_.8
OK: 4 passed
PASS

Now I refactor my test to removes some of the duplication and put some common code into the fixture.

type S struct{
  i *Image
}

func (s *S) SetUpTest(c *C) {
  s.i = New(1, 2)
}

func (s *S) TearDownTest(c *C)  {
  s.i = nil
}

func (s *S) TestValueAt(c *C) {
  v, ok := s.i.ValueAt(-1, 2)
  c.Assert(v, Equals, 0)
  c.Assert(ok, Equals, false)

  v, ok = s.i.ValueAt(1, 1)
  c.Assert(v, Equals, 0)
  c.Assert(ok, Equals, true)
}

func (s *S) TestSetValueAt(c *C) {
  c.Assert(s.i.SetValueAt(-1, -2, 3), Equals, false)
  c.Assert(s.i.SetValueAt(1, 2, 3), Equals, true)

  v, ok := s.i.ValueAt(1, 2)
  c.Assert(v, Equals, 3)
  c.Assert(ok, Equals, true)
}

Now it would be handy to actually see the image so lets make it to conform to the Stringer interface required by the various Print* functions in the fmt package. First the test:

func (s *S) TestString(c *C) {
  var stringer fmt.Stringer
  c.Assert(s.i, Implements, &stringer)
}

Which fails:

make test
gotest
rm -f _test/test.a
8g  -o _gotest_.8 image.go  image_test.go
rm -f _test/test.a
gopack grc _test/test.a _gotest_.8

----------------------------------------------------------------------
FAIL: image_test.go:54: test.S.TestString

image_test.go:56:
    c.Assert(s.i, Implements, &stringer)
... obtained *image.Image = &image.Image{M:1, N:2, content:[][]int{[]int{0, 0}}}
... ifaceptr *fmt.Stringer = (*fmt.Stringer)(0x500017a8)

OOPS: 4 passed, 1 FAILED
--- FAIL: image.Test (0.01 seconds)
FAIL
gotest: "./8.out" failed: exit status 1
make: *** [test] Error 2

I add a simple String function:

func (i Image) String() string {
  return "Image"
}

And the tests pass:

make test
gotest
rm -f _test/test.a
8g  -o _gotest_.8 image.go  image_test.go
rm -f _test/test.a
gopack grc _test/test.a _gotest_.8
OK: 5 passed
PASS

So we have some basic functionality and it is all tested. Next time I will extend this example and write a custom checker for gocheck to help out.

by
tim
on
31/07/11

Using goinstall for your own local code

I am working towards finishing a small project in go and my thoughts have turned to how I will package it up and release the code. The latest release of go has made some changes to goinstall so that code installed by you as a user can be kept separate from the base installation of go.

The first step in using goinstall is to create a directory structure for your local source, packages and binaries. This can be done like this

% cd ~/work
% mkdir -p gocode/src gocode/pkg gocode/bin

You can then create a GOPATH variable:

export GOPATH=~/work/gocode

Now any go projects can be placed in the $GOPATH/src directory and installed locally, you don’t even need to write makefiles.

For example:

% cd ~/work/gocode/src
% mkdir -p experiment/plutonium

Now create a file experiment.go in the directory:

package plutonium

import (
  "fmt"
)

func ExperimentOne(s string) string {
  return fmt.Sprintf("! DANGER ! %s ! DANGER !", s)
}

Now this code can be installed locally making it available to your other go projects:

% cd ~/work/gocode/src/experiment/plutonium
% goinstall .
% ls -l ~/work/gocode/pkg/linux_386/experiment
-rw-r--r-- 1 tim tim 9004 Jun 30 13:29 plutonium.a

You can also create executable programs:

% cd ~/work/gocode/src
% mkdir -p experiment/bang
% cd experiment/bang

Create a file called main.go:

package main

import (
  "experiment/plutonium"
  "log"
)

func main() {
  log.Print("Running experiment", plutonium.ExperimentOne("Highly dangerous"))
  log.Print("Careful now")
}

Install and run the program:

% cd ~/work/gocode/src/experiment/bang
% goinstall .
% cd ~/work/gocode/bin
% ./bang
2011/06/30 13:38:45 Running experiment! DANGER ! Highly dangerous ! DANGER !
2011/06/30 13:38:45 Careful now

So no Makefiles are needed! I would still create Makefiles because you will need them if you want to use gotest to run your unit tests.

by
tim
on
30/06/11

Using the syntax tree in Go

The Go programming language provides an abstract syntax tree, parser and pretty printer as libraries written in Go. This enables you to mangle your Go code by writing code in Go – this isn’t quite lisp macros but it is a nice facility.

Here is some example code that I will transform by manipulating the AST.

package simple

import (
  "fmt"
)

func A(val int) int {
  fmt.Println("Checking value")
  if val > 0 {
    return 1
  }
  return 0
}

func b(val int) int {
  return -1
}

The Go parser and printer package are the key to performing the transformation, we can parse a Go source file and print it out again with these lines of code:

fset := token.NewFileSet()
file, err := parser.ParseFile(fset, "test_data/simple.go", nil, 0)
if err != nil {
  // Whoops!
}
printer.Fprint(os.Stdout, fset, file)

The fset variable holds tokenisation information about the file being parsed. The file variable is the root node of the AST. The final line of code prints the AST to the standard output, so in the current extract it will just echo its input.

Now the ast package provides a walk method to traverse the AST using a Visitor. We can define a simple visitor to make all functions public by uppercasing the function name.

ast.Walk(new(FuncVisitor), file)

type FuncVisitor struct {
}

func (v *FuncVisitor) Visit(node ast.Node) (w ast.Visitor)  {
  switch t := node.(type) {
  case *ast.FuncDecl:
    t.Name =  ast.NewIdent(strings.Title(t.Name.Name))
  }

  return v
}

The type defined here is just an empty struct since I don’t currently need to hold any data in my visitor. All this visitor does is match on the AST node for a function declaration and then change its name so that it is now exported by the package. The visitor returns itself and the Walk method will carry on with a depth first descent of the AST, if nil is returned the walk will stop.

Next, I’ll present a slightly more complicated case. This time I will add an import to the source file.

type ImportVisitor struct{}

func (i *ImportVisitor) Visit(node ast.Node) (w ast.Visitor) {
  switch t := node.(type) {
  case *ast.GenDecl:
    if t.Tok == token.IMPORT {
      newSpecs := make([]ast.Spec, len(t.Specs)+1)
      for i, spec := range t.Specs {
        newSpecs[i] = spec
      }
      newPackage := &ast.BasicLit{token.NoPos, token.STRING, []byte("\"fandango\"")}
      newSpecs[len(t.Specs)] = &ast.ImportSpec{nil, nil, newPackage, nil}
      t.Specs = newSpecs
    }
    return nil
  }

  return i
}

This visitor copies any existing imports to a new array and adds one additional import, the additional imports are specified as types from the ast package such as ast.ImportSpec and ast.BasicLit. Once the imports have been modified nil is returned so that the Walk is terminated.

If we run both walks and pretty print the results we get this:

package simple

import (
 "fmt"
   "fandango"
)

func A(val int) int {
    fmt.Println("Checking value")
   if val > 0 {
        return 1
    }
   return 0
}

func B(val int) int {
  return -1
}

So by combining multiple walks you can transform the source code in arbitrary ways, for example the gofix tool works in this way to refactor code as the language specification evolves. This sort of code would also be highly useful when creating all sorts of source code refactoring and analysis tools.

by
tim
on
30/04/11

Search

Categories

You are currently browsing the archives for the Go category.

Feeds

Archives

2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us