technology from back to front

Archive for June, 2014

Optimising compilers as adversaries

Suppose that you want to handle some secret data in C and, in the wake of some high-profile vulnerability or other, want to take precautions against your secret being leaked. Perhaps you’d write something along these lines:

#include <string .h>

typedef struct {
  char password[16];
} secret_t;

void get_secret(secret_t* secret);
void use_secret(secret_t* secret);

void wipe_secret(secret_t* secret) {
  memset(secret, 0, sizeof(secret_t));
}

int main() {
  secret_t secret;
  get_secret(&secret);
  use_secret(&secret);
  wipe_secret(&secret);
  return 0;
}

I think you could be forgiven for assuming that this does what it says. However, if you have what John Regehr calls ‘a proper sense of paranoia’, you might actually check. Here’s an excerpt of what I got when I ran clang -S -O2 -emit-llvm on this example:

define i32 @main() #0 {
  %secret = alloca %struct.secret_t, align 1
  call void @get_secret(%struct.secret_t* %secret) #4
  call void @use_secret(%struct.secret_t* %secret) #4
  ret i32 0
}

As if by magic, wipe_secret has completely disappeared.

Read more…

by
ash
on
30/06/14

Dockerising an XMPP Server

As part of an internal migration of our XMPP server, we thought this would also present a good opportunity to test drive Docker to see if it would be useful for other infrastructure projects in the future. Docker is fast becoming the industry standard for deployment on Linux platforms, and for a number of good reasons:

* Very lightweight, unlike conventional virtual machines
* Good isolation between multiple containers running on the same host machine
* Allows for multiple applications that rely on different versions of the same package to run on the same box
* Provides repeatability in deployments

For this example, we’ll be looking to Dockerise the Prosody XMPP server, with a PostgreSQL backend. If you are completely new to Docker, it would be useful to read the official documentation first to familiarise yourself with the basic concepts.

To start with, we’ll consider the PostgreSQL side, which will be split amongst two containers. One container will contain the application software (version 9.3 in this case), while the second will simply provide a container for persisting data. This means the first container can be swapped at a later time (to upgrade to a later Postgres version for example), while retaining the database data in the second container (which is quite desirable).

For the data container, the Dockerfile is specified as follows:

FROM busybox

# build data image:
#   docker build -t data_postgres .
# create data container:
#   docker run --name data_postgres data_postgres true
# data container directory listing:
#   docker run --volumes-from data_postgres busybox ls -al /data

RUN mkdir /data
ADD postgresql.conf /data/
ADD pg_hba.conf /data/

RUN adduser -u 5432 -D postgres
RUN chown -R postgres:postgres /data

VOLUME /data

This uses the very lightweight busybox base image, which provides a minimal set of userland software, and exposes a volume for writing to at /data. Two files with the Postgres configuration settings are also added to this directory, which can be picked up by the application container later, allowing the application container to be replaced without losing config information. A postgres user is also created with a specific UID of 5432 with ownership of this directory, meaning another container can create a postgres user with the same UID and have the correct read permissions on the directory.

As outlined in the comments at the top of the Dockerfile, we can build the image and create the container by running the /bin/true command, which exits quickly leaving a container behind named “data_postgres” with no running processes.

For the application container, the Dockerfile is as follows:

# run postgres:
#   docker run --volumes-from data_postgres -d --name postgres postgres93

FROM phusion/baseimage:0.9.11

# disable sshd and set up baseimage
RUN rm -rf /etc/service/sshd /etc/my_init.d/00_regen_ssh_host_keys.sh
ENV HOME /root
CMD ["/sbin/my_init"]

# install postgres 9.3
RUN useradd --uid 5432 postgres
RUN apt-get update && apt-get install -y \
    postgresql-9.3 \
    postgresql-client-9.3 \
    postgresql-contrib-9.3 \
    language-pack-en

# configure postgres
RUN locale-gen en_GB
RUN mkdir /etc/service/postgres
ADD run_postgres.sh /etc/service/postgres/run

EXPOSE 5432

# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

This uses the phusion/baseimage container, which is essentially an Ubuntu 14.04 image with some tweaks to the init process that can monitor and restart processes if they crash. One such service is added for running Postgres, which is defined in an executable bash script as follows:

#!/bin/bash

DATADIR=${DATADIR:-"/data/main"}
CONF=${CONF:-"/data/postgresql.conf"}
POSTGRES=${POSTGRES:-"/usr/lib/postgresql/9.3/bin/postgres"}
INITDB=${INITDB:-"/usr/lib/postgresql/9.3/bin/initdb"}
DB_USER=${DB_USER:-"db_user"}
DB_PASS=${DB_PASS:-"db_pass"}
DATABASE=${DATABASE:-"prosody"}

# test if DATADIR exists
if [ ! -d $DATADIR ]; then
  mkdir -p $DATADIR
fi

# test if DATADIR has content
if [ ! "$(ls -A $DATADIR)" ]; then
  chown -R postgres:postgres $DATADIR
  sudo -u postgres $INITDB -D $DATADIR
  sudo -u postgres $POSTGRES --single -D $DATADIR -c config_file=$CONF \
    <<< "CREATE USER $DB_USER WITH SUPERUSER PASSWORD '$DB_PASS';"
  sudo -u postgres $POSTGRES --single -D $DATADIR -c config_file=$CONF \
    <<< "CREATE DATABASE $DATABASE OWNER $DB_USER;"
fi

exec /sbin/setuser postgres $POSTGRES -D $DATADIR -c config_file=$CONF

After building the image and running the container (using the command outlined in the comment at the top of the Dockerfile), we’ll have a container with Postgres running, linked with the data volume created earlier for persisting database data separately, and exposing the Postgres port at 5432 for other containers to access.

The Prosody container is created with the following Dockerfile:

# run prosody:
#   docker run -t -i -d -p 5222:5222 -p 5269:5269 -p 5280:5280 -p 5347:5347 --link postgres:postgres --name prosody prosody

FROM phusion/baseimage:0.9.11

# disable sshd and set up baseimage
RUN rm -rf /etc/service/sshd /etc/my_init.d/00_regen_ssh_host_keys.sh
ENV HOME /root
CMD ["/sbin/my_init"]

# prosody installation
RUN curl https://prosody.im/files/prosody-debian-packages.key \
    | apt-key add -
RUN echo "deb http://packages.prosody.im/debian trusty main" \
    >> /etc/apt/sources.list
RUN apt-get update && apt-get install -y \
    prosody \
    lua-dbi-postgresql

# prosody config
ADD prosody.cfg.lua /etc/prosody/prosody.cfg.lua
ADD certs /etc/prosody/certs
RUN mkdir /etc/service/prosody
ADD run_prosody.sh /etc/service/prosody/run

EXPOSE 5222 5269 5280 5347

# Clean up APT when done.
RUN apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

This uses a simple bash script for running the Prosody service:

#!/bin/bash
exec /etc/init.d/prosody restart

When creating the image, SSL certificates will be picked up from the certs directory relative to the build path and embedded in the container, as well as the prosody.cfg.lua file which contains the XMPP server settings.

When running the container, a link is made between this container and the Postgres application one, which will set up an entry in this container’s /etc/hosts file that points to the correct IP for the Postgres container. For example, the Postgres settings for Prosody are set up as follows:

sql = {
  driver = "PostgreSQL",
  database = "prosody",
  username = "db_user",
  password = "db_pass",
  host = "postgres"
}

This means the XMPP server can point to a database at host “postgres”, which is the name given to the link, and the correct container IP will be used for writing to the database.

One final note would be around creating new XMPP server users with the prosodyctl command. This means running a command on the Prosody container which doesn’t have SSHD running on it, which can be achieved with nsenter. The easiest way to do this is by running the docker-enter bash script it provides that will inspect running containers by name to retrieve their process ID then enter the namespace of that container:

docker-enter prosody

This will provide a bash terminal inside the Prosody container, which allows the prosodyctl command to be run to set up new users at the XMPP server. This data will be persisted in the data volume created at the start, meaning new Prosody containers can be created at a later time without needing to repeat these steps again for the same users.

by
Shaun Taheri
on

Super-simple JavaScript inheritance

JavaScript uses prototype-based inheritance, which can prove a bit of a puzzler for those of us used to class-based object orientation. At first glance it seems like it’s basically the same and as if it can be used in very nearly the same way. If you pretend that those prototype objects are in fact classes and ignore the nuances you can get surprisingly far and code up a decent-sized heap of working code. However you will eventually be bitten and have to read up on what’s really happening at the nuts and bolts level. This is guaranteed to happen just when you’re under pressure, trying to get a critical feature working. It’s a horrible feeling, the dawning realisation that the subtle bug you can’t grok is because things don’t work the way you thought at a very basic level and that your heap of code is founded on quicksand. This happened to … a friend of mine.

This post isn’t going to attempt to explain the depths and subtleties of JavaScript’s prototype model. Plenty of others have been there before. In fact we will embrace our class-based stubbornness and attempt to get it working the way we really wanted. Plenty of others have done this too, but there are a few issues with most of their solutions:

  • They are too simplistic and don’t cover all the cases required, like having an arbitrarily deep hierarchy that can call up the constructor chain neatly
  • They are too complicated, having developed into powerful libraries with many features
  • The perennial problem: I didn’t write them, so am not in control and able to understand exactly what’s going on and adapt to exactly my needs – no more, no less.*

I present the result below, wrapped up for require.js. There is really very little code indeed – just two functions: inherit and superConstructor.

// Because class is a keyword in JS, consumers should inject this as clazz.
define(function() {

  return {
    // In the unlikely event that you need to explicitly call a superclass implementation
    // of a method, because a method with the same name exists in the current class:
    //  foo.parent.bar.call(this, x, y);
    inherit: function(child, parent) {
      child.prototype = Object.create(parent.prototype);
      child.prototype.constructor = child;
      child.prototype.parent = parent.prototype;
    },

    // The superclass constructor should generally be called from child's constructor
    // otherwise it won't run and fields defined there will be missing:
    //   superConstructor(this);
    superConstructor: function(self) {
      // The constructor that we call here may in turn wish to call superConstructor() to
      // call its own parent's constructor (but with the same 'self') so we must take
      // special measures to allow this, as self will be the same object with each recursion.
      var constructor = (self.nextParent) ? self.nextParent.constructor : self.parent.constructor;
      self.nextParent = self.parent.parent;
      constructor.call(self);
      self.nextParent = undefined;
    }
  }

});

The contents of inherit are much as you’ll find in many a blog post, though there’s a surprising amount of subtle variation out there!

More interesting is superConstructor, which employes a somewhat offensive tactic to allow calls all the way up the constructor chain. What makes this difficult is that ‘this’ must remain the actual object being constructed throughout those nested calls, so we need to manually provide the context to know what the next constructor up the chain is.

Having done this and saved the code above into clazz.js, we can write code with inheritance as follows (working example as a jsfiddle).

// A Dog can bark.
function Dog() {
    console.log('Constructed a dog');
}
Dog.prototype.bark = function() { return 'Woof' };

// A Yorkie is a Dog that barks a lot!
clazz.inherit(Yorkie, Dog);
function Yorkie() {
    var self = this;
    clazz.superConstructor(this);
}
Yorkie.prototype.bark = function() {
    var noise = this.parent.bark.call(this);
    return noise + noise + noise;
};

// Create dogs and see what noises they make.
console.log(new Dog().bark());
console.log(new Yorkie().bark());

To be fair, my super-simple inheritance library is extremely restricted in its abilities, for instance not handling constructor parameters. But that’s because I didn’t need them, and any extra features should be easy to add. Most of all it was a valuable learning experience.

* Actually I love an off-the-shelf library as much as the next chap (or chappess) – but if you don’t feel comfortable with the libraries on offer and the problem seems nicely tractable and a worthwhile learning experience then why not go for it. You can always change your mind.

by
Sam Carr
on
16/06/14

Search

Categories

You are currently browsing the LShift Ltd. blog archives for June, 2014.

Feeds

Archives

2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us