technology from back to front

Simple inter-process locks

I recently faced a very common problem, how to make sure that only one instance of my program is running at a time on the host.

There are a lot of approaches that can be taken to solve this problem, but I needed a portable solution for Python.

My first idea was to use widely known IPC techniques to lock some global resource. In C I would just create a semaphore and lock it. One problem is that a semaphore is not unlocked when a process dies. Another issue is a lack of support of named semaphores for Python.

The best solution on Unix is to gain an exclusive write lock on a file using fcntl(LOCK_EX).

Of course it doesn’t work on Windows. But for this OS the solution is to take advantage of their mutex facilities using pywin32 module. I was surprised to see that this method works quite well.

It’s also possible to use the fact that only one process at a time can bind to specific tcp/ip port (unless you use SO_REUSEPORT). This is the most portable, but also the most obscure method.

Here’s the code for this inter process “locking”. It’s not really locking, because you can’t block and wait for a lock. All you can do is grab a lock or get an exception. But this is enough to make sure that there is only one process that’s using a resource. This is how you can use this module:

import interlocks, time

lock = interlocks.InterProcessLock("my resource name")
try:
    lock.lock()
except interlocks.SingleInstanceError:
    print "Other process has acquired this lock."
else:
    print "Press CTRL+C to release the lock..."
    while True: time.sleep(32767)

Test code for the interlocks module needs to open an external process that blocks the resource. The code is not perfect (race conditions), but should be enough for just a test case:

def execute(cmd):
    ''' spawn a new python process that will execute 'cmd' '''
    cmd = '''import time;''' + cmd + '''time.sleep(10);'''
    pid = os.spawnv(os.P_NOWAIT,'/usr/bin/python', ['/usr/bin/python', '-c', cmd])
    time.sleep(1) # poor man's synchronization
    return pid

lock = interlocks.InterProcessLock('test')

# lock resource from other process
pid = execute("import interlocks; a=interlocks.InterProcessLock('test');a.lock();")
try: # fail to grab a lock
    lock.lock()
except interlocks.SingleInstanceError: print "success: the lock is blocked by spawned process"
else: print "FAILURE: the lock should be blocked by spawned process (pid=%i), but isn't" % (pid,)

os.kill(pid, signal.SIGKILL)
time.sleep(1) # poor man's synchronization

Coding the tests wasn’t so painful, much more problematic was to make tests run on Windows. Obviously we need an os.kill replacement for this platform. The next problem is to make os.spawnv() work on Windows at all: which slashes to use or how to encode spaces in the path. Another issue is that the process pid returned from os.spawnv() can’t be killed. It seems that the return value is not really a proper pid. Don’t waste your time like I did, use subprocess.Popen(). Fixed test code, without os.spawnv is included in the lib.

by
marek
on
05/11/08
  1. Nice!

    You should use sys.executable to get the full path of the interpreter, and use os.name to decide to use os.kill on posix, and win32kill on win32 — rather than using hardcoded interpreter paths.

  2. Thanks tef. I just fixed it.

  3. Nice, but the simplest solution working on all known OS is trying to create a directory. Under Windows, OS/X and UNIX this is guaranteed to be atomic. It fails reliably when the directory exists. This even works in shell scripts!

  4. The problem with directories is that this “resource” is not automatically freed when the owner crashes. You need to use some kind of PID files. Then things became messy. You need to check if the process with this PID exists, if it does one should check if the name is correct and so on.

    That’s why I haven’t written about directories/pid files. All of the resources I wrote about are automatically cleared when the process crashes.

  5. JJ Johns
    on 24/05/09 at 12:55 pm

    I found this when looking for a solution to the same problem you describe.
    I had an urge to make this act more like the standard thread lock, so I refactored it a bit and added a locked() method. I also extracted a subclass and made the testing you’d implemented into doctest.
    I also added the ability to have acquire() (aka lock()) take a wait parameter, a ‘la thread locks, for all Lock types.

    http://ender.snowburst.org:4747/~jjohns/interlocks.py

    Thanks for the code.

 
 


+ 3 = four

2000-14 LShift Ltd, 1st Floor, Hoxton Point, 6 Rufus Street, London, N1 6PE, UK+44 (0)20 7729 7060   Contact us