Documentation - Redise Pack

A guide to Redise Pack installation, operation and administration

open all | close all

11.2.2 Rewriting our lock

As you may remember from section 6.2, locking involved generating an ID, conditionally setting a key with SETNX, and upon success setting the expiration time of the key. Though conceptually simple, we had to deal with failures and retries, which resulted in the original code shown in the next listing.

Listing 11.4Our final acquire_lock_with_timeout() function from section 6.2.5
def acquire_lock_with_timeout(
    conn, lockname, acquire_timeout=10, lock_timeout=10):
    identifier = str(uuid.uuid4())

A 128-bit random identifier.

    lockname = 'lock:' + lockname
    lock_timeout = int(math.ceil(lock_timeout))

Only pass integers to our EXPIRE calls.

    end = time.time() + acquire_timeout
    while time.time() < end:
        if conn.setnx(lockname, identifier):
            conn.expire(lockname, lock_timeout)

Get the lock and set the expiration.

            return identifier
        elif not conn.ttl(lockname):
            conn.expire(lockname, lock_timeout)

Check and update the expiration time as necessary.

        time.sleep(.001)

    return False

There’s nothing too surprising here if you remember how we built up to this lock in section 6.2. Let’s go ahead and offer the same functionality, but move the core locking into Lua.

Listing 11.5A rewritten acquire_lock_with_timeout() that uses Lua
def acquire_lock_with_timeout(
    conn, lockname, acquire_timeout=10, lock_timeout=10):
    identifier = str(uuid.uuid4())
    lockname = 'lock:' + lockname
    lock_timeout = int(math.ceil(lock_timeout))
    acquired = False
    end = time.time() + acquire_timeout
    while time.time() < end and not acquired:
        acquired = acquire_lock_with_timeout_lua(
            conn, [lockname], [lock_timeout, identifier]) == 'OK'

Actually acquire the lock, checking to verify that the Lua call completed successfully.

        time.sleep(.001 * (not acquired))

    return acquired and identifier

acquire_lock_with_timeout_lua = script_load('''
if redis.call('exists', KEYS[1]) == 0 then

If the lock doesn’t already exist, again remembering that tables use 1-based indexing.

    return redis.call('setex', KEYS[1], unpack(ARGV))

Set the key with the provided expiration and identifier.

end
''')

There aren’t any significant changes in the code, except that we change the commands we use so that if a lock is acquired, it always has a timeout. Let’s also go ahead and rewrite the release lock code to use Lua.

Previously, we watched the lock key, and then verified that the lock still had the same value. If it had the same value, we removed the lock; otherwise we’d say that the lock was lost. Our Lua version of release_lock() is shown next.

Listing 11.6A rewritten release_lock() that uses Lua
def release_lock(conn, lockname, identifier):
    lockname = 'lock:' + lockname
    return release_lock_lua(conn, [lockname], [identifier])'

Call the Lua function that releases the lock.

release_lock_lua = script_load(''
if redis.call('get', KEYS[1]) == ARGV[1] then

Make sure that the lock matches.

    return redis.call('del', KEYS[1]) or true

Delete the lock and ensure that we return true.

end
''')

Unlike acquiring the lock, releasing the lock became shorter as we no longer needed to perform all of the typical WATCH/MULTI/EXEC steps.

Reducing the code is great, but we haven’t gotten far if we haven’t actually improved the performance of the lock itself. We’ve added some instrumentation to the locking code along with some benchmarking code that executes 1, 2, 5, and 10 parallel processes to acquire and release locks repeatedly. We count the number of attempts to acquire the lock and how many times the lock was acquired over 10 seconds, with both our original and Lua-based acquire and release lock functions. Table 11.2 shows the number of calls that were performed and succeeded.

Table 11.2 Performance of our original lock against a Lua-based lock over 10 seconds
Benchmark configuration Tries in 10 seconds Acquires in 10 seconds
Original lock, 1 client 31,359 31,359
Original lock, 2 clients 30,085 22,507
Original lock, 5 clients 47,694 19,695
Original lock, 10 clients 71,917 14,361
Lua lock, 1 client 44,494 44,494
Lua lock, 2 clients 50,404 42,199
Lua lock, 5 clients 70,807 40,826
Lua lock, 10 clients 96,871 33,990

Looking at the data from our benchmark (pay attention to the right column), one thing to note is that Lua-based locks succeed in acquiring and releasing the lock in cycles significantly more often than our previous lock—by more than 40% with a single client, 87% with 2 clients, and over 100% with 5 or 10 clients attempting to acquire and release the same locks. Comparing the middle and right columns, we can also see how much faster attempts at locking are made with Lua, primarily due to the reduced number of round trips.

But even better than performance improvements, our code to acquire and release the locks is significantly easier to understand and verify as correct.

Another example where we built a synchronization primitive is with semaphores; let’s take a look at building them next.