EBOOK – REDIS IN ACTION

This book covers the use of Redis, an in-memory database/data structure server.

open all | close all

11.1.2 Creating a new status message

As we build Lua scripts to perform a set of operations, it’s good to start with a short example that isn’t terribly complicated or structure-intensive. In this case, we’ll start by writing a Lua script combined with some wrapper code to post a status message.

LUA SCRIPTS—AS ATOMIC AS SINGLE COMMANDS OR MULTI/EXECAs you already
know, individual commands in Redis are atomic in that they’re run one at a
time. With MULTI/EXEC, you can prevent other commands from running
while you’re executing multiple commands. But to Redis, EVAL and EVALSHA
are each considered to be a (very complicated) command, so they’re executed
without letting any other structure operations occur.

LUA SCRIPTS—CAN’T BE INTERRUPTED IF THEY HAVE MODIFIED STRUCTURESWhen
executing a Lua script with EVAL or EVALSHA, Redis doesn’t allow any other read/
write commands to run. This can be convenient. But because Lua is a generalpurpose
programming language, you can write scripts that never return, which
could stop other clients from executing commands. To address this, Redis offers
two options for stopping a script in Redis, depending on whether you’ve performed
a Redis call that writes.

If your script hasn’t performed any calls that write (only reads), you can execute
SCRIPT KILL if the script has been executing for longer than the configured
lua-time-limit (check your Redis configuration file).

If your script has written to Redis, then killing the script could cause Redis to
be in an inconsistent state. In that situation, the only way you can recover is to
kill Redis with the SHUTDOWN NOSAVE command, which will cause Redis to lose
any changes that have been made since the last snapshot, or since the last
group of commands was written to the AOF.

Because of these limitations, you should always test your Lua scripts before
running them in production.

As you may remember from chapter 8, listing 8.2 showed the creation of a status message.
A copy of the original code we used for posting a status message appears next.

Listing 11.2Our function from listing 8.2 that creates a status message HASH
def create_status(conn, uid, message, **data):
    pipeline = conn.pipeline(True)
    pipeline.hget('user:%s' % uid, 'login')

Get the user’s login name from their user ID.

    pipeline.incr('status:id:')

Create a new ID for the status message.

    login, id = pipeline.execute()
    if not login:
        return None

Verify that we have a proper user account before posting.

    data.update({
        'message': message,
        'posted': time.time(),
        'id': id,
        'uid': uid,
        'login': login,
    })
    pipeline.hmset('status:%s' % id, data)

Prepare and set the data for the status message.

    pipeline.hincrby('user:%s' % uid, 'posts')

Record the fact that a status message has been posted.

    pipeline.execute()
    return id

Return the ID of the newly created status message.

Generally speaking, the performance penalty for making two round trips to Redis in
order to post a status message isn’t very much—twice the latency of one round trip.
But if we can reduce these two round trips to one, we may also be able to reduce the
number of round trips for other examples where we make many round trips. Fewer
round trips means lower latency for a given group of commands. Lower latency means less waiting, fewer required web servers, and higher performance for the
entire system overall.

To review what happens when we post a new status message: we look up the user’s
name in a HASH, increment a counter (to get a new ID), add data to a Redis HASH, and
increment a counter in the user’s HASH. That doesn’t sound so bad in Lua; let’s give it
a shot in the next listing, which shows the Lua script, with a Python wrapper that
implements the same API as before.

Listing 11.3Creating a status message with Lua
def create_status(conn, uid, message, **data):

Take all of the arguments as before.

    args = [
        'message', message,
        'posted', time.time(),
        'uid', uid,
    ]
    for key, value in data.iteritems():
        args.append(key)
        args.append(value)

Prepare the arguments/attributes to be set on the status message.

    return create_status_lua(
        conn, ['user:%s' % uid, 'status:id:'], args)

Call the script.

create_status_lua = script_load('''
local login = redis.call('hget', KEYS[1], 'login')

Fetch the user’s login name from their ID; remember that tables in Lua are 1-indexed, not 0-indexed like Python and most other languages.

if not login then
    return false

If there’s no login, return that no login was found.

end
local id = redis.call('incr', KEYS[2])

Get a new ID for the status message.

local key = string.format('status:%s', id)

Prepare the destination key for the status message.

redis.call('hmset', key,
    'login', login,
    'id', id,
    unpack(ARGV))

Set the data for the status message.

redis.call('hincrby', KEYS[1], 'posts', 1)

Increment the post count of the user.

return id

Return the ID of the status message.

''')

This function performs all of the same operations that the previous all-Python version
performed, only instead of needing two round trips to Redis with every call, it should
only need one (the first call will load the script into Redis and then call the loaded
script, but subsequent calls will only require one). This isn’t a big issue for posting status
messages, but for many other problems that we’ve gone through in previous chapters,
making multiple round trips can take more time than is necessary, or lend to
WATCH/MULTI/EXEC contention.

WRITING KEYS THAT AREN’T A PART OF THE KEYS ARGUMENT TO THE SCRIPTIn the
note in section 11.1.1, I mentioned that we should pass all keys to be modified
as part of the keys argument of the script, yet here we’re writing a HASH based
on a key that wasn’t passed. Doing this makes this Lua script incompatible with the future Redis cluster. Whether this operation is still correct in a noncluster
sharded server scenario will depend on your sharding methods. I did this to
highlight that you may need to do this kind of thing, but doing so prevents you
from being able to use Redis cluster.

SCRIPT LOADERS AND HELPERSYou’ll notice in this first example that we have
two major pieces. We have a Python function that handles the translation of
the earlier API into the Lua script call, and we have the Lua script that’s
loaded from our earlier script_load() function. We’ll continue this pattern
for the remainder of the chapter, since the native API for Lua scripting (KEYS
and ARGV) can be difficult to call in multiple contexts.

Since Redis 2.6 has been completed and released, libraries that support Redis scripting
with Lua in the major languages should get better and more complete. On the
Python side of things, a script loader similar to what we’ve written is already available
in the source code repository for the redis-py project, and is currently available from
the Python Package Index. We use our script loader due to its flexibility and ease of
use when confronted with sharded network connections.

As our volume of interactions with Redis increased over time, we switched to using
locks and semaphores to help reduce contention issues surrounding WATCH/MULTI/
EXEC transactions. Let’s take a look at rewriting locks and semaphores to see if we
might be able to further improve performance.