Documentation - Redise Pack

A guide to Redise Pack installation, operation and administration

open all | close all

11.1.2 Creating a new status message

As we build Lua scripts to perform a set of operations, it’s good to start with a short example that isn’t terribly complicated or structure-intensive. In this case, we’ll start by writing a Lua script combined with some wrapper code to post a status message.

LUA SCRIPTS—AS ATOMIC AS SINGLE COMMANDS OR MULTI/EXECAs you already know, individual commands in Redis are atomic in that they’re run one at a time. With MULTI/EXEC, you can prevent other commands from running while you’re executing multiple commands. But to Redis, EVAL and EVALSHA are each considered to be a (very complicated) command, so they’re executed without letting any other structure operations occur.

LUA SCRIPTS—CAN’T BE INTERRUPTED IF THEY HAVE MODIFIED STRUCTURES

When executing a Lua script with EVAL or EVALSHA, Redis doesn’t allow any other read/ write commands to run. This can be convenient. But because Lua is a generalpurpose programming language, you can write scripts that never return, which could stop other clients from executing commands. To address this, Redis offers two options for stopping a script in Redis, depending on whether you’ve performed a Redis call that writes.

If your script hasn’t performed any calls that write (only reads), you can execute SCRIPT KILL if the script has been executing for longer than the configured lua-time-limit (check your Redis configuration file).

If your script has written to Redis, then killing the script could cause Redis to be in an inconsistent state. In that situation, the only way you can recover is to kill Redis with the SHUTDOWN NOSAVE command, which will cause Redis to lose any changes that have been made since the last snapshot, or since the last group of commands was written to the AOF.

Because of these limitations, you should always test your Lua scripts before running them in production.

As you may remember from chapter 8, listing 8.2 showed the creation of a status message. A copy of the original code we used for posting a status message appears next.

Listing 11.2Our function from listing 8.2 that creates a status message HASH
def create_status(conn, uid, message, **data):
    pipeline = conn.pipeline(True)
    pipeline.hget('user:%s' % uid, 'login')

Get the user’s login name from their user ID.

    pipeline.incr('status:id:')

Create a new ID for the status message.

    login, id = pipeline.execute()
    if not login:
        return None

Verify that we have a proper user account before posting.

    data.update({
        'message': message,
        'posted': time.time(),
        'id': id,
        'uid': uid,
        'login': login,
    })
    pipeline.hmset('status:%s' % id, data)

Prepare and set the data for the status message.

    pipeline.hincrby('user:%s' % uid, 'posts')

Record the fact that a status message has been posted.

    pipeline.execute()
    return id

Return the ID of the newly created status message.

Generally speaking, the performance penalty for making two round trips to Redis in order to post a status message isn’t very much—twice the latency of one round trip. But if we can reduce these two round trips to one, we may also be able to reduce the number of round trips for other examples where we make many round trips. Fewer round trips means lower latency for a given group of commands. Lower latency means less waiting, fewer required web servers, and higher performance for the entire system overall.

To review what happens when we post a new status message: we look up the user’s name in a HASH, increment a counter (to get a new ID), add data to a Redis HASH, and increment a counter in the user’s HASH. That doesn’t sound so bad in Lua; let’s give it a shot in the next listing, which shows the Lua script, with a Python wrapper that implements the same API as before.

Listing 11.3Creating a status message with Lua
def create_status(conn, uid, message, **data):

Take all of the arguments as before.

    args = [
        'message', message,
        'posted', time.time(),
        'uid', uid,
    ]
    for key, value in data.iteritems():
        args.append(key)
        args.append(value)

Prepare the arguments/attributes to be set on the status message.

    return create_status_lua(
        conn, ['user:%s' % uid, 'status:id:'], args)

Call the script.

create_status_lua = script_load('''
local login = redis.call('hget', KEYS[1], 'login')

Fetch the user’s login name from their ID; remember that tables in Lua are 1-indexed, not 0-indexed like Python and most other languages.

if not login then
    return false

If there’s no login, return that no login was found.

end
local id = redis.call('incr', KEYS[2])

Get a new ID for the status message.

local key = string.format('status:%s', id)

Prepare the destination key for the status message.

redis.call('hmset', key,
    'login', login,
    'id', id,
    unpack(ARGV))

Set the data for the status message.

redis.call('hincrby', KEYS[1], 'posts', 1)

Increment the post count of the user.

return id

Return the ID of the status message.

''')

This function performs all of the same operations that the previous all-Python version performed, only instead of needing two round trips to Redis with every call, it should only need one (the first call will load the script into Redis and then call the loaded script, but subsequent calls will only require one). This isn’t a big issue for posting status messages, but for many other problems that we’ve gone through in previous chapters, making multiple round trips can take more time than is necessary, or lend to WATCH/MULTI/EXEC contention.

WRITING KEYS THAT AREN’T A PART OF THE KEYS ARGUMENT TO THE SCRIPTIn the note in section 11.1.1, I mentioned that we should pass all keys to be modified as part of the keys argument of the script, yet here we’re writing a HASH based on a key that wasn’t passed. Doing this makes this Lua script incompatible with the future Redis cluster. Whether this operation is still correct in a noncluster sharded server scenario will depend on your sharding methods. I did this to highlight that you may need to do this kind of thing, but doing so prevents you from being able to use Redis cluster.

SCRIPT LOADERS AND HELPERSYou’ll notice in this first example that we have two major pieces. We have a Python function that handles the translation of the earlier API into the Lua script call, and we have the Lua script that’s loaded from our earlier script_load() function. We’ll continue this pattern for the remainder of the chapter, since the native API for Lua scripting (KEYS and ARGV) can be difficult to call in multiple contexts.

Since Redis 2.6 has been completed and released, libraries that support Redis scripting with Lua in the major languages should get better and more complete. On the Python side of things, a script loader similar to what we’ve written is already available in the source code repository for the redis-py project, and is currently available from the Python Package Index. We use our script loader due to its flexibility and ease of use when confronted with sharded network connections.

As our volume of interactions with Redis increased over time, we switched to usinglocks and semaphores to help reduce contention issues surrounding WATCH/MULTI/EXEC transactions. Let’s take a look at rewriting locks and semaphores to see if we might be able to further improve performance.