EBOOK – REDIS IN ACTION

This book covers the use of Redis, an in-memory database/data structure server.

  • Foreword
  • Preface
  • Acknowledgments
  • About this Book
  • About the Cover Illustration
  • Part 1: Getting Started
  • Part 2: Core concepts
  • Part 3: Next steps
  • Appendix A
  • Appendix B
  • Buy the paperback

    11.1.2 Creating a new status message

    As we build Lua scripts to perform a set of operations, it’s good to start with a short example that isn’t terribly complicated or structure-intensive. In this case, we’ll start by writing a Lua script combined with some wrapper code to post a status message.

    LUA SCRIPTS—AS ATOMIC AS SINGLE COMMANDS OR MULTI/EXECAs you already
    know, individual commands in Redis are atomic in that they’re run one at a
    time. With MULTI/EXEC, you can prevent other commands from running
    while you’re executing multiple commands. But to Redis, EVAL and EVALSHA
    are each considered to be a (very complicated) command, so they’re executed
    without letting any other structure operations occur.

    LUA SCRIPTS—CAN’T BE INTERRUPTED IF THEY HAVE MODIFIED STRUCTURESWhen
    executing a Lua script with EVAL or EVALSHA, Redis doesn’t allow any other read/
    write commands to run. This can be convenient. But because Lua is a generalpurpose
    programming language, you can write scripts that never return, which
    could stop other clients from executing commands. To address this, Redis offers
    two options for stopping a script in Redis, depending on whether you’ve performed
    a Redis call that writes.

    If your script hasn’t performed any calls that write (only reads), you can execute
    SCRIPT KILL if the script has been executing for longer than the configured
    lua-time-limit (check your Redis configuration file).

    If your script has written to Redis, then killing the script could cause Redis to
    be in an inconsistent state. In that situation, the only way you can recover is to
    kill Redis with the SHUTDOWN NOSAVE command, which will cause Redis to lose
    any changes that have been made since the last snapshot, or since the last
    group of commands was written to the AOF.

    Because of these limitations, you should always test your Lua scripts before
    running them in production.

    As you may remember from chapter 8, listing 8.2 showed the creation of a status message.
    A copy of the original code we used for posting a status message appears next.

    Listing 11.2Our function from listing 8.2 that creates a status message HASH
    def create_status(conn, uid, message, **data):
        pipeline = conn.pipeline(True)
    
        pipeline.hget('user:%s' % uid, 'login')
    

    Get the user’s login name from their user ID.

        pipeline.incr('status:id:')
    

    Create a new ID for the status message.

        login, id = pipeline.execute()
    
        if not login:
            return None
    

    Verify that we have a proper user account before posting.

        data.update({
            'message': message,
            'posted': time.time(),
            'id': id,
            'uid': uid,
            'login': login,
        })
        pipeline.hmset('status:%s' % id, data)
    

    Prepare and set the data for the status message.

        pipeline.hincrby('user:%s' % uid, 'posts')
    

    Record the fact that a status message has been posted.

        pipeline.execute()
    
        return id
    

    Return the ID of the newly created status message.

    Generally speaking, the performance penalty for making two round trips to Redis in
    order to post a status message isn’t very much—twice the latency of one round trip.
    But if we can reduce these two round trips to one, we may also be able to reduce the
    number of round trips for other examples where we make many round trips. Fewer
    round trips means lower latency for a given group of commands. Lower latency means less waiting, fewer required web servers, and higher performance for the
    entire system overall.

    To review what happens when we post a new status message: we look up the user’s
    name in a HASH, increment a counter (to get a new ID), add data to a Redis HASH, and
    increment a counter in the user’s HASH. That doesn’t sound so bad in Lua; let’s give it
    a shot in the next listing, which shows the Lua script, with a Python wrapper that
    implements the same API as before.

    Listing 11.3Creating a status message with Lua
    def create_status(conn, uid, message, **data):
    

    Take all of the arguments as before.

        args = [
            'message', message,
            'posted', time.time(),
            'uid', uid,
        ]
        for key, value in data.iteritems():
            args.append(key)
            args.append(value)
    

    Prepare the arguments/attributes to be set on the status message.

        return create_status_lua(
            conn, ['user:%s' % uid, 'status:id:'], args)
    

    Call the script.

    create_status_lua = script_load('''
    
    local login = redis.call('hget', KEYS[1], 'login')
    

    Fetch the user’s login name from their ID; remember that tables in Lua are 1-indexed, not 0-indexed like Python and most other languages.

    if not login then
        return false
    

    If there’s no login, return that no login was found.

    end
    
    local id = redis.call('incr', KEYS[2])
    

    Get a new ID for the status message.

    local key = string.format('status:%s', id)
    

    Prepare the destination key for the status message.

    redis.call('hmset', key,
        'login', login,
        'id', id,
        unpack(ARGV))
    

    Set the data for the status message.

    redis.call('hincrby', KEYS[1], 'posts', 1)
    

    Increment the post count of the user.

    return id
    

    Return the ID of the status message.

    ''')
    

    This function performs all of the same operations that the previous all-Python version
    performed, only instead of needing two round trips to Redis with every call, it should
    only need one (the first call will load the script into Redis and then call the loaded
    script, but subsequent calls will only require one). This isn’t a big issue for posting status
    messages, but for many other problems that we’ve gone through in previous chapters,
    making multiple round trips can take more time than is necessary, or lend to
    WATCH/MULTI/EXEC contention.

    WRITING KEYS THAT AREN’T A PART OF THE KEYS ARGUMENT TO THE SCRIPTIn the
    note in section 11.1.1, I mentioned that we should pass all keys to be modified
    as part of the keys argument of the script, yet here we’re writing a HASH based
    on a key that wasn’t passed. Doing this makes this Lua script incompatible with the future Redis cluster. Whether this operation is still correct in a noncluster
    sharded server scenario will depend on your sharding methods. I did this to
    highlight that you may need to do this kind of thing, but doing so prevents you
    from being able to use Redis cluster.

    SCRIPT LOADERS AND HELPERSYou’ll notice in this first example that we have
    two major pieces. We have a Python function that handles the translation of
    the earlier API into the Lua script call, and we have the Lua script that’s
    loaded from our earlier script_load() function. We’ll continue this pattern
    for the remainder of the chapter, since the native API for Lua scripting (KEYS
    and ARGV) can be difficult to call in multiple contexts.

    Since Redis 2.6 has been completed and released, libraries that support Redis scripting
    with Lua in the major languages should get better and more complete. On the
    Python side of things, a script loader similar to what we’ve written is already available
    in the source code repository for the redis-py project, and is currently available from
    the Python Package Index. We use our script loader due to its flexibility and ease of
    use when confronted with sharded network connections.

    As our volume of interactions with Redis increased over time, we switched to using
    locks and semaphores to help reduce contention issues surrounding WATCH/MULTI/
    EXEC transactions. Let’s take a look at rewriting locks and semaphores to see if we
    might be able to further improve performance.