e-Book - Redis in Action

This book covers the use of Redis, an in-memory database/data structure server.
  • Foreword
  • Preface
  • Acknowledgments
  • About this Book
  • About the Cover Illustration
  • Part 1: Getting Started
  • Part 2: Core concepts
  • Part 3: Next steps
  • Appendix A
  • Appendix B
  • Buy the paperback

    11.1.1 Loading Lua scripts into Redis

    Some older (and still used) Python Redis libraries for Redis 2.6 don’t yet offer the
    capability to load or execute Lua scripts directly, so we’ll spend a few moments to create
    a loader for the scripts. To load scripts into Redis, there’s a two-part command
    called SCRIPT LOAD that, when provided with a string that’s a Lua script, will store the
    script for later execution and return the SHA1 hash of the script. Later, when we want
    to execute that script, we run the Redis command EVALSHA with the hash that was
    returned by Redis, along with any arguments that the script needs.

    Our code for doing these operations will be inspired by the current Python Redis
    code. (We use our method primarily because it allows for using any connection we
    want without having to explicitly create new scripting objects, which can be useful
    when dealing with server sharding.) When we pass a string to our script_load()
    function, it’ll create a function that can later be called to execute the script in Redis.
    When calling the object to execute the script, we must provide a Redis connection,
    which will then call SCRIPT LOAD on its first call, followed by EVALSHA for all future
    calls. The script_load() function is shown in the following listing.

    Listing 11.1A function that loads scripts to be called later
    def script_load(script):
    
        sha = [None]
    

    Store the cached SHA1 hash of the result of SCRIPT LOAD in a list so we can change it later from within the call() function.

        def call(conn, keys=[], args=[], force_eval=False):
    

    When calling the loaded script, we must provide a connection, a set of keys that the script will manipulate, and any other arguments to the function.

                if not force_eval:
    
                    if not sha[0]:
    

    We’ll only try loading the script if the SHA1 hash isn’t cached.

                            sha[0] = conn.execute_command(
                                "SCRIPT", "LOAD", script, parse="LOAD")
    
                    try:
    
                                return conn.execute_command(
                                   "EVALSHA", sha[0], len(keys), *(keys+args))
    

    Execute the command from the cached SHA1.

                            except redis.exceptions.ResponseError as msg:
    
                                if not msg.args[0].startswith("NOSCRIPT"):
                                    raise
    

    If the error was unrelated to a missing script, raise the exception again.
    If we received a script-related error, or if we need to force-execute the script, directly execute the script, which will automatically cache the script on the server (with the same SHA1 that we’ve already cached) when done.

                    return conn.execute_command(
                        "EVAL", script, len(keys), *(keys+args))
    
                return call
    

    Return the function that automatically loads and executes scripts when called.

    You’ll notice that in addition to our SCRIPT LOAD and EVALSHA calls, we captured an
    exception that can happen if we’ve cached a script’s SHA1 hash locally, but the server
    doesn’t know about it. This can happen if the server were to be restarted, if someone
    had executed the SCRIPT FLUSH command to clean out the script cache, or if we provided
    connections to two different Redis servers to our function at different times. If
    we discover that the script is missing, we execute the script directly with EVAL, which
    caches the script in addition to executing it. Also, we allow clients to directly execute
    the script, which can be useful when executing a Lua script as part of a transaction or
    other pipelined sequence.

    KEYS AND ARGUMENTS TO LUA SCRIPTSBuried inside our script loader, you
    may have noticed that calling a script in Lua takes three arguments. The first
    is a Redis connection, which should be standard by now. The second argument
    is a list of keys. The third is a list of arguments to the function.

    The difference between keys and arguments is that you’re supposed to pass
    all of the keys that the script will be reading or writing as part of the keys
    argument. This is to potentially let other layers verify that all of the keys are
    on the same shard if you were using multiserver sharding techniques like
    those described in chapter 10.

    When Redis cluster is released, which will offer automatic multiserver sharding,
    keys will be checked before a script is run, and will return an error if any
    keys that aren’t on the same server are accessed.

    The second list of arguments has no such limitation and is meant to hold data
    to be used inside the Lua call.

    Let’s try it out in the console for a simple example to get started.

    >>> ret_1 = script_load("return 1")
    

    Most uses will load the script and store a reference to the returned function.

    >>> ret_1(conn)
    

    We can then call the function by passing the connection object and any desired arguments.

    1L
    

    Results will be returned and converted into the appropriate Python types, when possible.

    As you can see in this example, we created a simple script whose only purpose is to
    return a value of 1. When we call it with the connection object, the script is loaded
    and then executed, resulting in the value 1 being returned.

    RETURNING NON-STRING AND NON-INTEGER VALUES FROM LUA

    Due to limitations in how Lua allows data to be passed in and out of it, some data types
    that are available in Lua aren’t allowed to be passed out, or are altered before being
    returned. Table 11.1 shows how this data is altered upon return.

    Table 11.1Values returned from Lua and what they’re translated into

    Lua value

    What happens during conversion to Python

    true

    Turns into 1

    false

    Turns into None

    nil

    Doesn’t turn into anything, and stops remaining values in a table from being returned

    1.5 (or any other float)

    Fractional part is discarded, turning it into an integer

    1e30 (or any other large float)

    Is turned into the minimum integer for your version of Python

    "strings"

    Unchanged

    1 (or any other integer +/-253-1)

    Integer is returned unchanged

    Because of the ambiguity that results when returning a variety of data types, you
    should do your best to explicitly return strings whenever possible, and perform any
    parsing manually. We’ll only be returning Booleans, strings, integers, and Lua tables
    (which are turned into Python lists) for our examples.

    Now that we can load and execute scripts, let’s get started with a simple example
    from chapter 8, creating a status message.