e-Book - Redis in Action

This book covers the use of Redis, an in-memory database/data structure server.
  • Foreword
  • Preface
  • Acknowledgments
  • About this Book
  • About the Cover Illustration
  • Part 1: Getting Started
  • Part 2: Core concepts
  • Part 3: Next steps
  • Appendix A
  • Appendix B
  • Buy the paperback

    11.3.2 Improving the marketplace, again

    In section 6.2, we revisited the marketplace example we introduced in section 4.4,
    replacing our use of WATCH, MULTI, and EXEC with locks, and showed how using
    coarse-grained and fine-grained locks in Redis can help reduce contention and
    improve performance.

    In this section, we’ll again work on the marketplace, further improving performance
    by removing the locks completely and moving our code into a Lua script.

    Let’s first look at our marketplace code with a lock. As a review of what goes on,
    first the lock is acquired, and then we watch the buyer’s user information HASH and let
    the buyer purchase the item if they have enough money. The original function from
    section 6.2 appears next.

    Listing 11.12The purchase item with lock function from section 6.2
    def purchase_item_with_lock(conn, buyerid, itemid, sellerid):
       buyer = "users:%s" % buyerid
       seller = "users:%s" % sellerid
       item = "%s.%s" % (itemid, sellerid)
       inventory = "inventory:%s" % buyerid
       end = time.time() + 30
    
       locked = acquire_lock(conn, 'market:')
    

    Get the lock

       if not locked:
          return False
    
       pipe = conn.pipeline(True)
       try:
          while time.time() < end:
             try:
                pipe.watch(buyer)
    
                pipe.zscore("market:", item)
                pipe.hget(buyer, 'funds')
                price, funds = pipe.execute()
                if price is None or price > funds:
                   pipe.unwatch()
                   return None
    

    Check for a sold item or insufficient funds.

                pipe.hincrby(seller, int(price))
                pipe.hincrby(buyerid, int(-price))
                pipe.sadd(inventory, itemid)
                pipe.zrem("market:", item)
                pipe.execute()    
    

    Transfer funds from the buyer to the seller, and transfer the item to the buyer.

                return True
             except redis.exceptions.WatchError:
                pass
       finally:
    
          release_lock(conn, 'market:', locked)
    

    Release the lock.

    Despite using a lock, we still needed to watch the buyer’s user information HASH to
    ensure that enough money was available at the time of purchase. And because of that,
    we have the worst of both worlds: chunks of code to handle locking, and other chunks
    of code to handle the potential WATCH errors. Of all of the solutions we’ve seen so far,
    this one is a prime candidate for rewriting in Lua.

    When rewriting this function in Lua, we can do away with the locking, the WATCH/
    MULTI/EXEC transactions, and even timeouts. Our code becomes straightforward and
    simple: make sure the item is available, make sure the buyer has enough money, transfer
    the item to the buyer, and transfer the buyer’s money to the seller. The following
    listing shows the rewritten item-purchasing function.

    Listing 11.13The purchase item function rewritten with Lua
    def purchase_item(conn, buyerid, itemid, sellerid):
    
       buyer = "users:%s" % buyerid
       seller = "users:%s" % sellerid
       item = "%s.%s"%(itemid, sellerid)
       inventory = "inventory:%s" % buyerid
    

    Prepare all of the keys and arguments for the Lua script.

       return purchase_item_lua(conn,
          ['market:', buyer, seller, inventory], [item, itemid])
    
    purchase_item_lua = script_load('''
    
    local price = tonumber(redis.call('zscore', KEYS[1], ARGV[1]))
    local funds = tonumber(redis.call('hget', KEYS[2], 'funds'))
    

    Get the item price and the buyer’s available funds.

    if price and funds and funds >= price then
       redis.call('hincrby', KEYS[3], 'funds', price)
       redis.call('hincrby', KEYS[2], 'funds', -price)
       redis.call('sadd', KEYS[4], ARGV[2])
       redis.call('zrem', KEYS[1], ARGV[1])
    

    If the item is still available and the buyer has enough money, transfer the item.

       return true
    

    Signify that the purchase
    completed successfully.

    end
    ''')
    

    Just comparing the two code listings, the Lua-based item-purchase code is far easier to
    understand. And without the multiple round trips to complete a single purchase
    (locking, watching, fetching price and available money, then the purchase, and then
    the unlocking), it’s obvious that purchasing an item will be even faster than the finegrained
    locking that we used in chapter 6. But how much faster?

    Exercise: Rewrite item listing in Lua

    We rewrote the purchase-item function in Lua for our benchmarks, but can you rewrite
    the original item-listing function from section 4.4.2 into Lua? Hint: The source code
    for this chapter includes the answer to this exercise, just as the source code for each
    of the other chapters includes the solutions to almost all of the exercises.

    We now have an item-purchase function rewritten in Lua, and if you perform the exercise,
    you’ll also have an item-listing function written in Lua. If you read the hint to our
    exercise, you’ll notice that we also rewrote the item-listing function in Lua. You may
    remember that at the end of section 6.2.4, we ran some benchmarks to compare the
    performance of WATCH/MULTI/EXEC transactions against coarse-grained and finegrained
    locks. We reran the benchmark for five listing and five buying processes using
    our newly rewritten Lua versions, to produce the last row in table 11.4.

    Table 11.4Performance of Lua compared with no locking, coarse-grained locks, and fine-grained locks over 60 seconds

    Listed items

    Bought items

    Purchase retries

    Average wait per purchase

    5 listers, 5 buyers, no lock

    206,000

    <600

    161,000

    498ms

    5 listers, 5 buyers, with lock

    21,000

    20,500

    0

    14ms

    5 listers, 5 buyers, with fine-grained lock

    116,000

    111,000

    0

    <3ms

    5 listers, 5 buyers, using Lua

    505,000

    480,000

    0

    <1ms

    As in other cases where we’ve moved functionality into Lua, we see a substantial performance
    increase. Numerically, we see an improvement in listing and buying
    performance of more than 4.25 times compared with fine-grained locking, and
    see latencies of under 1 millisecond to execute a purchase (actual latencies were
    consistently around .61 milliseconds). From this table, we can see the performance
    advantages of coarse-grained locks over WATCH/MULTI/EXEC, fine-grained locks over coarse-grained locks, and Lua over fine-grained locks. That said, try to remember
    that while Lua can offer extraordinary performance advantages (and substantial
    code simplification in some cases), Lua scripting in Redis is limited to data we can
    access from within Lua and Redis, whereas there are no such limits when using locks
    or WATCH/MULTI/EXEC transactions.

    Now that you’ve seen some of the amazing performance advantages that are available
    with Lua, let’s look at an example where we can save memory with Lua.