e-Book - Redis in Action

This book covers the use of Redis, an in-memory database/data structure server.
  • Foreword
  • Preface
  • Acknowledgments
  • About this Book
  • About the Cover Illustration
  • Part 1: Getting Started
  • Part 2: Core concepts
  • Part 3: Next steps
  • Appendix A
  • Appendix B
  • Buy the paperback

    10.2.1 Handling shard configuration

    As you may remember from chapter 5, we wrote a method to create and use named
    Redis configurations automatically. This method used a Python decorator to fetch
    configuration information, compare it with preexisting configuration information,
    and create or reuse an existing connection. We’ll extend this idea to add support for
    sharded connections. With these changes, we can use much of our code developed in
    chapter 9 with minor changes.

    To get started, first let’s make a simple function that uses the same configuration
    layout that we used in our connection decorator from chapter 5. If you remember, we
    use JSON-encoded dictionaries to store connection information for Redis in keys of
    the format config:redis:<component>. Pulling out the connection management
    part of the decorator, we end up with a simple function to create or reuse a Redis connection,
    based on a named configuration, shown here.

    Listing 10.1A function to get a Redis connection based on a named configuration
    def get_redis_connection(component, wait=1):
       key = 'config:redis:' + component
       old_config = CONFIGS.get(key, object())

    Fetch the old configuration, if any.

       config = get_config(
          config_connection, 'redis', component, wait)

    Get the new configuration, if any.

       if config != old_config:
          REDIS_CONNECTIONS[key] = redis.Redis(**config)

    If the new and old configuration don’t match, create a new connection.

       return REDIS_CONNECTIONS.get(key)

    Return the desired connection object.

    This simple function fetches the previously known as well as the current configuration.
    If they’re different, it updates the known configuration, creates a new connection,
    and then stores and returns that new connection. If the configuration hasn’t
    changed, it returns the previous connection.

    When we have the ability to fetch connections easily, we should also add support
    for the creation of sharded Redis connections, so even if our later decorators aren’t
    useful in every situation, we can still easily create and use sharded connections. To
    connect to a new sharded connection, we’ll use the same configuration methods,
    though sharded configuration will be a little different. For example, shard 7 of component
    logs will be stored at a key named config:redis:logs:7. This naming scheme will let us reuse the existing connection and configuration code we already
    have. Our function to get a sharded connection is in the following listing.

    Listing 10.2Fetch a connection based on shard information
    def get_sharded_connection(component, key, shard_count, wait=1):
       shard = shard_key(component, 'x'+str(key), shard_count, 2)

    Calculate the shard ID of the form <component>:<shard>.

       return get_redis_connection(shard, wait)

    Return the connection.

    Now that we have a simple method of fetching a connection to a Redis server that’s
    sharded, we can create a decorator like we saw in chapter 5 that creates a sharded connection