This book covers the use of Redis, an in-memory database/data structure server.

open all | close all

10.2.1 Handling shard configuration

As you may remember from chapter 5, we wrote a method to create and use named
Redis configurations automatically. This method used a Python decorator to fetch
configuration information, compare it with preexisting configuration information,
and create or reuse an existing connection. We’ll extend this idea to add support for
sharded connections. With these changes, we can use much of our code developed in
chapter 9 with minor changes.

To get started, first let’s make a simple function that uses the same configuration
layout that we used in our connection decorator from chapter 5. If you remember, we
use JSON-encoded dictionaries to store connection information for Redis in keys of
the format config:redis:<component>. Pulling out the connection management
part of the decorator, we end up with a simple function to create or reuse a Redis connection,
based on a named configuration, shown here.

Listing 10.1A function to get a Redis connection based on a named configuration
def get_redis_connection(component, wait=1):
   key = 'config:redis:' + component
   old_config = CONFIGS.get(key, object())

Fetch the old configuration, if any.

   config = get_config(
      config_connection, 'redis', component, wait)

Get the new configuration, if any.

   if config != old_config:
      REDIS_CONNECTIONS[key] = redis.Redis(**config)

If the new and old configuration don’t match, create a new connection.

   return REDIS_CONNECTIONS.get(key)

Return the desired connection object.

This simple function fetches the previously known as well as the current configuration.
If they’re different, it updates the known configuration, creates a new connection,
and then stores and returns that new connection. If the configuration hasn’t
changed, it returns the previous connection.

When we have the ability to fetch connections easily, we should also add support
for the creation of sharded Redis connections, so even if our later decorators aren’t
useful in every situation, we can still easily create and use sharded connections. To
connect to a new sharded connection, we’ll use the same configuration methods,
though sharded configuration will be a little different. For example, shard 7 of component
logs will be stored at a key named config:redis:logs:7. This naming scheme will let us reuse the existing connection and configuration code we already
have. Our function to get a sharded connection is in the following listing.

Listing 10.2Fetch a connection based on shard information
def get_sharded_connection(component, key, shard_count, wait=1):
   shard = shard_key(component, 'x'+str(key), shard_count, 2)

Calculate the shard ID of the form <component>:<shard>.

   return get_redis_connection(shard, wait)

Return the connection.

Now that we have a simple method of fetching a connection to a Redis server that’s
sharded, we can create a decorator like we saw in chapter 5 that creates a sharded connection