Documentation - Redise Pack

A guide to Redise Pack installation, operation and administration

open all | close all

5.4.2 One Redis server per application component

As countless developers have discovered during our increasing use of Redis, at some point we outgrow our first Redis server. Maybe we need to log more information, maybe we need more space for caching, or maybe we’ve already skipped ahead and are using one of the more advanced services described in later chapters. For whatever reason, we’ll need more servers.

To help with the ease of transitioning to more servers, I recommend running one Redis server for every separate part of your application—one for logging, one for statistics, one for caching, one for cookies, and so forth. Note that you can run multiple Redis servers on a single machine; they just need to run on different ports. Alternatively, if you want to reduce your system administration load, you can also use different “databases” in Redis. Either way, by having different data split up into different key spaces, your transition to more or larger servers is somewhat simplified. Unfortunately, as your number of servers and/or Redis databases increases, managing and distributing configuration information for all of those servers becomes more of a chore.

In the previous section, we used Redis as our source for configuration information about whether we should serve a maintenance page. We can again use Redis to tell us information about other Redis servers. More specifically, let’s use a single known Redis server as a directory of configuration information to discover how to connect to all of the other Redis servers that provide data for different application or service components. While we’re at it, we’ll build it in such a way that when configurations change, we’ll connect to the correct servers. Our implementation will be more generic than this example calls for, but I’m sure that after you start using this method for getting configuration information, you’ll start using it for non-Redis servers and services.

We’ll build a function that will fetch a JSON-encoded configuration value from a key that’s named after the type of service and the application component that service is for. For example, if we wanted to fetch connection information for the Redis server that holds statistics, we’d fetch the key config:redis:statistics. The following listing shows the code for setting configurations.

Listing 5.14The set_config() function
def set_config(conn, type, component, config):
   conn.set(
      'config:%s:%s'%(type, component),
      json.dumps(config))

With this set_config() function, we can set any JSON-encodable configuration that we may want. With a slight change in semantics and a get_config() function structured similarly to our earlier is_under_maintenance() function, we could replace is_under_maintenance(). Consult the following listing for a function that matches set_config() and will allow us to locally cache configuration information for 1 second, 10 seconds, or 0 seconds, depending on our needs.

Listing 5.15 The get_config() function
CONFIGS = {}
CHECKED = {}

def get_config(conn, type, component, wait=1):
   key = 'config:%s:%s'%(type, component)
   
   if CHECKED.get(key) < time.time() - wait:

Check to see if we should update the configuration information about this component.

      CHECKED[key] = time.time()

We can, so update the last time we checked this connection.

      config = json.loads(conn.get(key) or '{}')

Fetch the configuration for this component.

      config = dict((str(k), config[k]) for k in config)

Convert potentially Unicode keyword arguments into string keyword arguments.

      old_config = CONFIGS.get(key)

Get the old configuration for this component.

      if config != old_config:

If the configurations are different…

         CONFIGS[key] = config

…update the configuration.

   return CONFIGS.get(key)

Now that we have a pair of functions for getting and setting configurations, we can go farther. We started down this path of storing and fetching configurations in order to set up and create connections to a variety of different Redis servers. But the first argument to almost every function that we’ve written so far is a connection argument. Rather than needing to manually fetch connections for the variety of services that we’re using, let’s build a method to help us automatically connect to these services.