This book covers the use of Redis, an in-memory database/data structure server.

open all | close all

5.4.2 One Redis server per application component

As countless developers have discovered during our increasing use of Redis, at some
point we outgrow our first Redis server. Maybe we need to log more information,
maybe we need more space for caching, or maybe we’ve already skipped ahead and
are using one of the more advanced services described in later chapters. For whatever
reason, we’ll need more servers.

To help with the ease of transitioning to more servers, I recommend running one
Redis server for every separate part of your application—one for logging, one for statistics,
one for caching, one for cookies, and so forth. Note that you can run multiple
Redis servers on a single machine; they just need to run on different ports. Alternatively,
if you want to reduce your system administration load, you can also use different
“databases” in Redis. Either way, by having different data split up into different key
spaces, your transition to more or larger servers is somewhat simplified. Unfortunately,
as your number of servers and/or Redis databases increases, managing and distributing
configuration information for all of those servers becomes more of a chore.

In the previous section, we used Redis as our source for configuration information
about whether we should serve a maintenance page. We can again use Redis to
tell us information about other Redis servers. More specifically, let’s use a single
known Redis server as a directory of configuration information to discover how to
connect to all of the other Redis servers that provide data for different application or
service components. While we’re at it, we’ll build it in such a way that when configurations
change, we’ll connect to the correct servers. Our implementation will be
more generic than this example calls for, but I’m sure that after you start using this
method for getting configuration information, you’ll start using it for non-Redis servers
and services.

We’ll build a function that will fetch a JSON-encoded configuration value from a
key that’s named after the type of service and the application component that service
is for. For example, if we wanted to fetch connection information for the Redis server
that holds statistics, we’d fetch the key config:redis:statistics. The following listing
shows the code for setting configurations.

Listing 5.14The set_config() function
def set_config(conn, type, component, config):
      'config:%s:%s'%(type, component),

With this set_config() function, we can set any JSON-encodable configuration that
we may want. With a slight change in semantics and a get_config() function structured
similarly to our earlier is_under_maintenance() function, we could replace
is_under_maintenance(). Consult the following listing for a function that matches set_config() and will allow us to locally cache configuration information for 1 second,
10 seconds, or 0 seconds, depending on our needs.

Listing 5.15The get_config() function

def get_config(conn, type, component, wait=1):
   key = 'config:%s:%s'%(type, component)

   if CHECKED.get(key) < time.time() - wait:

Check to see if we should update the configuration information about this component.

      CHECKED[key] = time.time()

We can, so update the last time we checked this connection.

      config = json.loads(conn.get(key) or '{}')

Fetch the configuration for this component.

      config = dict((str(k), config[k]) for k in config)

Convert potentially Unicode keyword arguments into string keyword arguments.

      old_config = CONFIGS.get(key)

Get the old configuration for this component.

      if config != old_config:

If the configurations are different…

         CONFIGS[key] = config

…update the configuration.

   return CONFIGS.get(key)

Now that we have a pair of functions for getting and setting configurations, we can go
farther. We started down this path of storing and fetching configurations in order to
set up and create connections to a variety of different Redis servers. But the first argument
to almost every function that we’ve written so far is a connection argument.
Rather than needing to manually fetch connections for the variety of services that
we’re using, let’s build a method to help us automatically connect to these services.