e-Book - Redis in Action

This book covers the use of Redis, an in-memory database/data structure server.
  • Foreword
  • Preface
  • Acknowledgments
  • About this Book
  • About the Cover Illustration
  • Part 1: Getting Started
  • Part 2: Core concepts
  • Part 3: Next steps
  • Appendix A
  • Appendix B
  • Buy the paperback

    5.4.2 One Redis server per application component

    As countless developers have discovered during our increasing use of Redis, at some
    point we outgrow our first Redis server. Maybe we need to log more information,
    maybe we need more space for caching, or maybe we’ve already skipped ahead and
    are using one of the more advanced services described in later chapters. For whatever
    reason, we’ll need more servers.

    To help with the ease of transitioning to more servers, I recommend running one
    Redis server for every separate part of your application—one for logging, one for statistics,
    one for caching, one for cookies, and so forth. Note that you can run multiple
    Redis servers on a single machine; they just need to run on different ports. Alternatively,
    if you want to reduce your system administration load, you can also use different
    “databases” in Redis. Either way, by having different data split up into different key
    spaces, your transition to more or larger servers is somewhat simplified. Unfortunately,
    as your number of servers and/or Redis databases increases, managing and distributing
    configuration information for all of those servers becomes more of a chore.

    In the previous section, we used Redis as our source for configuration information
    about whether we should serve a maintenance page. We can again use Redis to
    tell us information about other Redis servers. More specifically, let’s use a single
    known Redis server as a directory of configuration information to discover how to
    connect to all of the other Redis servers that provide data for different application or
    service components. While we’re at it, we’ll build it in such a way that when configurations
    change, we’ll connect to the correct servers. Our implementation will be
    more generic than this example calls for, but I’m sure that after you start using this
    method for getting configuration information, you’ll start using it for non-Redis servers
    and services.

    We’ll build a function that will fetch a JSON-encoded configuration value from a
    key that’s named after the type of service and the application component that service
    is for. For example, if we wanted to fetch connection information for the Redis server
    that holds statistics, we’d fetch the key config:redis:statistics. The following listing
    shows the code for setting configurations.

    Listing 5.14The set_config() function
    def set_config(conn, type, component, config):
       conn.set(
          'config:%s:%s'%(type, component),
          json.dumps(config))
    

    With this set_config() function, we can set any JSON-encodable configuration that
    we may want. With a slight change in semantics and a get_config() function structured
    similarly to our earlier is_under_maintenance() function, we could replace
    is_under_maintenance(). Consult the following listing for a function that matches set_config() and will allow us to locally cache configuration information for 1 second,
    10 seconds, or 0 seconds, depending on our needs.

    Listing 5.15The get_config() function
    CONFIGS = {}
    CHECKED = {}
    
    def get_config(conn, type, component, wait=1):
       key = 'config:%s:%s'%(type, component)
       
    

       if CHECKED.get(key) < time.time() - wait:
    

    Check to see if we should update the configuration information about this component.

          CHECKED[key] = time.time()
    

    We can, so update the last time we checked this connection.

          config = json.loads(conn.get(key) or '{}')
    

    Fetch the configuration for this component.

          config = dict((str(k), config[k]) for k in config)
    

    Convert potentially Unicode keyword arguments into string keyword arguments.

          old_config = CONFIGS.get(key)
    
    

    Get the old configuration for this component.

          if config != old_config:
    

    If the configurations are different…

             CONFIGS[key] = config
    
    

    …update the configuration.

       return CONFIGS.get(key)
    

    Now that we have a pair of functions for getting and setting configurations, we can go
    farther. We started down this path of storing and fetching configurations in order to
    set up and create connections to a variety of different Redis servers. But the first argument
    to almost every function that we’ve written so far is a connection argument.
    Rather than needing to manually fetch connections for the variety of services that
    we’re using, let’s build a method to help us automatically connect to these services.