EBOOK – REDIS IN ACTION

This book covers the use of Redis, an in-memory database/data structure server.

open all | close all

4.1.1 Persisting to disk with snapshots

In Redis, we can create a point-in-time copy of in-memory data by creating a snapshot. After creation, these snapshots can be backed up, copied to other servers to create a clone of the server, or left for a future restart.

On the configuration side of things, snapshots are written to the file referenced as dbfilename in the configuration, and stored in the path referenced as dir. Until the next snapshot is performed, data written to Redis since the last snapshot started (and completed) would be lost if there were a crash caused by Redis, the system, or the hardware.

As an example, say that we have Redis running with 10 gigabytes of data currently in memory. A previous snapshot had been started at 2:35 p.m. and had finished. Now a snapshot is started at 3:06 p.m., and 35 keys are updated before the snapshot completes at 3:08 p.m. If some part of the system were to crash and prevent Redis from completing its snapshot operation between 3:06 p.m. and 3:08 p.m., any data written between 2:35 p.m. and now would be lost. But if the system were to crash just after the snapshot had completed, then only the updates to those 35 keys would be lost.

There are five methods to initiate a snapshot, which are listed as follows:

  • Any Redis client can initiate a snapshot by calling the BGSAVE command. On platforms that support BGSAVE (basically all platforms except for Windows), Redis will fork, 1 and the child process will write the snapshot to disk while the parent process continues to respond to commands.
  • A Redis client can also initiate a snapshot by calling the SAVE command, which causes Redis to stop responding to any/all commands until the snapshot completes. This command isn’t commonly used, except in situations where we need our data on disk, and either we’re okay waiting for it to complete, or we don’t have enough memory for a BGSAVE.
  • If Redis is configured with save lines, such as save 60 10000, Redis will automatically trigger a BGSAVE
    operation if 10,000 writes have occurred within 60 seconds since the last successful save has started (using the configuration option described). When multiple save lines are present, any time one of the rules match, a BGSAVE is triggered.
  • When Redis receives a request to shut down by the SHUTDOWN command, or it
    receives a standard TERM signal, Redis will perform a SAVE, blocking clients from
    performing any further commands, and then shut down.
  • If a Redis server connects to another Redis server and issues the SYNC command
    to begin replication, the master Redis server will start a BGSAVE operation if one
    isn’t already executing or recently completed. See section 4.2 for more information
    about replication.

When using only snapshots for saving data, you must remember that if a crash were to happen, you’d lose any data changed since the last snapshot. For some applications, this kind of loss isn’t acceptable, and you should look into using append-only file persistence, as described in section 4.1.2. But if your application can live with data loss, snapshots can be the right answer. Let’s look at a few scenarios and how you may want to configure Redis to get the snapshot persistence behavior you’re looking for.

DEVELOPMENT

For my personal development server, I’m mostly concerned with minimizing the overhead
of snapshots. To this end, and because I generally trust my hardware, I have a
single rule: save 900 1. The save option tells Redis that it should perform a BGSAVE
operation based on the subsequent two values. In this case, if at least one write has
occurred in at least 900 seconds (15 minutes) since the last BGSAVE, Redis will automatically
start a new BGSAVE.

If you’re planning on using snapshots on a production server, and you’re going to
be storing a lot of data, you’ll want to try to run a development server with the same or
similar hardware, the same save options, a similar set of data, and a similar expected
load. By setting up an environment equivalent to what you’ll be running in production,
you can make sure that you’re not snapshotting too often (wasting resources) or
too infrequently (leaving yourself open for data loss).

AGGREGATING LOGS

In the case of aggregating log files and analysis of page views, we really only need to
ask ourselves how much time we’re willing to lose if something crashes between
dumps. If we’re okay with losing up to an hour of work, then we can use save 3600 1
(there are 3600 seconds in an hour). But how might we recover if we were processing
logs?

To recover from data loss, we need to know what we lost in the first place. To
know what we lost, we need to keep a record of our progress while processing logs.
Let’s imagine that we have a function that’s called when new logs are ready to be processed.
This function is provided with a Redis connect, a path to where log files are
stored, and a callback that will process individual lines in the log file. With our function,
we can record which file we’re working on and the file position information as
we’re processing. A log-processing function that records this information can be seen
in the next listing.

Listing 4.2The process_logs() function that keeps progress information in Redis
def process_logs(conn, path, callback):

Our function will be provided
with a callback that will take
a connection and a log line,
calling methods on the
pipeline as necessary.

   current_file, offset = conn.mget(
      'progress:file', 'progress:position')

Get the current progress.


   pipe = conn.pipeline()

   def update_progress():

This closure is
meant primarily to
reduce the number of
duplicated lines later.

      pipe.mset({
         'progress:file': fname,
         'progress:position': offset

We want to update our
file and line number
offsets into the log file.

      })

      pipe.execute()

This will execute any
outstanding log updates,
as well as actually write
our file and line number
updates to Redis.

   for fname in sorted(os.listdir(path)):

Iterate over the log
files in sorted order.

      if fname < current_file:

Skip over files
that are before
the current file.

         continue

      inp = open(os.path.join(path, fname), 'rb')

      if fname == current_file:
         inp.seek(int(offset, 10))

If we’re continuing
a file, skip over the
parts that we’ve
already processed.

      else:
         offset = 0

      current_file = None

      for lno, line in enumerate(inp):

The enumerate function
iterates over a sequence (in
this case lines from a file),
and produces pairs
consisting of a numeric
sequence starting from 0,
and the original data.

         callback(pipe, line)

Handle the log line.

         offset = int(offset) + len(line)

Update our
information
about the offset
into the file.

         if not (lno+1) % 1000:
            update_progress()
      update_progress()

Write our progress back
to Redis every 1000 lines, or
when we’re done with a file.

      inp.close()


By keeping a record of our progress in Redis, we can pick up with processing logs if at
any point some part of the system crashes. And because we used MULTI/EXEC pipelines
as introduced in chapter 3, we ensure that the dump will only include processed log
information when it also includes progress information.

BIG DATA

When the amount of data that we store in Redis tends to be under a few gigabytes,
snapshotting can be the right answer. Redis will fork, save to disk, and finish the snapshot
faster than you can read this sentence. But as our Redis memory use grows over
time, so does the time to perform a fork operation for the BGSAVE. In situations where
Redis is using tens of gigabytes of memory, there isn’t a lot of free memory, or if we’re
running on a virtual machine, letting a BGSAVE occur may cause the system to pause
for extended periods of time, or may cause heavy use of system virtual memory, which
could degrade Redis’s performance to the point where it’s unusable.

This extended pausing (and how significant it is) will depend on what kind of system
we’re running on. Real hardware, VMWare virtualization, or KVM virtualization will generally
allow us to create a fork of a Redis process at roughly 10–20ms per gigabyte of memory that Redis is using. If our system is running within Xen virtualization, those
numbers can be closer to 200–300ms per gigabyte of memory used by Redis, depending
on the Xen configuration. So if we’re using 20 gigabytes of memory with Redis, running
BGSAVE on standard hardware will pause Redis for 200–400 milliseconds for the fork. If
we’re using Redis inside a Xen-virtualized machine (as is the case with Amazon EC2 and
some other cloud providers), that same fork will cause Redis to pause for 4–6 seconds.
You need to decide for your application whether this pause is okay.

To prevent forking from causing such issues, we may want to disable automatic saving
entirely. When automatic saving is disabled, we then need to manually call BGSAVE
(which has all of the same potential issues as before, only now we know when they will
happen), or we can call SAVE. With SAVE, Redis does block until the save is completed,
but because there’s no fork, there’s no fork delay. And because Redis doesn’t have to
fight with itself for resources, the snapshot will finish faster.

As a point of personal experience, I’ve run Redis servers that used 50 gigabytes of
memory on machines with 68 gigabytes of memory inside a cloud provider running
Xen virtualization. When trying to use BGSAVE with clients writing to Redis, forking
would take 15 seconds or more, followed by 15–20 minutes for the snapshot to complete.
But with SAVE, the snapshot would finish in 3–5 minutes. For our use, a daily
snapshot at 3 a.m. was sufficient, so we wrote scripts that would stop clients from trying
to access Redis, call SAVE, wait for the SAVE to finish, back up the resulting snapshot,
and then signal to the clients that they could continue.

Snapshots are great when we can deal with potentially substantial data loss in
Redis, but for many applications, 15 minutes or an hour or more of data loss or processing
time is too much. To allow Redis to keep more up-to-date information about
data in memory stored on disk, we can use append-only file persistence.

1 When a process forks, the underlying operating system makes a copy of the process. On Unix and Unix-like systems, the copying process is optimized such that, initially, all memory is shared between the child and parent processes. When either the parent or child process writes to memory, that memory will stop being shared.