This book covers the use of Redis, an in-memory database/data structure server.

open all | close all

4.1.2 Append-only file persistence

In basic terms, append-only log files keep a record of data changes that occur by writing
each change to the end of the file. In doing this, anyone could recover the entire
dataset by replaying the append-only log from the beginning to the end. Redis has
functionality that does this as well, and it’s enabled by setting the configuration option
appendonly yes, as shown in listing 4.1. Table 4.1 shows the appendfsync options and
how they affect file-write syncing to disk.

FILE SYNCINGWhen writing files to disk, at least three things occur. The first is
writing to a buffer, and this occurs when calling file.write() or its equivalent
in other languages. When the data is in the buffer, the operating system can
take that data and write it to disk at some point in the future. We can optionally
take a second step and ask the operating system to write the data provided to
disk when it next has a chance, with file.flush(), but this is only a request.
Because data isn’t actually on disk until the operating system writes it to disk,
we can tell the operating system to “sync” the files to disk, which will block until
it’s completed. When that sync is completed, we can be fairly certain that our
data is on disk and we can read it later if the system otherwise fails.

Table 4.1Sync options to use with appendfsync


How often syncing will occur


Every write command to Redis results in a write to disk. This
slows Redis down substantially if used.


Once per second, explicitly syncs write commands to disk.


Lets the operating system control syncing to disk.

If we were to set appendfsync always, every write to Redis would result in a write to
disk, and we can ensure minimal data loss if Redis were to crash. Unfortunately,
because we’re writing to disk with every write to Redis, we’re limited by disk performance,
which is roughly 200 writes/second for a spinning disk, and maybe a few tens
of thousands for an SSD (a solid-state drive).

WARNING: SSDS AND appendfsync always You’ll want to be careful if you’re
using SSDs with appendfsync always. Writing every change to disk as they
happen, instead of letting the operating system group writes together as is the
case with the other appendfsync options, has the potential to cause an
extreme form of what is known as write amplification. By writing small amounts
of data to the end of a file, you can reduce the lifetime of SSDs from years to
just a few months in some cases.

As a reasonable compromise between keeping data safe and keeping our write performance
high, we can also set appendfsync everysec. This configuration will sync the
append-only log once every second. For most common uses, we’ll likely not find significant
performance penalties for syncing to disk every second compared to not
using any sort of persistence. By syncing to disk every second, if the system were to
crash, we could lose at most one second of data that had been written or updated in
Redis. Also, in the case where the disk is unable to keep up with the write volume
that’s happening, Redis would gracefully slow down to accommodate the maximum
write rate of the drive.

As you may guess, when setting appendfsync no, Redis doesn’t perform any
explicit file syncing, leaving everything up to the operating system. There should be
no performance penalties in this case, though if the system were to crash in one way
or another, we’d lose an unknown and unpredictable amount of data. And if we’re
using a hard drive that isn’t fast enough for our write load, Redis would perform
fine until the buffers to write data to disk were filled, at which point Redis would get
very slow as it got blocked from writing. For these reasons, I generally discourage
the use of this configuration option, and include its description and semantics here
for completeness.

Append-only files are flexible, offering a variety of options to ensure that almost
every level of paranoia can be addressed. But there’s a dark side to AOF persistence,
and that is file size.