e-Book - Redis in Action

This book covers the use of Redis, an in-memory database/data structure server.
  • Foreword
  • Preface
  • Acknowledgments
  • About this Book
  • About the Cover Illustration
  • Part 1: Getting Started
  • Part 2: Core concepts
  • Part 3: Next steps
  • Appendix A
  • Appendix B
  • Buy the paperback

    4.1.2 Append-only file persistence

    In basic terms, append-only log files keep a record of data changes that occur by writing
    each change to the end of the file. In doing this, anyone could recover the entire
    dataset by replaying the append-only log from the beginning to the end. Redis has
    functionality that does this as well, and it’s enabled by setting the configuration option
    appendonly yes, as shown in listing 4.1. Table 4.1 shows the appendfsync options and
    how they affect file-write syncing to disk.

    FILE SYNCINGWhen writing files to disk, at least three things occur. The first is
    writing to a buffer, and this occurs when calling file.write() or its equivalent
    in other languages. When the data is in the buffer, the operating system can
    take that data and write it to disk at some point in the future. We can optionally
    take a second step and ask the operating system to write the data provided to
    disk when it next has a chance, with file.flush(), but this is only a request.
    Because data isn’t actually on disk until the operating system writes it to disk,
    we can tell the operating system to “sync” the files to disk, which will block until
    it’s completed. When that sync is completed, we can be fairly certain that our
    data is on disk and we can read it later if the system otherwise fails.

    Table 4.1Sync options to use with appendfsync

    Option

    How often syncing will occur

    always

    Every write command to Redis results in a write to disk. This
    slows Redis down substantially if used.

    everysec

    Once per second, explicitly syncs write commands to disk.

    no

    Lets the operating system control syncing to disk.

    If we were to set appendfsync always, every write to Redis would result in a write to
    disk, and we can ensure minimal data loss if Redis were to crash. Unfortunately,
    because we’re writing to disk with every write to Redis, we’re limited by disk performance,
    which is roughly 200 writes/second for a spinning disk, and maybe a few tens
    of thousands for an SSD (a solid-state drive).

    WARNING: SSDS AND appendfsync always You’ll want to be careful if you’re
    using SSDs with appendfsync always. Writing every change to disk as they
    happen, instead of letting the operating system group writes together as is the
    case with the other appendfsync options, has the potential to cause an
    extreme form of what is known as write amplification. By writing small amounts
    of data to the end of a file, you can reduce the lifetime of SSDs from years to
    just a few months in some cases.

    As a reasonable compromise between keeping data safe and keeping our write performance
    high, we can also set appendfsync everysec. This configuration will sync the
    append-only log once every second. For most common uses, we’ll likely not find significant
    performance penalties for syncing to disk every second compared to not
    using any sort of persistence. By syncing to disk every second, if the system were to
    crash, we could lose at most one second of data that had been written or updated in
    Redis. Also, in the case where the disk is unable to keep up with the write volume
    that’s happening, Redis would gracefully slow down to accommodate the maximum
    write rate of the drive.

    As you may guess, when setting appendfsync no, Redis doesn’t perform any
    explicit file syncing, leaving everything up to the operating system. There should be
    no performance penalties in this case, though if the system were to crash in one way
    or another, we’d lose an unknown and unpredictable amount of data. And if we’re
    using a hard drive that isn’t fast enough for our write load, Redis would perform
    fine until the buffers to write data to disk were filled, at which point Redis would get
    very slow as it got blocked from writing. For these reasons, I generally discourage
    the use of this configuration option, and include its description and semantics here
    for completeness.

    Append-only files are flexible, offering a variety of options to ensure that almost
    every level of paranoia can be addressed. But there’s a dark side to AOF persistence,
    and that is file size.