This book covers the use of Redis, an in-memory database/data structure server.

open all | close all

5.1 Logging to Redis

As we build applications and services, being able to discover information about the
running system becomes increasingly important. Being able to dig into that information
to diagnose problems, discover problems before they become severe, or even just
to discover information about users—these all necessitate logging.

In the world of Linux and Unix, there are two common logging methods. The first
is logging to a file, where over time we write individual log lines to a file, and every
once in a while, we write to a new file. Many thousands of pieces of software have been
written do this (including Redis itself). But this method can run into issues because we
have many different services writing to a variety of log files, each with a different way
of rolling them over, and no common way of easily taking all of the log files and doing
something useful with them.

Running on TCP and UDP port 514 of almost every Unix and Linux server available
is a service called syslog, the second common logging method. Syslog accepts log messages
from any program that sends it a message and routes those messages to various
on-disk log files, handling rotation and deletion of old logs. With configuration, it can
even forward messages to other servers for further processing. As a service, it’s far
more convenient than logging to files directly, because all of the special log file rotation
and deletion is already handled for us.

REPLACING SYSLOGWhether you end up using the logging methods
described here, you owe it to yourself to consider replacing your current syslog
daemon (which is likely Rsyslogd) with syslog-ng. Having used and configured
both systems, I find that the configuration language that syslog-ng
provides for directing log messages is easier to use. And though I don’t have
the space or time to build it in this book, building a service that consumes syslog
messages and puts them into Redis is a great way to offer a layer of indirection
between what needs to happen now for processing a request, and what
can happen later (like logging or updating counters).

Having logs available in files on a single server (thanks to syslog forwarding) is a great
long-term plan with logging (remember to back them up). In this section, we’ll talk
about using Redis as a way of keeping more time-sensitive logs, to function as a
replacement for syslog messages being stored in the short term. Our first view into
changing logs is a continuously updated stream of recent log messages.