As we monitor our application, being able to gather information over time becomes
ever more important. Code changes (that can affect how quickly our site responds,
and subsequently how many pages we serve), new advertising campaigns, or new users
to our system can all radically change the number of pages that are loaded on a site.
Subsequently, any number of other performance metrics may change. But if we aren’t
recording any metrics, then it’s impossible to know how they’re changing, or whether
we’re doing better or worse.
In an attempt to start gathering metrics to watch and analyze, we’ll build a tool to
keep named counters over time (counters with names like site hits, sales, or database queries
can be crucial). Each of these counters will store the most recent 120 samples at a
variety of time precisions (like 1 second, 5 seconds, 1 minute, and so on). Both the number
of samples and the selection of precisions to record can be customized as necessary.
The first step for keeping counters is actually storing the counters themselves.
UPDATING A COUNTER
In order to update counters, we’ll need to store the actual counter information. For each counter and precision, like site hits and 5 seconds, we’ll keep a HASH that stores information about the number of site hits that have occurred in each 5-second time slice. The keys in the hash will be the start of the time slice, and the value will be the number of hits. Figure 5.1 shows a selection of data from a hit counter with 5-second time slices.
As we start to use counters, we need to record what counters have been written to so
that we can clear out old data. For this, we need an ordered sequence that lets us iterate
one by one over its entries, and that also doesn’t allow duplicates. We could use a LIST
combined with a SET, but that would take extra code and round trips to Redis. Instead,
we’ll use a ZSET, where the members are the combinations of precisions and names that
have been written to, and the scores are all 0. By setting all scores to 0 in a ZSET, Redis
will try to sort by score, and finding them all equal, will then sort by member name. This gives us a fixed order for a given set of members, which will make it easy to sequentially
scan them. An example ZSET of known counters can be seen in figure 5.2.
Now that we know what our structures for counters look like, what goes on to make that
happen? For each time slice precision, we’ll add a reference to the precision and the
name of the counter to the known ZSET, and we’ll increment the appropriate time window
by the count in the proper HASH. Our code for updating a counter looks like this.
Updating the counter information isn’t so bad; just a ZADD and HINCRBY for each time
slice precision. And fetching the information for a named counter and a specific precision
is also easy. We fetch the whole HASH with HGETALL, convert our time slices and
counters back into numbers (they’re all returned as strings), sort them by time, and
finally return the values. The next listing shows our code for fetching counter data.