5.2.2 Storing statistics in Redis
Truth be told, I’ve personally implemented five different methods of storing statistics in Redis. The method described here takes many of the good ideas from those methods and combines them in a way that allows for the greatest flexibility and opportunity to scale. What are we going to build?
We’ll build a method to store statistics that have a similar scope to our
log_common() function from section 5.1.2 (the current hour and the last hour). We’ll
collect enough information to keep track of the minimum, maximum, average value,
standard deviation, sample count, and the sum of values that we’re recording. We
record so much information because we can just about guarantee that if we aren’t
recording it, we’ll probably need it.
For a given named context and type, we’ll store a group of values in a ZSET. We
won’t use the ZSET for its ability to sort scores, but instead for its ability to be unioned
against another ZSET, keeping only the MIN or MAX of items that intersect. The precise
information that we’ll store for that context and type is the minimum value, the maximum
value, the count of values, the sum of the values, and the sum of the squares of
the values. With that information, we can calculate the average and standard deviation.
Figure 5.3 shows an example of a ZSET holding this information for the ProfilePage
context with statistics on AccessTime.
Now that we know the type of data that we’ll be storing, how do we get the data in
there? We’ll start like we did with our common logs by checking to make sure that our
current data is for the correct hour, moving the old data to an archive if it’s not for the
current hour. We’ll then construct two temporary ZSETs—one with the minimum
value, the other with the maximum value—and ZUNIONSTORE them with the current
stats with an aggregate of MIN and MAX, respectively. That’ll allow us to quickly update
the data without needing to WATCH a potentially heavily updated stats key. After cleaning
up those temporary ZSETs, we’ll then ZINCRBY the count, sum, and sumsq members
of the statsZSET. Our code for performing this operation is shown next.
We can ignore almost all of the first half of the code listing, since it’s a verbatim copy
of the rollover code from our log_common() function from section 5.1.2. The latter
half does exactly what we described: creating temporary ZSETs, ZUNIONSTOREing them
with our destination ZSET with the proper aggregates, cleaning the temporary ZSETs,
and then adding our standard statistics information. But what about pulling the statistics
information back out?
To pull the information back out, we need to pull all of the values from the ZSET
and then calculate the average and standard deviation. The average is simply the sum
member divided by the count member. But the standard deviation is more difficult.
With a bit of work, we can derive the standard deviation from the information we
have, though for the sake of brevity I won’t explain the math behind it. Our code for
fetching stats is shown here.
Aside from the calculation of the standard deviation, the get_stats() function isn’t
surprising. And for those who’ve spent some time on the Wikipedia page for standard
deviation, even calculating the standard deviation shouldn’t be all that surprising. But
with all of this statistical information being stored, how do we know what information
to look at? We’ll be answering that question and more in the next section.