# 5.2.2 Storing statistics in Redis

Truth be told, I’ve personally implemented five different methods of storing statistics in Redis. The method described here takes many of the good ideas from those methods and combines them in a way that allows for the greatest flexibility and opportunity to scale. What are we going to build?

We’ll build a method to store statistics that have a similar scope to our

log_common() function from section 5.1.2 (the current hour and the last hour). We’ll

collect enough information to keep track of the minimum, maximum, average value,

standard deviation, sample count, and the sum of values that we’re recording. We

record so much information because we can just about guarantee that if we aren’t

recording it, we’ll probably need it.

For a given named context and type, we’ll store a group of values in a ZSET. We

won’t use the ZSET for its ability to sort scores, but instead for its ability to be unioned

against another ZSET, keeping only the MIN or MAX of items that intersect. The precise

information that we’ll store for that context and type is the minimum value, the maximum

value, the count of values, the sum of the values, and the sum of the squares of

the values. With that information, we can calculate the average and standard deviation.

Figure 5.3 shows an example of a ZSET holding this information for the ProfilePage

context with statistics on AccessTime.

Now that we know the type of data that we’ll be storing, how do we get the data in

there? We’ll start like we did with our common logs by checking to make sure that our

current data is for the correct hour, moving the old data to an archive if it’s not for the

current hour. We’ll then construct two temporary ZSETs—one with the minimum

value, the other with the maximum value—and ZUNIONSTORE them with the current

stats with an aggregate of MIN and MAX, respectively. That’ll allow us to quickly update

the data without needing to WATCH a potentially heavily updated stats key. After cleaning

up those temporary ZSETs, we’ll then ZINCRBY the count, sum, and sumsq members

of the statsZSET. Our code for performing this operation is shown next.

We can ignore almost all of the first half of the code listing, since it’s a verbatim copy

of the rollover code from our log_common() function from section 5.1.2. The latter

half does exactly what we described: creating temporary ZSETs, ZUNIONSTOREing them

with our destination ZSET with the proper aggregates, cleaning the temporary ZSETs,

and then adding our standard statistics information. But what about pulling the statistics

information back out?

To pull the information back out, we need to pull all of the values from the ZSET

and then calculate the average and standard deviation. The average is simply the sum

member divided by the count member. But the standard deviation is more difficult.

With a bit of work, we can derive the standard deviation from the information we

have, though for the sake of brevity I won’t explain the math behind it. Our code for

fetching stats is shown here.

Aside from the calculation of the standard deviation, the get_stats() function isn’t

surprising. And for those who’ve spent some time on the Wikipedia page for standard

deviation, even calculating the standard deviation shouldn’t be all that surprising. But

with all of this statistical information being stored, how do we know what information

to look at? We’ll be answering that question and more in the next section.