download & runRedis Labs Enterprise Cluster

Learn more

free download

Connect & useRedis Cloud





By creating your account, you agree to our Terms of Use

Forgot your password?

Not Registered?

Forgot your password?

Enter your email address below to have your password reset. We'll send you an email with a link to change your password.

Your IP address will be logged and monitored for this request.

Cancel

Your password has reset!

We have sent you an email with a link to change your password.

OK

Blog

Posted on by Tung Nguyen

Redis Labs Performance Testing With Live Traffic

Originally published on bleacherreport.com

Intro

Our API service uses Redis as it’s caching layer. The Redis set operations are extremely useful for the logic required in our API service. Unfortunately, we have had some issues scaling Redis. Redis is a single-threaded server that reads incoming connections using an event-based paradigm, thus uses only 1 CPU core. So you can scale Redis vertically but it is very challenging to scale Redis horizontally. We see this when our app calls heavier zset and zunionstore operations. Redis hits the CPU limit and it gets pegged at 100%.

image

We’ve also seen issues with the way Redis snapshots the data in memory to disk. Redis forks off another process to save the data and that process doubles the RAM footprint of the Redis. Once Redis runs out of memory on the box it gets painfully slow. This effectively reduces the memory you can use on the box to half of the total actually available. In our case we use Redis as a cache, so we’ve simply disabled saves so we could use all the memory on the box and we’ve been veritically scaling. However, vertically scaling will only get you so far. Our current bottleneck is that we’re approaching the upper bound of available CPU. We see response time spikes that correspond to these CPU spikes in New Relic:

image

Redis Labs

We wanted try out Redis clustering, but the open source version is a still work in progress. Redis Labs, one of the top contributers to the open source Redis project, already has clustering working on their platform. Redis Labs is engineered to scale up automatically and seamlessly. The Redis db will grow in capacity on a single shard and, when it needs to, automatically start to cluster across multiple shards. All of this happens behind the scenes transparently. Customers need only manage a simple single endpoint.

Performance Testing

Beyond basic sanity tests and functional testing, we required quantitative measures and performance metrics to legitimize our move.

We used em-proxy to blue-green traffic in migrating our servers to a new cluster. This allows us to proxy a portion of traffic to Redis Labs, testing it, and reacting to changes with contained exposure.

After a few google searches that night, I ran into it, and then I did some testing that same night with a single instance. Then on the following night, I created an additional test cluster, which took about 5 minutes, and then directed ALL of traffic from live production cluster to the test cluster and let it run through the night and following day. The powerful thing about this technique is that we are not guessing with input benchmark data, instead we’re actually using live production traffic. On top of that, we can compare the New Relic data sets against each other in real time.

The results were incredible. Here is are the Newrelic response times side by side:

image

The left graph is based on Redis servers that we maintain in-house. On the right is the test cluster connected to Redis Labs. The only difference between the clusters is that one uses open source Redis server and the other one uses Redis Labs. With the standard open source Redis server we see intermittent spikes in the response times up to 700ms and averaged 435ms; it’s all over the place. Redis Labs brings our response times to a consistent and smooth average of 55ms! Whatever voodoo blackmagic Redis Labs is conjuring up, it’s obviously working.

Finding Issues Before Production

During our testing we found that Redis Labs disables some Redis commands, thus were required to make some changes to our API code.

For instance, Redis Labs disables the object command.

image

The next error is due to the fact that Redis Labs was sharding, yet our app was not equipped to handle it.

image

To resolve this issue you need to name Redis keys such that Redis can manage the data shards. Keys need to be named with curly braces like so hello{dynamic}world. So the following keys map to separate shards:

It is explained in more detailed here: http://redislabs.com/kb/redis-cloud-cluster. We ended up deploying to a single shard, which is apples-to-apples with our prior environment, and will circle back to sharding at a later time.

This method of testing is extremely powerful. We found a lot of issues that we were able to fix before we actually rolled out Redis Labs on production. Had we instead moved all traffic to Redis Labs, we would have been in a fire fighting situation and probably would have rolled back, not realizing the dramatic performance opportunities Redis Labs offers.

Conclusions

Redis Labs is fast! If you’ve read closely, you’ve probably realized the kicker. We’re getting this performance increase from Redis Labs without even sharding. We still have more follow up work to do, including sharding, which should improve our performance even more dramatically. If you are using Redis and are having issues scaling it beyond a single CPU or are getting memory bounded, Redis Labs is a great option.

Remarkably Simple. Epic Performance.

60

Subscribe

Get the latest blog posts by email

8 32

Redis Watch

Get the hottest Redis news with our
periodic newsletter!

27

Interact with Redis

Enter your Redis commands,
see how fast it responds!

Redis Cloud > |
37 59

Gartner Acknowledges Redis Labs as a Leader

In the 2015 Magic Quadrant for Operational Database Management Systems (ODBMS)

Leaders
47

User Satisfaction and
Market Presence -
Redis Ranked as
#1 NoSQL

g2crowd
64

#2 Database in
User Satisfaction

user satisfaction

g2crowd.com

63

Top 10 Data Stores

Redis in containers
1st
MySQL
2nd
Redis
3rd
MongoDB
4th
PostgreSQL

stackshare.io

45
62

Top 10 Technologies
on Docker

Percent of companies running this technology

Redis in containers

datadoghq.com

28

See Our Open Source Contributions

48

NoSQL Databases
in Containers

Redis in containers

DevOps.com & ClusterHQ.com

11

About Us

Redis Labs is the open source home and commercial provider of Redis, a database benchmarked as the world’s fastest. Gartner has named the company as a Leader in it's 2015 ODBMS Magic Quadrant. Redis Labs' software and service solutions power cutting edge applications with blazing fast enterprise-class Redis and are trusted by thousands of customers for high performance, seamless scalability, true high availability and best-in-class expertise. Redis is ranked the #1 NoSQL (and #2 database) in User Satisfaction and Market Presence by G2 Crowd, the top database technology on Docker by Datadog, the most popular NoSQL database in containers by DevOps.com and ClusterHQ, the #1 NoSQL among Top 10 Data Stores by Stackshare and both the fastest growing database since January 2013 and one of the top three NoSQL databases by DB-engines.

66

Why Redis Labs?

Watch the video
61 53 Stance

"How We Perform
in Peak Times at Sub-millisecond Latencies with Redis Labs"

21

Latest News

19 18 44 7 55
60

Subscribe

Get the latest blog posts by email

8 32

Redis Watch

Get the hottest Redis news with our
periodic newsletter!

27

Interact with Redis

Enter your Redis commands,
see how fast it responds!

Redis Cloud > |
37 59

Gartner Acknowledges Redis Labs as a Leader

In the 2015 Magic Quadrant for Operational Database Management Systems (ODBMS)

Leaders