Our newly launched Redis Enterprise 5.0 introduced support for the Open Source (OSS) cluster API, which allows a Redis Enterprise cluster to scale infinitely and in a linear manner by just adding shards and nodes. The OSS cluster API relies on the intelligence of the clients to decide to which shard/node to send the request to based on the key part of the key/value item and a hashing function shared across the clients and the cluster. This post explains how Redis Enterprise works with the OSS cluster API and validates the infinite linear performance scalability.
For those of you who are not familiar with the Redis Enterprise architecture, let’s start with some short background:
A cluster, in Redis Enterprise terms, is a set of cloud instances, virtual machine/container nodes or bare-metal servers that allows you to create any number of Redis databases (A database in Redis Enterprise terminology is the entity that manages your entire dataset across multiple Redis shards/instances. Don’t confuse this with the databases inside every Redis instance that you can leverage to do some segmentation in your keyspace using the Redis SELECT command.) in the memory pool that is shared across the set. The cluster has a symmetric shared-nothing architecture, complete separation between the data-path and control & management-path, and it includes the following main components:
A database in the Redis Enterprise cluster can be created in any one of the configurations below:
Based on its strong multi-tenant technology, a Redis Enterprise cluster can manage multiple databases of different types on the same cluster resources in a completely isolated manner.
In order to utilize the new OSS cluster API, you should use the following properties when creating a database with the Redis Enterprise API:
This will create a clustered database with properties similar to those shown below:
As you can see:
With that in mind, we decided to build a test environment on AWS to validate whether Redis Enterprise can really scale infinitely and in a linear manner. Since we planned to have some serious load, we decided to use EC2 m4.16xlarge instances (64 cores, 256GB RAM) for the cluster nodes and c4.8xlarge instances (36 cores, 60GB RAM) for running memtier_benchmark, an open source multi-threaded load generation tool.
Using multiple instances of memtier_benchmark is a must, since in many cases a single Redis Enterprise node can deal with more traffic than the volume a single memtier_benchmark instance can generate. This approach also allows us to avoid the network bandwidth and packet per second limitations of a single NIC and makes it easy to increase the traffic load in a step by step (instance by instance) manner.
This is what our final setup looks like:
We created a 192-shard clustered Redis database using the Redis Enterprise API with the following parameters:
{
"name": "api-benchmark",
"replication": false,
"port" : 12345,
"sharding": true,
"shards_count" : 192,
"version": "4.0",
"memory_size": 100000000000,
"type": "redis",
"oss_cluster":true,
"proxy_policy": "all-master-shards",>
"shards_placement": "sparse",
"shard_key_regex": [{"regex": ".*{(?.*)}.*"}, {"regex": "(?.*)"}]
}
> curl -k -X POST -u ":" -H "Content-Type: application/json" -d @create_db.json https://localhost:9443/v1/bdbs
We tuned the proxy on each node to cope with the expected load by setting the number of proxy threads to 24:
> rladmin tune proxy all max_threads 24
> rladmin tune proxy all threads 24
We used the new version of memtier_benchmark that supports the OSS cluster API to first populate the database with around 10 million items, and then run the tests.
Here are the memtier_benchmark parameters we used during our population and benchmarking stages:
Here is a memtier_benchmark command line example:
> memtier_benchmark -s $DB_SERVER -p $DB_PORT
--pipeline=$PIPELINE_SIZE -c $NUM_CLIENTS -t $NUM_THREADS
-d $DATA_SIZE --key-minimum=$KEY_MIN
--key-maximum=$MAX_KEYS_PER_SLOT --key-pattern=R:R
--ratio=$WR_RATIO --run-count=1 --requests=$NUM_REQ
--out-file=$OUT_FILE --oss-cluster
Our final setup ran a 192-shard database over only 6 nodes on the Redis Enterprise cluster and demonstrated outstanding results: over 10 million ops/sec at a slightly higher than 1msec latency! Here is a screenshot taken from the Redis Enterprise UI:
We conducted this experiment in order to validate that the shared-nothing architecture of Redis Enterprise can scale linearly thanks to the new OSS cluster API that was introduced in Redis Enterprise 5.0. Our experiment included:
We found linear performance scalability when scaling from a 1-node cluster to a 6-node cluster, as shown in the following graph:
A deeper analysis of these results indicates that the throughput per node did not change by more than 10% when scaling from a single node cluster to a two-node cluster and then to a 6-node cluster. We believe that these changes in performance between the tests might be related to different resource conditions (network, VM, etc.) on each test iteration.
With the introduction of the OSS cluster API, Redis Enterprise 5.0 can easily scale linearly by just adding shards to the database and nodes to the cluster, as proven by this benchmark. We plan to continue breaking performance records in the database space, but we wanted to share this for re:invent 2017, so we stopped here. Please stay tuned!
