Watch all RedisConf 2021 sessions on demand
RedisGears is an engine for data processing in Redis. RedisGears supports batch- and event-driven processing for Redis data. To use RedisGears, you write functions that describe how your data should be processed. You then submit this code to your Redis deployment for remote execution.
RedisGears’ built-in coordinator for cluster support lets you execute server-side code agnostic of your Redis setup—be it standalone, open source Redis cluster, or Redis Enterprise.
At its core, RedisGears has three main components:
On top of these three core components, RedisGears also includes a fast, low-level C-API that integrates via Python for programmability.
RedisGears minimizes the execution time and the data flow between shards by running your functions as close as possible to your data.
A collection of RedisGears functions and any dependencies they may have that implement a high-level functional purpose is called a recipe. The first recipe supported by RedisGears is write-behind.
Write-behind is a caching strategy in which the cache layer itself connects to the backing database. This means that your applications need only connect to your cache layer, and the cache then reads from or updates the backing database as needed. Redis Labs currently supports write-behind caching in Redis Enterprise Software.
Here’s how these caching patterns work:
You can find out more about this recipe in the documentation.
The goal of RedisGears is to help you build an operations pipe (OPP) for each key in Redis to pass through. Developers can build this pipe using Python and the script will run a background thread through RedisGears’ execution management. To streamline OPP creation, RedisGears provides a new Python class called GearsBuilder, which starts out with an empty OPP and provides a set of functions that lets you easily add operations to the pipe.
Here are some recipes that demonstrate what you can accomplish with RedisGears. You can execute these recipes via the RG.PYEXECUTE command:
RG.PYEXECUTE python-script [UNBLOCKING]
Find more information in our documentation on redisgears.io. Alternatively, save this script in a separate Python file and upload it to Redis:
Recipe 1: Maintain a set of all keys in Redis
This specific recipe registers each key change in Redis, apart from the key all_keys, and triggers a RedisGears script.
# create the builder
builder = GearsBuilder()
# filter events on key:'all_keys'
builder.filter(lambda x: x['key'] != 'all_keys')
# repartition to the shard hold the ‘all_keys’ key
builder.repartition(lambda x: ‘all_keys’)
# add the keys to 'all_keys' set
builder.map(lambda x: execute('sadd', 'all_keys', x['key']))
# register the execution on key space notification
Recipe 2: Count movies by genre
In this sample Redis database, each movie is represented as a Hash, for which the key has a prefix ‘movie:’. Each Hash also has a field named ‘genres’, which is a comma-separated list of genres.
redis:6379> hgetall movie:tt0208092
4) "Guy Ritchie"
# create the pipe builder. KeysOnlyReader is a performance
# improvement only piping the keys.
builder = GearsBuilder('KeysOnlyReader')
# get from each hash the genres field
builder.map(lambda x: execute('hget', x, 'genres'))
# filter those who do not have genres
builder.filter(lambda x: x is not None)
# split genres by comma
builder.flatmap(lambda x: x.split(','))
# count for each genre the number of times it appears
# start the execution
This demonstrates the power of the MapReduce capabilities in RedisGears. We can query the entire key-space for keys starting with ‘movie:’ and run an aggregation on nested data without having to worry about which shards contain our keys.
Recipe 3: Consume a Redis Stream and update Redis accordingly
# create the builder with a StreamReader
builder = GearsBuilder('StreamReader')
# extract each field value pair from the message and increase
# the pipe granularity
builder.flatmap(lambda x: [(a, a) for a in x[‘value’].items()])
# make sure the gears data lives in the correct shard
builder.repartition(lambda x: x)
# apply each field value pair to a key
builder.foreach(lambda x: execute('set', x, x))
# register on new messages on the stream 'inputStream'
The images below show what happens in a two-shard database when a message is inserted into a stream with the above recipe. For simplicity, we show the key-space below the yellow line, with RedisGears’ working memory above the yellow line. Our stream message contains two field-value pairs, K1 V1 and K2 V2.
Find out more in our docs.