RedisGears

RedisGears is a Redis module that adds a serverless engine for multi-model and cluster operations in Redis, supporting both event-driven and batch operations.

RedisGears’ built-in coordinator for cluster support allows you to execute server-side code agnostic of your Redis setup—be it standalone, open source Redis cluster, or Redis Enterprise.

Architecture

 

At its core, RedisGears has three main components:

  1. Cluster management: keeps an internal mapping of slots and the shards responsible for those slots. It creates an abstraction layer for all other components that need to access the data.
  2. Execution management: registers the correct events on keys to trigger and schedule the execution of your scripts. It can execute scripts for events on keys and streams, as well as time-based events.
  3. Map/Reduce: schedules and gathers the parts of your script that need to run on different shards.

On top of these three core components, RedisGears also includes a fast low-level C-API for programmability. You can integrate to this C-API via Python today, with more languages to be added in the future.

Gears recipes

The end goal of RedisGears is to help you build an operations pipe (OPP) for each key in Redis to pass through. Developers can build this pipe using Python and the script will run a background thread through RedisGears’ execution management. To streamline OPP creation, RedisGears provides a new Python class called GearsBuilder, which starts out with an empty OPP and provides a set of functions that lets you easily add operations to the pipe.

Here are some recipes that demonstrate what you can accomplish with RedisGears. You can execute these recipes via the PYEXECUTE command:

RG.PYEXECUTE python-script [UNBLOCKING]

 

Find more information in our documentation on redisgears.io. Alternatively, save this script in a separate Python file and upload it to Redis (example here):

Recipe 1: Maintain a set of all keys in Redis

This specific recipe registers each key change in Redis, apart from the key all_keys, and triggers a RedisGears script.

# create the builder
builder = GearsBuilder()
# filter events on key:'all_keys'
builder.filter(lambda x: x['key'] != 'all_keys')
# add the keys to 'all_keys' set
builder.map(lambda x: execute('sadd', 'all_keys', x['key']))
# register the execution on key space notification
builder.register()

Recipe 2: Count movies by genre

In this sample Redis database, each movie is represented as a Hash, for which the key has a prefix ‘movie:’. Each Hash also has a field named ‘genres’, which is a comma-separated list of genres.

redis:6379> hgetall movie:tt0208092
1) "genres"
2) "Comedy,Crime"
3) "director"
4) "Guy Ritchie"
5) "startYear"
6) "2000"
7) "runtimeMinutes"
8) "104"
9) "originalTitle"
10) "Snatch"
redis:6379>

 

If we’d like to know how often each genre appears across all the movies in our database, we’d run this RedisGears script:

# create the pipe builder. KeysOnlyReader is a performance
# improvement only piping the keys.
builder = GearsBuilder('KeysOnlyReader')
# get from each hash the genres field
builder.map(lambda x: execute('hget', x, 'genres'))
# filter those who do not have genres
builder.filter(lambda x: x is not None)
# split genres by comma
builder.flatmap(lambda x: x.split(','))
# count for each genre the number of times it appears
builder.countby()
# start the execution
builder.run('movie:*')

 

This demonstrates the power of the Map/Reduce capabilities in RedisGears. We can query the entire key-space for keys starting with ‘movie:’ and run an aggregation on nested data without having to worry about which shards contain our keys.

Recipe 3: Consume a Redis Stream and update Redis accordingly

In this recipe, we use a StreamReader that consumes each message in a stream exactly once. These messages contain multiple field value pairs, which are each added to Redis. While similar to our first recipe, this one shows the strength of combining RedisGears with Redis Streams to easily consume an event-sourcing stream and apply it to multiple database models.

# create the builder with a StreamReader
builder = GearsBuilder('StreamReader')
# extract each field value pair from the message and increase
# the pipe granularity
builder.flatmap(lambda x: [(a[0], a[1]) for a in x.items()])
# filter out the streamId itself
builder.filter(lambda x: x[0] != 'streamId')
# make sure the gears data lives in the correct shard
builder.repartition(lambda x: x[0])
# apply each field value pair to a key
builder.foreach(lambda x: execute('set', x[0], x[1]))
# register on new messages on the stream 'inputStream'
builder.register('inputStream')

 

The images below explain what happens in a two-shard database when a message is inserted into a stream with the above recipe. For simplicity, we show the key-space below the yellow line, with RedisGears’ working memory above the yellow line. Our stream message contains two field-value pairs, K1 V1 and K2 V2.

Start: empty stream

New message received in Stream, passed to RedisGears.

RedisGears has a single item in its OPP, containing the two key-value pairs.

FlatMap

Repartition

Set the keys in the Redis key-space in the correct shards.

Documentation

Find out more at Redisgears.io.