Redis: Swiss army knife at HackerRank

Avatar by Blue Coast
Kamal Joshi, Senior Software Engineer, HackerRank

Redis is mostly used for caching but it has far more interesting use cases outside of caching which we make use of at HackerRank. Redis has powered a lot of critical components like Leaderboard, Rating, Feed etc. which make extensive use of data structures like sorted sets, sets, lists and make use of Lua scripts to make atomic updates to multiple values and embed strategies in them. Additionally, in case of feed Redis made fan-out of content easier as it allowed us to push updates to groups of users very fast. The key intersection operations were critical in building social oriented features like friends leaderboard which can be real-time. Redis has powered our first ML initiative which was in form of content recommendation. The collaborative filtering powered recommendation engine uses Redis as the data store and allowed us to test out multiple iterations of the system without building a complete data pipeline. Additionally, data structures like Hyperloglog provided a base for doing count estimations which allows for dynamic updates to the recommendations based on changing user pattern. Redis also powers the job queues at HackerRank, being the backend store for Resque, Sidekiq and Celery. These use Redis lists and sorted sets to provide queues and schedules. We have started looking at Redis streams to have queues where we can have at least once guarantee through delayed acknowledgment. Other use cases of Redis being Pub/Sub, rate limiting, locks, ReJSON, ReBloom which are kind of similar to how others would also be using them. Problems we ran into: Huge lists/sorted sets, multi key operations blocking switching to cluster and quirks of running Redis on VMs with network mounted disks.