11.4 Sharding LISTs with Lua
Back in sections 9.2 and 9.3, we sharded HASHes, SETs, and even STRINGs as a way of reducing memory. In section 10.3, we sharded ZSETs to allow for search indexes to grow beyond one machine’s memory and to improve performance.
As promised in section 9.2, in this section we’ll create a sharded LIST in order to reduce memory use for long LISTs. We’ll support pushing to both ends of the LIST, and we’ll support blocking and nonblocking pops from both ends of the list.
Before we get started on actually implementing these features, we’ll look at how to structure the data itself.