This book covers the use of Redis, an in-memory database/data structure server.

open all | close all

8.5 Streaming API

As development of our social network continues, at some point we’ll want to learn
more about what’s going on—maybe to discover how many posts are made every hour,
the most-talked-about topics, or even who’s being mentioned all the time. One way of
doing this is to make calls to gather this information. Another way is to record this
information inside the functions that perform all of the operations. The third way,
which we’ll explore in this section, is to build our functions to broadcast simple
events, which are received and processed by event listeners to analyze the data.

In this section, I’ll describe how to build the back end for a streaming API that
functions similar to the streaming API offered by Twitter.

Unlike the other parts of the system that we’ve already built, the streaming API is a
different group of functionalities altogether. The functions that we built to support
the typical operations of a site like Twitter in the last several sections were meant to
execute and complete quickly. On the other hand, a streaming API request is meant to
return data over a longer period of time.

Most modern social networks offer the ability to gather information from their system
via some sort of API. One advantage that Twitter has shown over the last several
years is that by offering real-time events to third parties, those third parties can
develop unique and interesting analyses of the data that Twitter itself may not have
had the time or interest to develop.

The first step in building a streaming API is understanding what kind of data we’ll
be processing and producing.