8.5.1 Data to be streamed

As people perform a variety of actions within our social network, those actions are
seen at the various functions that defined our API. In particular, we spent most of our
time building out the ability to follow/unfollow users, and post/delete messages. If
we’d built other pieces of our social network, we’d also find a variety of other events
that occur as the result of user behavior. A streaming API is meant to produce a
sequence of these events over time as a way of keeping clients or other services
updated about a subset of what’s going on across the entire network.

In the process of building a streaming API, a variety of decisions must be made,
which can be generally reduced to three major questions:

• Which events should be exposed?
• What access restrictions (if any) should exist?
• What kinds of filtering options should be provided?

For now, I won’t answer the second question about access restrictions. That’s a question
that we need to answer when we’re building our social network based on expectations
of privacy and system resources. But I’ll answer the other two questions.

Because we focused on posting/deleting messages and following/unfollowing
users, we should offer at least some of those events. To keep things simple for now, we’ll only produce message posting and deletion events. But based on the structures
that we create and pass around, adding functionality to support follow/unfollow
events or events for other actions that we’ve added should be easy.

The types of filtering options that we’ll provide will overlap significantly with the
API features and functionality that Twitter provides on the public side of things. In
particular, we’ll offer the ability to filter over messages with an equivalent of follow
(users), track (keywords), and location filters, in addition to a randomly selected subset
of messages, similar to Twitter’s firehose and sample streams.

Now that we know what data we’ll have access to, let’s start looking at how we’ll
serve the data.