Documentation - Redise Pack

A guide to Redise Pack installation, operation and administration

open all | close all

3.6 Publish/subscribe

If you’re confused because you can’t remember reading about publish or subscribe yet, don’t be — this is the first time we’ve talked about it. Generally, the concept of publish/ subscribe, also known as pub/sub, is characterized by listeners subscribing to channels, with publishers sending binary string messages to channels. Anyone listening to a given channel will receive all messages sent to that channel while they’re connected and listening. You can think of it like a radio station, where subscribers can listen to multiple radio stations at the same time, and publishers can send messages on any radio station. In this section, we’ll discuss and use operations involving publish and subscribe.

When finished with this section, you’ll know how to use these commands, and why we use other similar solutions in later chapters.

In Redis, the pub/sub concept has been included through the use of a collection of the five commands shown in table 3.11.

Table 3.11 Commands for handling pub/sub in Redis
Command Example use and description
SUBSCRIBE SUBSCRIBE channel [channel …] — Subscribes to the given channels
UNSUBSCRIBE UNSUBSCRIBE [channel [channel …]] — Unsubscribes from the provided channels, or unsubscribes all channels if no channel is given
PUBLISH PUBLISH channel message — Publishes a message to the given channel
PSUBSCRIBE PSUBSCRIBE pattern [pattern …] — Subscribes to messages broadcast to channels that match the given pattern
PUNSUBSCRIBE PUNSUBSCRIBE [pattern [pattern …]] — Unsubscribes from the provided patterns, or unsubscribes from all subscribed patterns if none are given

With the way the PUBLISH and SUBSCRIBE commands are implemented on the Python side of things, it’s easier to demonstrate the feature if we use a helper thread to handle the PUBLISHing. You can see an example of PUBLISH/SUBSCRIBE in the next listing.3

Listing 3.11 Using PUBLISH and SUBSCRIBE in Redis
>>> def publisher(n):
…	time.sleep(1)

We sleep initially in the function to let the SUBSCRIBEr connect and start listening for messages.

…	for i in xrange(n):
…				conn.publish('channel', i)
…				time.sleep(1)

After publishing, we’ll pause for a moment so that we can see this happen over time.

>>> def run_pubsub():
…	threading.Thread(target=publisher, args=(3,)).start()

Let’s start the publisher thread to send three messages.

…	pubsub = conn.pubsub()
…	pubsub.subscribe(['channel'])

We’ll set up the pubsub object and subscribe to a channel.

…	count = 0
…	for item in pubsub.listen():

We can listen to subscription messages by iterating over the result of pubsub.listen().

…		print item

We’ll print every message that we receive.

…		count += 1
…		if count == 4:
…			pubsub.unsubscribe()

We’ll stop listening for new messages after the subscribe message and three real messages by unsubscribing.

…		if count == 5:
…			break

When we receive the unsubscribe message, we need to stop receiving messages.

>>> run_pubsub()

Actually run the functions to see them work.

{'pattern': None, 'type': 'subscribe', 'channel': 'channel', 'data': 1L}

When subscribing, we receive a message on the listen channel.

{'pattern': None, 'type': 'message', 'channel': 'channel', 'data': '0'}
{'pattern': None, 'type': 'message', 'channel': 'channel', 'data': '1'}
{'pattern': None, 'type': 'message', 'channel': 'channel', 'data': '2'}

These are the structures that are produced as items when we iterate over pubsub.listen().

{'pattern': None, 'type': 'unsubscribe', 'channel': 'channel', 'data': 0L}

When we unsubscribe, we receive a message telling us which channels we have unsubscribed from and the number of channels we are still subscribed to.

The publish/subscribe pattern and its Redis implementation can be useful. If you skip ahead and scan around other chapters, you’ll notice that we only use publish/ subscribe in one other section, section 8.5. If PUBLISH and SUBSCRIBE are so useful, why don’t we use them very much? There are two reasons.

One reason is because of Redis system reliability. In older versions of Redis, a client that had subscribed to channels but didn’t read sent messages fast enough could cause Redis itself to keep a large outgoing buffer. If this outgoing buffer grew too large, it could cause Redis to slow down drastically or crash, could cause the operating system to kill Redis, and could even cause the operating system itself to become unusable. Modern versions of Redis don’t have this issue, and will disconnect subscribed clients that are unable to keep up with the client-output-buffer-limit pubsub configuration option (which we’ll talk about in chapter 8).

The second reason is for data transmission reliability. Within any sort of networked system, you must operate under the assumption that your connection could fail at some point. Typically, this is handled by one side or the other reconnecting as a result of a connection error. Our Python Redis client will normally handle connection issues well by automatically reconnecting on failure, automatically handling connection pooling (we’ll talk about this more in chapter 4), and more. But in the case of clients that have subscribed, if the client is disconnected and a message is sent before it can reconnect, the client will never see the message. When you’re relying on receiving messages over a channel, the semantics of PUBLISH/SUBSCRIBE in Redis may let you down.

It’s for these two reasons that we write two different methods to handle reliable message delivery in chapter 6, which works in the face of network disconnections, and which won’t cause Redis memory to grow (even in older versions of Redis) unless you want it to.

If you like the simplicity of using PUBLISH/SUBSCRIBE, and you’re okay with the chance that you may lose a little data, then feel free to use pub/sub instead of our methods, as we also do in section 8.5; just remember to configure client-output-buffer-limit pubsub reasonably before starting.

At this point, we’ve covered the majority of commands that you’ll use on a regular basis that are related to individual data types. There are a few more commands that you’ll also likely use, which don’t fit into our nice five structures-plus-pub/sub theme.

3If you’d like to run this code yourself, you can: I included the publisher() and run_pubsub() functions in the source code for this chapter.