Skip to content

Message Queues and Pub/Sub for Cloud Builders

6-minute read. Assumes you've read Serverless explained.

This is the short cloud-builder version. For queues vs streams, delivery guarantees and ordering in depth, read Queues vs streams on Data Zero to Hero.

The one-line answer

A queue holds messages until one worker takes each one and processes it. Pub/sub (publish/subscribe) delivers a copy of every message to every subscriber. Both let one part of a system hand off work to another without waiting for it and without knowing who is on the other end.

Why you want one

Without a queue, a web request that has to resize an image, send an email and update a search index does all three before it answers. If the email provider is slow, the user waits. If it is down, the request fails.

With a queue, the request writes "order 1042 placed" to a queue and returns in milliseconds. Workers process it in the background. That buys you:

  • Decoupling. The web tier and the workers deploy, scale and fail independently.
  • Load levelling. A burst of 10,000 orders sits in the queue while a steady pool of workers drains it. Nothing falls over.
  • Retries for free. If a worker crashes mid-message, the message becomes visible again and another worker retries it.
  • A natural scaling signal. Queue depth tells you exactly how far behind you are (see Autoscaling explained).

Queue vs pub/sub

flowchart LR
  subgraph Queue[Queue: each message to one worker]
    P1[Producer] --> Q[(Queue)]
    Q --> W1[Worker 1]
    Q --> W2[Worker 2]
  end
  subgraph PubSub[Pub/sub: every message to every subscriber]
    P2[Publisher] --> T((Topic))
    T --> S1[Email service]
    T --> S2[Analytics]
    T --> S3[Search indexer]
  end

On the left, two workers share the queue and each message is handled once by whichever worker takes it. On the right, one event fans out so three different services each get their own copy.

The two combine. The fan-out pattern puts a queue behind each pub/sub subscriber: the topic copies the event to three queues, and each service drains its own queue at its own pace. If analytics is down for an hour, its queue just fills up and nothing is lost.

The managed services

Job AWS Azure GCP
Simple work queue SQS Storage Queues, Service Bus queues Pub/Sub (pull subscription), Cloud Tasks
Pub/sub fan-out SNS Service Bus topics, Event Grid Pub/Sub
Event routing by content EventBridge Event Grid Eventarc
Ordered, replayable stream Kinesis, MSK Event Hubs Pub/Sub with ordering keys, Managed Kafka

Four things every builder must know

  1. At-least-once delivery is the normal case. SQS standard queues, SNS and Pub/Sub can deliver the same message more than once. Your consumer must be idempotent: processing message 1042 twice must not charge the customer twice. See Idempotency explained.
  2. The visibility timeout. When a worker takes a message from SQS, it is hidden, not deleted. The worker deletes it after success. If the timeout is shorter than your processing time, a second worker grabs the same message while the first is still working.
  3. Dead-letter queues (DLQs). A message that fails every time ("poison message") will retry forever and block or waste workers. After a set number of attempts, move it to a DLQ, and alarm on the DLQ being non-empty.
  4. Ordering is not free. Standard queues are best-effort ordered. If order matters, use FIFO queues (SQS FIFO), sessions (Service Bus) or ordering keys (Pub/Sub), and accept lower throughput per group.

Common mistakes

  • Non-idempotent consumers, then surprise duplicates in production.
  • No DLQ, or a DLQ nobody watches.
  • Visibility timeout shorter than the work, causing duplicate processing.
  • Using a queue when the caller needs the answer now. Queues are for work that can finish later.
  • Scaling workers on CPU instead of queue depth or age of the oldest message.

How exams test it

  • "Decouple the web tier from order processing" or "absorb traffic spikes" -> a queue (SQS, Service Bus, Pub/Sub).
  • "One event must trigger several independent services" -> pub/sub fan-out (SNS to multiple SQS queues, Service Bus topics, Pub/Sub subscriptions).
  • "Exactly-once processing, strict order" -> SQS FIFO or Service Bus sessions.
  • "Messages that repeatedly fail" -> dead-letter queue.

Go deeper

What to look at next