Messaging & Event Streaming

Message-Oriented Middleware: Queues, Topics and Why Systems Talk Asynchronously

Beginner11 min readReviewed by on Updated

Message-oriented middleware lets applications exchange data through a broker instead of calling each other directly. After this guide you will be able to explain the point-to-point and publish-subscribe models, name the parts of a message, compare the main protocols and brokers, and judge when asynchronous messaging is the right integration style for a problem.

What you will learn

  • Compare the four integration styles and explain what messaging adds over file transfer, shared databases and remote calls
  • Describe the anatomy of a message and the roles of producer, broker, destination and consumer
  • Distinguish point-to-point queues from publish-subscribe topics and pick the right one for a use case
  • Explain the three forms of decoupling a broker provides and their costs
  • Position JMS, AMQP, STOMP, MQTT and the Kafka protocol relative to each other
  • Choose between ActiveMQ, RabbitMQ and Kafka for a given workload

Two applications can share data by exchanging files, by reading the same database, by calling each other over the network, or by sending each other messages through an intermediary. The last option — message-oriented middleware (MOM) — is the one that lets each side keep running when the other is slow, busy or down. That property is what makes messaging the backbone of order processing, payment pipelines, log collection and most event-driven architectures. This guide explains the model that all brokers share, so that the JMS, ActiveMQ, RabbitMQ and Kafka guides that follow make sense as variations on one theme rather than four unrelated products.

Four integration styles, one trade-off

Every integration problem comes down to moving data and triggering behaviour between systems that were not written together. The classic classification lists four styles, and each fixes a weakness of the previous one at the cost of a new one.

StyleHow it worksStrengthWeakness
File transferOne system writes a file, another reads it laterSimple, no shared runtime, any languageLatency of hours, format drift, no feedback
Shared databaseSeveral applications read and write the same schemaData is always consistent and currentSchema becomes a contract nobody can change; contention
Remote procedure call (HTTP, gRPC, RMI)One application calls a function exposed by anotherImmediate answer, familiar programming modelCaller blocks; both sides must be up; cascading failure
MessagingApplications hand small packets to a broker that delivers themAsynchronous, reliable, decoupledEventual consistency, harder debugging, more moving parts

Messaging is not "better" than remote calls. It is a different contract: the sender learns only that the broker accepted the message, not that anyone processed it. Systems built on messaging therefore need to be comfortable with work happening later and with results arriving as further messages rather than return values.

Anatomy of a message

Whatever the broker, a message has the same three parts.

  • Headers are fields the broker itself understands and may act on: a unique identifier, a timestamp, a destination, a priority, an expiry, a delivery mode (should the broker write this to disk before acknowledging it?) and, for request-reply, a reply-to address and a correlation identifier.
  • Properties (RabbitMQ and Kafka call them headers too) are application-defined key-value pairs. Consumers can filter on them, routers can branch on them, and tracing systems propagate trace identifiers in them. They are the place for metadata such as tenant, eventType or schemaVersion.
  • The body (payload) is an opaque sequence of bytes as far as the broker is concerned. JSON is the most common encoding today; binary formats such as Avro or Protocol Buffers are preferred at high volume because they are smaller and carry a schema.

A useful discipline is to keep everything a router or a filter needs in headers or properties, and never require anyone to parse the body to decide where a message goes. Brokers cannot look inside the body, and code that does becomes coupled to the payload format.

{
  "headers": {
    "messageId": "ID:app-1-45231-1725700000000-1:1:1:1",
    "timestamp": 1725700000000,
    "deliveryMode": "PERSISTENT",
    "replyTo": null,
    "correlationId": null
  },
  "properties": {
    "eventType": "OrderPlaced",
    "schemaVersion": "2",
    "tenant": "eu-west"
  },
  "body": "{\"orderId\":\"o-9911\",\"total\":149.90,\"currency\":\"EUR\"}"
}

The identifiers in the header are what allow a consumer to detect duplicates and what tie a reply back to its request; you can generate your own with a UUID generator when the broker does not assign one.

Producers, consumers, destinations and the broker

The vocabulary is small. A producer (publisher, sender) creates a message and hands it to the broker. The broker stores it in a destination and delivers it to one or more consumers (subscribers, receivers). The producer never learns who the consumers are, and the consumers never learn who produced the message: both only know the destination's name.

The broker is the component that makes this reliable. It accepts messages even when no consumer is connected, persists them if asked, redelivers them when a consumer fails, and enforces security on who may send to or read from a destination. It is also a single point of contact for monitoring: queue depth, consumer count and throughput are all visible in one place.

Not every messaging system needs a broker. Brokerless libraries such as ZeroMQ deliver directly between processes and win on latency, but then there is nowhere for a message to wait while the receiver is down. As soon as you need a queue that several applications share and that must survive restarts, you need a broker.

Two messaging domains: queues and topics

Destinations come in two flavours, and choosing the right one is the first design decision in any messaging system.

Point-to-point: the queue

A queue delivers each message to exactly one consumer. If ten consumers read from the same queue, the broker spreads the messages across them — a pattern called competing consumers — and this is how you scale throughput horizontally: add a consumer, and the queue drains faster. Messages wait in the queue until someone reads them, so producer and consumer do not have to be running at the same time. Order processing, image resizing jobs and email dispatch are typical queue workloads: each unit of work must be done once.

Publish-subscribe: the topic

A topic delivers each message to every current subscriber. A single "customer address changed" event can fan out to the billing system, the shipping system and the analytics pipeline without the producer knowing any of them exist. By default a subscription only receives messages published while it is active; a durable subscription asks the broker to keep messages for a named subscriber that is temporarily offline. Without durability, a topic is a broadcast; with it, each subscriber effectively gets a private queue.

QueueTopic
Consumers per messageOneZero or many
Consumer offlineMessage waitsMessage lost, unless the subscription is durable
Scaling consumersAdds throughputAdds copies
Typical useWork distribution, commandsEvent notification, broadcast

Modern brokers blur the line: RabbitMQ builds both from exchanges and bindings, and Kafka provides both through partitions and consumer groups. The two intents — do this once versus tell everyone — remain the right way to think about a design regardless of the product.

What decoupling actually buys you

"Loose coupling" is the standard justification for messaging, but it hides three separate benefits, each with a price.

Decoupling in time. The consumer does not need to be running when the producer sends. A nightly batch system can accept events all day; a payment provider that is down for maintenance receives its backlog when it returns. The price is that the producer cannot promise the user that anything has happened yet.

Decoupling in space. The producer addresses a destination name, not a host. You can move a consumer to another machine, run five copies of it, or replace it with a rewritten service, and the producer's configuration does not change. The price is that the broker becomes critical infrastructure and needs its own high-availability story.

Decoupling of load. A queue absorbs bursts. A web front end that receives ten thousand orders in a minute can enqueue them in seconds and let the fulfilment service work through them at its own pace, instead of timing out under a spike. The price is that the queue can grow without bound if consumers fall behind, so every system needs a plan for backlog: more consumers, message expiry, or backpressure on the producer.

Delivery guarantees in one paragraph

A broker can offer at-most-once delivery (fire and forget: fast, may lose messages), at-least-once delivery (the broker keeps the message until the consumer acknowledges it, so a crash before acknowledgement produces a redelivery and possibly a duplicate), or a form of exactly-once limited to a specific scope (Kafka transactions between Kafka topics, for example). Persistent delivery — writing the message to disk before confirming it to the producer — makes at-least-once survive a broker restart. Nearly every production system runs at-least-once and makes consumers idempotent, meaning that processing the same message twice has the same effect as processing it once. The patterns guide later in this subject covers how.

Protocols and APIs

The word "messaging" covers a standard API, several wire protocols and one product-specific protocol; confusing them causes real integration mistakes.

  • JMS (Jakarta Messaging) is a Java API, not a protocol. Code written against jakarta.jms works with any broker that ships a JMS client, but a JMS client talks to its broker in whatever wire format the vendor chose. Two brokers are not interoperable just because both "support JMS".
  • AMQP 0-9-1 is the wire protocol RabbitMQ was built around; it defines exchanges, queues, bindings and channels, and any language with an AMQP 0-9-1 client can talk to RabbitMQ.
  • AMQP 1.0 is an OASIS and ISO standard that shares little beyond the name with 0-9-1. It defines how peers transfer messages, not a broker topology. ActiveMQ Classic, ActiveMQ Artemis, RabbitMQ (natively since 4.0) and Azure Service Bus all speak it.
  • STOMP is a minimal text protocol you can drive from a Telnet session. It is popular for browser clients over WebSocket and for scripting languages.
  • MQTT is a compact publish-subscribe protocol designed for constrained devices and unreliable networks; ActiveMQ and RabbitMQ both accept MQTT connections.
  • The Kafka protocol is Kafka's own binary protocol. There is no standard behind it, but clients exist for every mainstream language.
  • OpenWire is ActiveMQ Classic's native binary protocol, used by its Java, .NET and C++ clients.

For a Java team the practical consequence is: use JMS when you want to keep the option of swapping brokers, use the native client (RabbitMQ's amqp-client, Kafka's kafka-clients) when you want every feature of the broker you chose.

Choosing a broker

The three open-source brokers this subject covers solve overlapping but different problems.

ActiveMQ Classic (and Artemis)RabbitMQApache Kafka
ModelJMS queues and topicsExchanges route to queuesPartitioned, replicated log
Message lifetimeDeleted on acknowledgementDeleted on acknowledgementRetained for a configured time regardless of consumption
OrderingPer queue (per message group with JMSXGroupID)Per queue with one consumerPer partition
Replay old messagesNoNo (streams excepted)Yes, by resetting offsets
Routing flexibilitySelectors, virtual topics, composite destinationsVery high (exchange types, bindings, headers)Low: a producer picks a topic and a key
Throughput sweet spotThousands to tens of thousands of messages/sSimilar, with rich routingHundreds of thousands to millions of records/s
ProtocolsOpenWire, AMQP 1.0, STOMP, MQTT, WebSocketAMQP 0-9-1, AMQP 1.0, STOMP, MQTT, streamsKafka protocol
Natural fitJava EE and Spring applications that need JMSTask queues, RPC, flexible routing, polyglot teamsEvent streaming, log pipelines, data integration, replay

A few rules of thumb hold up well. If your consumers are JMS message-driven beans or Spring @JmsListener methods and you need transactional queues, ActiveMQ is the least friction; prefer Artemis for new deployments that need higher throughput or clustering, and Classic where you have existing configuration and tooling. If you need to route one message to different queues depending on several attributes, or you have producers in Python and consumers in Go, RabbitMQ's exchange model is hard to beat. If the data is a stream of events that several teams will consume at different times, that must be replayable, or that arrives at very high volume, Kafka's retained log is the right primitive, and forcing it to behave like a task queue will disappoint you.

Where messaging goes wrong

  • Using a queue as a database. Queues are for messages in transit. Storing state in a queue and browsing it later leads to unbounded growth and unpredictable ordering. Persist state in a store; put events in the queue.
  • Assuming ordering across consumers. A queue with three competing consumers delivers in order but processes in parallel, so message 2 can finish before message 1. If order matters, use one consumer per key (message groups, RabbitMQ single-active-consumer, Kafka partitions).
  • Very large messages. A 50 MB PDF in a message body slows every hop and blows through broker memory limits. Store the file in object storage and send a reference — the claim-check pattern.
  • Ignoring the backlog. A queue nobody monitors fills up silently until the broker hits its disk limit and stops accepting messages from every producer. Alert on queue depth and consumer count from day one.
  • Coupling through the payload. If every consumer parses the same undocumented JSON, the "decoupled" system cannot change a field without a coordinated deployment. Version the payload and put a schemaVersion property on every message.

Key takeaways

  • Messaging is one of four integration styles; it trades immediate answers for resilience, elasticity and independence between systems.
  • A message is headers the broker understands, properties the application defines, and an opaque body.
  • Queues deliver each message once and scale with competing consumers; topics broadcast to every subscriber, durably if asked.
  • Decoupling in time, space and load is what a broker buys you; unbounded backlogs and a critical piece of infrastructure are what it costs.
  • JMS is a Java API, AMQP, STOMP and MQTT are wire protocols, and Kafka speaks its own; do not assume interoperability from a shared acronym.
  • ActiveMQ fits JMS-centric Java systems, RabbitMQ fits routing-heavy polyglot systems, and Kafka fits high-volume, replayable event streams.

Sources and further reading

Frequently asked questions

Is a message broker the same thing as an enterprise service bus?
No. A broker stores and forwards messages between destinations; it does not understand their content. An enterprise service bus sits on top of a broker or a transport and adds content-based routing, transformation, protocol bridging and orchestration. Many ESB products embed a broker such as ActiveMQ, but you can run a broker on its own and most modern systems do.
When is a plain HTTP call better than a message?
When the caller genuinely needs the answer before it can continue — validating a card before confirming an order, for example — and the callee is expected to be available. Messaging shines when the work can be deferred, when producers and consumers scale at different rates, when the same event interests several consumers, or when the target system is sometimes unavailable.
Does a queue guarantee that a message is processed exactly once?
Not on its own. A broker can promise at-most-once or at-least-once delivery; the consumer can still crash after processing and before acknowledging, so the same message is redelivered. Exactly-once processing is a property of the whole pipeline and normally comes from idempotent consumers or transactional bookkeeping on the consumer side.
Is Kafka a message queue?
Kafka is a distributed, replicated commit log. It supports queue-like work sharing through consumer groups and pub-sub through independent groups, but it does not delete a record when it is consumed and it does not track per-message acknowledgements. That difference in model is why Kafka is usually described as an event-streaming platform rather than a queue.