Messaging & Event Streaming
Message-Oriented Middleware: Queues, Topics and Why Systems Talk Asynchronously
Beginner11 min readReviewed by Mariem Ouled on Updated
Message-oriented middleware lets applications exchange data through a broker instead of calling each other directly. After this guide you will be able to explain the point-to-point and publish-subscribe models, name the parts of a message, compare the main protocols and brokers, and judge when asynchronous messaging is the right integration style for a problem.
What you will learn
- Compare the four integration styles and explain what messaging adds over file transfer, shared databases and remote calls
- Describe the anatomy of a message and the roles of producer, broker, destination and consumer
- Distinguish point-to-point queues from publish-subscribe topics and pick the right one for a use case
- Explain the three forms of decoupling a broker provides and their costs
- Position JMS, AMQP, STOMP, MQTT and the Kafka protocol relative to each other
- Choose between ActiveMQ, RabbitMQ and Kafka for a given workload
Two applications can share data by exchanging files, by reading the same database, by calling each other over the network, or by sending each other messages through an intermediary. The last option — message-oriented middleware (MOM) — is the one that lets each side keep running when the other is slow, busy or down. That property is what makes messaging the backbone of order processing, payment pipelines, log collection and most event-driven architectures. This guide explains the model that all brokers share, so that the JMS, ActiveMQ, RabbitMQ and Kafka guides that follow make sense as variations on one theme rather than four unrelated products.
Four integration styles, one trade-off
Every integration problem comes down to moving data and triggering behaviour between systems that were not written together. The classic classification lists four styles, and each fixes a weakness of the previous one at the cost of a new one.
| Style | How it works | Strength | Weakness |
|---|---|---|---|
| File transfer | One system writes a file, another reads it later | Simple, no shared runtime, any language | Latency of hours, format drift, no feedback |
| Shared database | Several applications read and write the same schema | Data is always consistent and current | Schema becomes a contract nobody can change; contention |
| Remote procedure call (HTTP, gRPC, RMI) | One application calls a function exposed by another | Immediate answer, familiar programming model | Caller blocks; both sides must be up; cascading failure |
| Messaging | Applications hand small packets to a broker that delivers them | Asynchronous, reliable, decoupled | Eventual consistency, harder debugging, more moving parts |
Messaging is not "better" than remote calls. It is a different contract: the sender learns only that the broker accepted the message, not that anyone processed it. Systems built on messaging therefore need to be comfortable with work happening later and with results arriving as further messages rather than return values.
Anatomy of a message
Whatever the broker, a message has the same three parts.
- Headers are fields the broker itself understands and may act on: a unique identifier, a timestamp, a destination, a priority, an expiry, a delivery mode (should the broker write this to disk before acknowledging it?) and, for request-reply, a reply-to address and a correlation identifier.
- Properties (RabbitMQ and Kafka call them headers too) are application-defined key-value pairs. Consumers can filter on them, routers can branch on them, and tracing systems propagate trace identifiers in them. They are the place for metadata such as
tenant,eventTypeorschemaVersion. - The body (payload) is an opaque sequence of bytes as far as the broker is concerned. JSON is the most common encoding today; binary formats such as Avro or Protocol Buffers are preferred at high volume because they are smaller and carry a schema.
A useful discipline is to keep everything a router or a filter needs in headers or properties, and never require anyone to parse the body to decide where a message goes. Brokers cannot look inside the body, and code that does becomes coupled to the payload format.
{
"headers": {
"messageId": "ID:app-1-45231-1725700000000-1:1:1:1",
"timestamp": 1725700000000,
"deliveryMode": "PERSISTENT",
"replyTo": null,
"correlationId": null
},
"properties": {
"eventType": "OrderPlaced",
"schemaVersion": "2",
"tenant": "eu-west"
},
"body": "{\"orderId\":\"o-9911\",\"total\":149.90,\"currency\":\"EUR\"}"
}
The identifiers in the header are what allow a consumer to detect duplicates and what tie a reply back to its request; you can generate your own with a UUID generator when the broker does not assign one.
Producers, consumers, destinations and the broker
The vocabulary is small. A producer (publisher, sender) creates a message and hands it to the broker. The broker stores it in a destination and delivers it to one or more consumers (subscribers, receivers). The producer never learns who the consumers are, and the consumers never learn who produced the message: both only know the destination's name.
The broker is the component that makes this reliable. It accepts messages even when no consumer is connected, persists them if asked, redelivers them when a consumer fails, and enforces security on who may send to or read from a destination. It is also a single point of contact for monitoring: queue depth, consumer count and throughput are all visible in one place.
Not every messaging system needs a broker. Brokerless libraries such as ZeroMQ deliver directly between processes and win on latency, but then there is nowhere for a message to wait while the receiver is down. As soon as you need a queue that several applications share and that must survive restarts, you need a broker.
Two messaging domains: queues and topics
Destinations come in two flavours, and choosing the right one is the first design decision in any messaging system.
Point-to-point: the queue
A queue delivers each message to exactly one consumer. If ten consumers read from the same queue, the broker spreads the messages across them — a pattern called competing consumers — and this is how you scale throughput horizontally: add a consumer, and the queue drains faster. Messages wait in the queue until someone reads them, so producer and consumer do not have to be running at the same time. Order processing, image resizing jobs and email dispatch are typical queue workloads: each unit of work must be done once.
Publish-subscribe: the topic
A topic delivers each message to every current subscriber. A single "customer address changed" event can fan out to the billing system, the shipping system and the analytics pipeline without the producer knowing any of them exist. By default a subscription only receives messages published while it is active; a durable subscription asks the broker to keep messages for a named subscriber that is temporarily offline. Without durability, a topic is a broadcast; with it, each subscriber effectively gets a private queue.
| Queue | Topic | |
|---|---|---|
| Consumers per message | One | Zero or many |
| Consumer offline | Message waits | Message lost, unless the subscription is durable |
| Scaling consumers | Adds throughput | Adds copies |
| Typical use | Work distribution, commands | Event notification, broadcast |
Modern brokers blur the line: RabbitMQ builds both from exchanges and bindings, and Kafka provides both through partitions and consumer groups. The two intents — do this once versus tell everyone — remain the right way to think about a design regardless of the product.
What decoupling actually buys you
"Loose coupling" is the standard justification for messaging, but it hides three separate benefits, each with a price.
Decoupling in time. The consumer does not need to be running when the producer sends. A nightly batch system can accept events all day; a payment provider that is down for maintenance receives its backlog when it returns. The price is that the producer cannot promise the user that anything has happened yet.
Decoupling in space. The producer addresses a destination name, not a host. You can move a consumer to another machine, run five copies of it, or replace it with a rewritten service, and the producer's configuration does not change. The price is that the broker becomes critical infrastructure and needs its own high-availability story.
Decoupling of load. A queue absorbs bursts. A web front end that receives ten thousand orders in a minute can enqueue them in seconds and let the fulfilment service work through them at its own pace, instead of timing out under a spike. The price is that the queue can grow without bound if consumers fall behind, so every system needs a plan for backlog: more consumers, message expiry, or backpressure on the producer.
Delivery guarantees in one paragraph
A broker can offer at-most-once delivery (fire and forget: fast, may lose messages), at-least-once delivery (the broker keeps the message until the consumer acknowledges it, so a crash before acknowledgement produces a redelivery and possibly a duplicate), or a form of exactly-once limited to a specific scope (Kafka transactions between Kafka topics, for example). Persistent delivery — writing the message to disk before confirming it to the producer — makes at-least-once survive a broker restart. Nearly every production system runs at-least-once and makes consumers idempotent, meaning that processing the same message twice has the same effect as processing it once. The patterns guide later in this subject covers how.
Protocols and APIs
The word "messaging" covers a standard API, several wire protocols and one product-specific protocol; confusing them causes real integration mistakes.
- JMS (Jakarta Messaging) is a Java API, not a protocol. Code written against
jakarta.jmsworks with any broker that ships a JMS client, but a JMS client talks to its broker in whatever wire format the vendor chose. Two brokers are not interoperable just because both "support JMS". - AMQP 0-9-1 is the wire protocol RabbitMQ was built around; it defines exchanges, queues, bindings and channels, and any language with an AMQP 0-9-1 client can talk to RabbitMQ.
- AMQP 1.0 is an OASIS and ISO standard that shares little beyond the name with 0-9-1. It defines how peers transfer messages, not a broker topology. ActiveMQ Classic, ActiveMQ Artemis, RabbitMQ (natively since 4.0) and Azure Service Bus all speak it.
- STOMP is a minimal text protocol you can drive from a Telnet session. It is popular for browser clients over WebSocket and for scripting languages.
- MQTT is a compact publish-subscribe protocol designed for constrained devices and unreliable networks; ActiveMQ and RabbitMQ both accept MQTT connections.
- The Kafka protocol is Kafka's own binary protocol. There is no standard behind it, but clients exist for every mainstream language.
- OpenWire is ActiveMQ Classic's native binary protocol, used by its Java, .NET and C++ clients.
For a Java team the practical consequence is: use JMS when you want to keep the option of swapping brokers, use the native client (RabbitMQ's amqp-client, Kafka's kafka-clients) when you want every feature of the broker you chose.
Choosing a broker
The three open-source brokers this subject covers solve overlapping but different problems.
| ActiveMQ Classic (and Artemis) | RabbitMQ | Apache Kafka | |
|---|---|---|---|
| Model | JMS queues and topics | Exchanges route to queues | Partitioned, replicated log |
| Message lifetime | Deleted on acknowledgement | Deleted on acknowledgement | Retained for a configured time regardless of consumption |
| Ordering | Per queue (per message group with JMSXGroupID) | Per queue with one consumer | Per partition |
| Replay old messages | No | No (streams excepted) | Yes, by resetting offsets |
| Routing flexibility | Selectors, virtual topics, composite destinations | Very high (exchange types, bindings, headers) | Low: a producer picks a topic and a key |
| Throughput sweet spot | Thousands to tens of thousands of messages/s | Similar, with rich routing | Hundreds of thousands to millions of records/s |
| Protocols | OpenWire, AMQP 1.0, STOMP, MQTT, WebSocket | AMQP 0-9-1, AMQP 1.0, STOMP, MQTT, streams | Kafka protocol |
| Natural fit | Java EE and Spring applications that need JMS | Task queues, RPC, flexible routing, polyglot teams | Event streaming, log pipelines, data integration, replay |
A few rules of thumb hold up well. If your consumers are JMS message-driven beans or Spring @JmsListener methods and you need transactional queues, ActiveMQ is the least friction; prefer Artemis for new deployments that need higher throughput or clustering, and Classic where you have existing configuration and tooling. If you need to route one message to different queues depending on several attributes, or you have producers in Python and consumers in Go, RabbitMQ's exchange model is hard to beat. If the data is a stream of events that several teams will consume at different times, that must be replayable, or that arrives at very high volume, Kafka's retained log is the right primitive, and forcing it to behave like a task queue will disappoint you.
Where messaging goes wrong
- Using a queue as a database. Queues are for messages in transit. Storing state in a queue and browsing it later leads to unbounded growth and unpredictable ordering. Persist state in a store; put events in the queue.
- Assuming ordering across consumers. A queue with three competing consumers delivers in order but processes in parallel, so message 2 can finish before message 1. If order matters, use one consumer per key (message groups, RabbitMQ single-active-consumer, Kafka partitions).
- Very large messages. A 50 MB PDF in a message body slows every hop and blows through broker memory limits. Store the file in object storage and send a reference — the claim-check pattern.
- Ignoring the backlog. A queue nobody monitors fills up silently until the broker hits its disk limit and stops accepting messages from every producer. Alert on queue depth and consumer count from day one.
- Coupling through the payload. If every consumer parses the same undocumented JSON, the "decoupled" system cannot change a field without a coordinated deployment. Version the payload and put a
schemaVersionproperty on every message.
Key takeaways
- Messaging is one of four integration styles; it trades immediate answers for resilience, elasticity and independence between systems.
- A message is headers the broker understands, properties the application defines, and an opaque body.
- Queues deliver each message once and scale with competing consumers; topics broadcast to every subscriber, durably if asked.
- Decoupling in time, space and load is what a broker buys you; unbounded backlogs and a critical piece of infrastructure are what it costs.
- JMS is a Java API, AMQP, STOMP and MQTT are wire protocols, and Kafka speaks its own; do not assume interoperability from a shared acronym.
- ActiveMQ fits JMS-centric Java systems, RabbitMQ fits routing-heavy polyglot systems, and Kafka fits high-volume, replayable event streams.
Sources and further reading
Frequently asked questions
- Is a message broker the same thing as an enterprise service bus?
- No. A broker stores and forwards messages between destinations; it does not understand their content. An enterprise service bus sits on top of a broker or a transport and adds content-based routing, transformation, protocol bridging and orchestration. Many ESB products embed a broker such as ActiveMQ, but you can run a broker on its own and most modern systems do.
- When is a plain HTTP call better than a message?
- When the caller genuinely needs the answer before it can continue — validating a card before confirming an order, for example — and the callee is expected to be available. Messaging shines when the work can be deferred, when producers and consumers scale at different rates, when the same event interests several consumers, or when the target system is sometimes unavailable.
- Does a queue guarantee that a message is processed exactly once?
- Not on its own. A broker can promise at-most-once or at-least-once delivery; the consumer can still crash after processing and before acknowledging, so the same message is redelivered. Exactly-once processing is a property of the whole pipeline and normally comes from idempotent consumers or transactional bookkeeping on the consumer side.
- Is Kafka a message queue?
- Kafka is a distributed, replicated commit log. It supports queue-like work sharing through consumer groups and pub-sub through independent groups, but it does not delete a record when it is consumed and it does not track per-message acknowledgements. That difference in model is why Kafka is usually described as an event-streaming platform rather than a queue.