Messaging & Event Streaming
ActiveMQ Classic in Production: KahaDB, High Availability, Security and Monitoring
Advanced10 min readUpdated
Running ActiveMQ Classic beyond a laptop means understanding where messages live and who may touch them. This guide explains the KahaDB journal and index, persistent versus non-persistent delivery, redelivery and dead-letter queues, master/slave failover on a shared store, authentication and authorization plugins, TLS for clients, and the JMX, Jolokia and CLI tools you use to watch a broker and diagnose a stuck queue.
What you will learn
- Explain how KahaDB journals, indexes and cleans up message data, and tune its key settings
- Configure redelivery policies, expiry and per-queue dead-letter queues
- Set up shared-store or JDBC master/slave failover and know what it does and does not protect against
- Secure a broker with JAAS authentication, destination-level authorization and TLS
- Monitor queues, memory and store usage through JMX, Jolokia and the activemq CLI
- Diagnose the common production failures: stuck queues, growing stores, flow-controlled producers
A broker that works on your machine and a broker you can trust with orders differ in three places: what happens to a message when something crashes, who is allowed to send and receive, and whether anyone notices when a queue stops moving. This guide covers those three areas for ActiveMQ Classic 6. It assumes you have a running broker and a basic activemq.xml, as described in the setup guide, and focuses on the decisions that matter once real traffic arrives.
How messages are stored
ActiveMQ keeps every persistent message in a persistence adapter until it has been acknowledged by all its consumers. The adapter you choose determines throughput, recovery time and your options for failover.
KahaDB: journal plus index
KahaDB is the default store and the right choice for almost everyone. It writes messages to an append-only journal and keeps a separate index that maps queues and subscriptions to journal positions. Sequential appends are what make it fast: a message is written once, at the end of the current journal file, regardless of which queue it belongs to.
The data/kahadb directory contains a few kinds of files:
db-<n>.log— journal files, each 32 MB by default. Messages, acknowledgements and transaction records are appended in order. When a file is full the next one is created.db.data— the B-tree index: for every queue, which journal positions hold its pending messages; for every durable subscription, how far it has consumed.db.redo— a redo log used to rebuilddb.dataif the broker was killed mid-write.lock— the file lock that a second broker sharing this directory waits on (the basis of shared-store failover).
Two background jobs keep the store healthy. A checkpoint (every 5 seconds by default) flushes the index to disk so that recovery has less journal to replay. A cleanup (every 30 seconds) deletes journal files that no longer contain any referenced message. This second rule is the source of the classic "why is my disk full?" question: a journal file survives as long as one message in it is still unconsumed, which means one forgotten message on an unused queue can pin gigabytes of newer journal files behind it.
The settings worth knowing sit on the <kahaDB> element:
<!-- enableJournalDiskSyncs: fsync on every persistent send (safe, slower).
indexCacheSize: index pages kept in memory; raise it for many destinations. -->
<persistenceAdapter>
<kahaDB directory="${activemq.data}/kahadb"
journalMaxFileLength="32mb"
enableJournalDiskSyncs="true"
checkpointInterval="5000"
cleanupInterval="30000"
indexCacheSize="10000"
concurrentStoreAndDispatchQueues="true"/>
</persistenceAdapter>
enableJournalDiskSyncs is the safety-versus-speed knob. With it on, the broker calls fsync before acknowledging a persistent send, so a power loss cannot lose an acknowledged message; with it off, the operating system's page cache decides, and throughput rises several-fold. concurrentStoreAndDispatchQueues lets the broker hand a message to a waiting consumer while the disk write is still in progress, and skip the write entirely if the consumer acknowledges first — a large gain for queues that are drained as fast as they fill.
When one broker hosts destinations with very different traffic patterns, the mKahaDB adapter assigns separate KahaDB instances to destination patterns, so a slow queue with old messages cannot pin the journal files of a fast one.
JDBC store
The jdbcPersistenceAdapter puts messages in three tables (ACTIVEMQ_MSGS, ACTIVEMQ_ACKS, ACTIVEMQ_LOCK) of any supported relational database. It is much slower than KahaDB — every message is an INSERT and a DELETE — but it lets you reuse a database team's backup, replication and monitoring, and it provides a lock table for master/slave failover without shared filesystem semantics.
<bean id="postgres" class="org.postgresql.ds.PGSimpleDataSource">
<property name="url" value="jdbc:postgresql://db:5432/activemq"/>
<property name="user" value="activemq"/>
<property name="password" value="${db.password}"/>
</bean>
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#postgres" createTablesOnStartup="true"/>
</persistenceAdapter>
Older documentation also mentions the LevelDB adapters and the AMQ store; both have been removed, and configuration that references them fails on 6.x.
Persistent versus non-persistent delivery
The store only sees messages sent with DeliveryMode.PERSISTENT (the JMS default). Non-persistent messages live in broker memory and, when memory is short, are spilled to a temporary file store bounded by tempUsage. The difference in producer latency is large because a persistent send waits for the journal write, while a non-persistent send is acknowledged as soon as the broker has the bytes. For topics the rule has a twist: a persistent message published on a topic is only stored if a durable subscription exists for it; otherwise there is nobody to keep it for.
Redelivery, expiry and dead-letter queues
A message the consumer cannot process is redelivered, not dropped. The number of attempts is decided by the client's RedeliveryPolicy: six redeliveries one second apart by default, configurable on the connection factory or the URL.
ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory("failover:(tcp://broker:61616)");
RedeliveryPolicy policy = factory.getRedeliveryPolicy();
policy.setMaximumRedeliveries(5);
policy.setInitialRedeliveryDelay(1_000);
policy.setUseExponentialBackOff(true);
policy.setBackOffMultiplier(3); // 1 s, 3 s, 9 s, 27 s, 81 s
policy.setMaximumRedeliveryDelay(60_000);
After the last attempt the message is moved to a dead-letter queue (DLQ). The default is one shared ActiveMQ.DLQ for the whole broker, which is convenient for a demo and useless in production, because you cannot tell which queue a dead message came from without opening it. A per-destination strategy in the destination policy is the usual fix:
<policyEntry queue=">">
<deadLetterStrategy>
<individualDeadLetterStrategy queuePrefix="DLQ." useQueueForQueueMessages="true"
processExpired="true" processNonPersistent="false"/>
</deadLetterStrategy>
</policyEntry>
Expired messages — those whose JMSExpiration passed before delivery — go to the DLQ too when processExpired is true, which is the default. That surprises teams who use short time-to-live values as a way of throwing away stale data: the stale data ends up in the DLQ instead. Set processExpired="false" on those destinations, or give the DLQ its own expiry with the expiration attribute on the strategy so that it does not grow forever.
Whatever you configure, monitor the DLQs: a rising count is the earliest signal that a consumer has a bug or a downstream dependency is failing.
High availability: master/slave on a shared store
ActiveMQ Classic has no replication built in. High availability comes from running two or more brokers that point at the same store; only one — the master — can hold the store lock and serve clients. The others start, fail to acquire the lock, and wait. When the master dies, its lock is released, a slave acquires it, replays the index and starts its transport connectors, and clients using the failover: transport reconnect to it.
| Shared store | Lock mechanism | Watch out for |
|---|---|---|
| KahaDB on shared filesystem (NFSv4, GFS2, a SAN) | File lock in the lock file | The filesystem must implement POSIX locking correctly; NFSv3 and some cloud file services do not, and a lock that is not released leaves you with two masters |
| JDBC store | Row lock in ACTIVEMQ_LOCK, refreshed periodically | Lower throughput; the database is now the single point of failure and must itself be highly available |
The slave configuration is identical to the master's: same store location, different brokerName is not required but helps in logs. Failover is not instant — replaying the KahaDB index of a large store takes seconds to minutes — and messages in flight to a consumer at the moment of failure are redelivered, so consumers must be idempotent. Note that this design protects against a broker process failing; if the storage underneath is lost, so are the messages, which is why the store belongs on replicated storage and in backups.
Securing the broker
A default ActiveMQ installation accepts anonymous connections on every transport. Securing it involves three layers: proving who a client is, deciding what it may do, and encrypting the wire.
Authentication
The simple authentication plugin lists users directly in activemq.xml and is fine for a small, internal broker:
<plugins>
<simpleAuthenticationPlugin anonymousAccessAllowed="false">
<users>
<authenticationUser username="orders-svc" password="${orders.password}" groups="producers"/>
<authenticationUser username="fulfilment" password="${fulfilment.password}" groups="consumers"/>
<authenticationUser username="ops" password="${ops.password}" groups="admins"/>
</users>
</simpleAuthenticationPlugin>
</plugins>
The JAAS plugin delegates to a standard Java login module configured in conf/login.config, with users and groups in conf/users.properties and conf/groups.properties by default, or an LDAP module for corporate directories. It is the right choice as soon as accounts must be managed outside the broker configuration:
<plugins>
<jaasAuthenticationPlugin configuration="activemq"/>
</plugins>
activemq {
org.apache.activemq.jaas.PropertiesLoginModule required
org.apache.activemq.jaas.properties.user="users.properties"
org.apache.activemq.jaas.properties.group="groups.properties";
};
Clients pass credentials at connection time: factory.createContext("orders-svc", password). Certificates can replace passwords through the jaasCertificateAuthenticationPlugin, which maps a client certificate's distinguished name to a user; combine it with ssl:// transports that require client authentication (transport.needClientAuth=true).
Authorization
The authorization plugin grants read, write and admin rights per destination pattern to groups. admin is the right to create the destination, which producers need if queues are created on first use. Advisory topics must stay readable by everyone, or clients that rely on them fail in confusing ways.
<authorizationPlugin>
<map>
<authorizationMap>
<authorizationEntries>
<authorizationEntry queue="orders.>" read="consumers" write="producers" admin="producers,consumers"/>
<authorizationEntry queue="DLQ.>" read="admins" write="admins" admin="admins"/>
<authorizationEntry queue=">" read="admins" write="admins" admin="admins"/>
<authorizationEntry topic=">" read="admins" write="admins" admin="admins"/>
<authorizationEntry topic="ActiveMQ.Advisory.>" read="everyone" write="everyone" admin="everyone"/>
</authorizationEntries>
<tempDestinationAuthorizationEntry>
<tempDestinationAuthorizationEntry read="everyone" write="everyone" admin="everyone"/>
</tempDestinationAuthorizationEntry>
</authorizationMap>
</map>
</authorizationPlugin>
Entries are matched most-specific first, so the orders.> rule wins over > for order queues. Temporary destinations get their own entry; without it, request-reply clients cannot create their reply queues.
Encryption
Add an <sslContext> inside <broker> pointing at a keystore, and expose an ssl:// or nio+ssl:// connector. The broker's JVM must be able to read the keystore, and clients must trust the certificate — a private certificate authority whose root is imported into the clients' truststore is the usual arrangement inside a company. The web console has its own TLS configuration in conf/jetty.xml, and its credentials in conf/jetty-realm.properties must be changed from admin/admin; passwords in any of these files can be encrypted with bin/activemq encrypt and referenced with ENC(...) so that plain text is not checked into version control.
Monitoring and diagnosis
Everything the broker knows about itself is exposed through JMX MBeans in the org.apache.activemq domain, and every other tool is a view on those beans.
| Tool | Reaches JMX through | Best for |
|---|---|---|
Web console (http://host:8161/admin) | In-process | Quick look at queues, consumers, browsing and purging |
bin/activemq bstat / dstat / query | Remote JMX | Scripts and shell diagnostics |
Jolokia (http://host:8161/api/jolokia) | HTTP/JSON bridge | Prometheus exporters, dashboards, anything that speaks HTTP |
| JConsole, VisualVM | Remote JMX | Heap, threads and MBeans in one place |
| Hawtio | Jolokia | Rich browser console, also used by Artemis |
Remote JMX is off by default. Enable it in bin/env with ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=true ...", bind it to a management interface, and never expose an unauthenticated JMX port: it allows arbitrary code execution.
The Jolokia endpoint answers plain HTTP requests, which makes it easy to script checks:
# Depth, consumers and in-flight count for one queue (credentials are the web console's)
curl -s -u ops:secret \
'http://broker:8161/api/jolokia/read/org.apache.activemq:type=Broker,brokerName=orders-broker,destinationType=Queue,destinationName=orders.placed/QueueSize,ConsumerCount,InFlightCount,EnqueueCount,DequeueCount'
The attributes that deserve alerts are few:
- QueueSize per queue — a queue that grows for more than a few minutes has lost its consumers or has a consumer that is too slow.
- ConsumerCount per queue — zero on a queue that should have consumers is an outage in progress.
- InFlightCount — messages dispatched but not acknowledged; a constant value with no dequeue means a stuck consumer.
- MemoryPercentUsage, StorePercentUsage, TempPercentUsage on the broker — approaching 100 means producer flow control is about to block your producers.
- ExpiredCount and the depth of every
DLQ.*queue. - TotalConnectionsCount and the connection count per transport, against the
maximumConnectionsyou set.
data/activemq.log is the other source. Two messages to grep for are the flow-control warning (Usage Manager Memory Limit reached), and Slow KahaDB access, which means the disk cannot keep up with journal writes and is the moment to move data/ to faster storage or reconsider enableJournalDiskSyncs.
Common mistakes
- KahaDB on NFSv3 or a cloud file share without proper locking. Both brokers believe they hold the lock; the store is corrupted. Use NFSv4 with locking verified, a cluster filesystem, or a JDBC store.
- A single
ActiveMQ.DLQnobody reads. Configure per-queue DLQs, alert on them, and build a replay procedure before you need one. - Short TTLs as garbage collection. Expired messages land in the DLQ by default. Set
processExpired="false"where stale data should simply vanish. - Forgetting the advisory topic entry in the authorization map. Clients get cryptic
SecurityExceptions on connect. KeepActiveMQ.Advisory.>readable by everyone. - Turning off journal disk syncs to fix latency. It works, and it silently downgrades persistent delivery to "survives a crash of the broker but not of the machine". Make that a conscious decision and write it down.
- Unmonitored durable subscriptions. A durable subscriber that disconnected months ago pins journal files forever. Remove inactive subscriptions in the console, or set
offlineDurableSubscriberTimeouton the<broker>element so that subscriptions offline longer than that are removed automatically.
Key takeaways
- KahaDB appends messages to journal files and indexes them separately; files are reclaimed only when every message in them is consumed, so one stale message can pin the whole store.
- Persistent delivery waits for the journal write; non-persistent delivery is faster and loses messages only if the broker process dies.
- Redelivery is decided by the client's policy; exhausted messages and, by default, expired ones go to the dead-letter queue — make it per-destination and watch it.
- High availability is a master/slave pair sharing a KahaDB or JDBC store with a lock; networks of brokers are routing, not redundancy.
- Secure in three layers: JAAS or simple authentication, per-destination read/write/admin authorization, and TLS on transports and the console.
- Monitor QueueSize, ConsumerCount, InFlightCount, memory/store/temp percentages and DLQ depth through JMX or Jolokia, and alert before producer flow control blocks your applications.
Sources and further reading
Frequently asked questions
- Why does the KahaDB directory keep growing even though my queues are empty?
- A journal file is only deleted when no message in it is still referenced. One unconsumed message, one un-acknowledged message on a durable subscription, or one uncommitted transaction pins the whole 32 MB file, and every later file that follows it. Look for a queue with a handful of old messages, an inactive durable subscriber, or a pending XA transaction, and clear it; the next cleanup cycle reclaims the files.
- Is a network of brokers enough for high availability?
- No. A network forwards messages between running brokers, but messages already stored on a broker that goes down are unavailable until it returns. High availability in ActiveMQ Classic requires a master/slave pair sharing one store, either KahaDB on shared storage that supports proper file locking or a JDBC store, so that the slave can take over the same messages.
- Does non-persistent delivery mean a message is lost on any failure?
- It means the broker does not write it to the store before acknowledging the send, so it is lost if the broker process dies. It is not lost when the consumer fails or the network drops, and the client's failover transport keeps unsent messages in memory during a reconnection. For metrics, presence updates and cache invalidations the trade-off is usually right.
- What is the first thing to check when a queue stops draining?
- Whether it has consumers, and what those consumers are doing. The web console or the queue MBean shows the consumer count and the in-flight count; a consumer with in-flight messages that never acknowledges is stuck, often on a downstream call without a timeout. If consumers are present and idle, check whether the messages are all in a single message group or blocked by a selector.