Data & Databases
Centralised Log Management with Elasticsearch, Logstash and Kibana
Intermediate8 min readUpdated
Logs only become useful when they are structured, centralised and searchable within seconds of being written. This guide walks the Elastic stack end to end: emitting JSON logs, shipping them with Beats or Elastic Agent, parsing with Logstash or ingest pipelines, controlling mappings with index templates, managing retention with ILM, and sizing shards so the cluster stays healthy.
What you will learn
- Describe the role of each Elastic stack component and choose between Logstash and ingest pipelines
- Emit structured JSON logs and map them onto a consistent field schema
- Write grok and dissect parsing rules and enrich events during ingestion
- Control mappings with index and component templates and retention with index lifecycle management
- Size shards and indices so the cluster stays stable as log volume grows
A single application server writing to a file is easy to debug: open the file, search it, done. Twenty containers behind a load balancer, restarting on deploy and disappearing when they scale in, is a different problem entirely. By the time you know something went wrong, the container that logged the cause may not exist. Centralised logging exists to make that failure mode impossible: every line is shipped off the machine as it is written, parsed into fields, and made searchable across all services at once.
The Elastic stack — Elasticsearch for storage and search, Logstash for processing, Kibana for querying and visualisation, and the Beats or Elastic Agent shippers — remains the most widely deployed answer. This guide covers the current major line, Elastic 9, and focuses on the decisions that determine whether the platform stays useful and affordable as volume grows.
What each component does
- Elasticsearch stores documents in indices and answers queries. It is a distributed search engine built on Apache Lucene, so it shares the inverted-index and analysis machinery described in the Apache Solr guide; this guide will not repeat it. What matters here is that it is a near-real-time store: a document becomes searchable after a refresh, by default within a second.
- Kibana is the query and visualisation front end, and also the administrative interface for pipelines, lifecycle policies, alerting rules and dashboards.
- Beats are single-purpose shippers written in Go: Filebeat for log files and containers, Metricbeat for system and service metrics, Heartbeat for uptime checks, Packetbeat and Auditbeat for network and audit data.
- Elastic Agent is the newer, unified shipper. One process handles logs, metrics and security data, and it downloads integrations — packaged input configuration, ingest pipelines, index templates and prebuilt dashboards — from a central Fleet server. For a new deployment this is the default path; Beats remain fully supported and are simpler when you need exactly one thing.
- Logstash is a standalone processing pipeline with inputs, filters and outputs. It buffers, transforms and routes, and it can read from sources nothing else can.
The stack is only half of the picture. Logs, metrics and traces are three views of the same system, and an application that exposes health and metrics endpoints as described in Actuator and observability gives you the other two.
Log at the source as if a machine will read it
The single highest-leverage change in any logging project happens before anything is shipped: make the application emit structured events rather than prose.
{
"@timestamp": "2026-09-07T08:14:22.481Z",
"log.level": "ERROR",
"service.name": "payment-api",
"service.version": "4.2.1",
"trace.id": "3f1c0a9d5e6b47c8",
"transaction.id": "9b2e77a1",
"event.dataset": "payment.settlement",
"http.request.method": "POST",
"url.path": "/v1/settlements",
"http.response.status_code": 502,
"message": "Settlement rejected by acquirer",
"error.type": "AcquirerTimeoutException",
"labels": { "merchant_id": "M-88213" }
}
Every field is queryable, aggregatable and chartable without a parser standing in between. Java applications get this from a Logback or Log4j2 JSON encoder; the ECS-compatible encoders emit the field names above directly. When a document looks wrong, pasting it into a JSON formatter makes a stray nesting level or a duplicated key obvious immediately.
Which brings up the second decision: adopt a field naming convention and keep it. The Elastic Common Schema defines names for the concepts every system logs — host.name, service.name, event.duration, error.stack_trace, user.id — and using it means the prebuilt dashboards and detection rules work on your data, and that two teams' logs can be correlated. Invent your own names and every dashboard becomes bespoke.
Two rules worth writing into a coding standard: never log secrets, card numbers or personal data you would not put in a database, because a log platform is usually less protected than the database; and put the identifiers that let you correlate — trace id, request id, tenant id — into a mapped diagnostic context so every line carries them automatically.
Shipping and parsing
Filebeat tails files, tracks its read position in a registry so a restart does not duplicate or lose data, handles multiline stack traces, and applies backpressure when the destination slows down.
filebeat.inputs:
- type: filestream
id: payment-api
paths: ["/var/log/payment-api/*.json"]
parsers:
- ndjson:
target: ""
overwrite_keys: true
add_error_key: true
processors:
- add_host_metadata: ~
- add_kubernetes_metadata: ~
- drop_event:
when:
equals:
log.level: "TRACE"
output.elasticsearch:
hosts: ["https://es01:9200"]
api_key: "${ES_API_KEY}"
Containers change the shipping model rather than the principle: write to stdout, let the runtime collect it, and run one shipper per node with autodiscovery that attaches the pod, namespace and container labels — see Docker fundamentals for why logging to stdout is the container-native default.
For sources you cannot change — a firewall syslog feed, an appliance emitting fixed-width lines — parsing is unavoidable. Two tools do the work.
Dissect splits on fixed delimiters. It is dramatically faster than a regular expression and should be your first choice when the format is stable:
%{clientip} - %{user} [%{timestamp}] "%{method} %{path} %{httpversion}" %{status} %{bytes}
Grok is a library of named regular expressions and handles genuinely irregular input:
%{IPORHOST:source.ip} %{USER:user.name} \[%{HTTPDATE:event.created}\] "%{WORD:http.request.method} %{URIPATHPARAM:url.original}" %{NUMBER:http.response.status_code:int}
Grok patterns are also the classic cause of ingestion falling over: an unanchored pattern that backtracks catastrophically will saturate a CPU core. Anchor patterns, prefer dissect where possible, and test in Kibana's Grok Debugger before deploying.
Where the parsing runs is the next choice. An Elasticsearch ingest pipeline executes inside the cluster on ingest-role nodes and needs no extra service:
PUT _ingest/pipeline/payment-logs
{
"processors": [
{ "dissect": { "field": "message", "pattern": "%{ts} %{level} [%{thread}] %{logger} - %{msg}" } },
{ "date": { "field": "ts", "formats": ["ISO8601"], "target_field": "@timestamp" } },
{ "geoip": { "field": "source.ip", "target_field": "source.geo", "ignore_missing": true } },
{ "remove": { "field": ["ts", "message"], "ignore_missing": true } }
],
"on_failure": [
{ "set": { "field": "event.kind", "value": "pipeline_error" } },
{ "set": { "field": "error.message", "value": "{{ _ingest.on_failure_message }}" } }
]
}
Note the on_failure block. Without it, a malformed line rejects the document and you lose the very event you most wanted to see. Tag failures and index them anyway.
Logstash is worth the extra service when you need a durable buffer between noisy producers and the cluster (its persistent queue survives restarts), when events must fan out to more than one destination, or when transformation is heavy enough that you would rather not spend data-node CPU on it. A common production shape is Beats to Kafka for buffering, Logstash consuming from Kafka, and Elasticsearch as the sink.
Mappings, data streams and retention
Elasticsearch will guess a mapping for any field it has not seen. That convenience causes two chronic problems: a field arriving first as a number and later as a string is rejected, and a mapping explosion — thousands of dynamically created fields from unbounded keys such as labels.<request-id> — bloats the cluster state until the cluster becomes unstable.
Control this with component templates composed into an index template:
PUT _index_template/logs-payment
{
"index_patterns": ["logs-payment-*"],
"data_stream": {},
"priority": 200,
"composed_of": ["logs-mappings", "logs-settings"],
"template": {
"settings": { "index.number_of_shards": 2, "index.number_of_replicas": 1,
"index.lifecycle.name": "logs-90d" },
"mappings": {
"dynamic": "strict",
"properties": {
"@timestamp": { "type": "date" },
"service.name":{ "type": "keyword" },
"log.level": { "type": "keyword" },
"message": { "type": "text" },
"http.response.status_code": { "type": "integer" }
}
}
}
}
The keyword versus text distinction is the one to internalise: keyword stores the value whole for exact filtering, aggregation and sorting; text is analysed into tokens for full-text search and cannot be aggregated efficiently. Service names, log levels and hostnames are keywords. Free-form messages are text.
A data stream is the modern shape for append-only time-series data. You write to one name, logs-payment-prod, and Elasticsearch manages a series of hidden backing indices behind it, rolling over to a new one on a size or age threshold. Rollover matters because deleting old data means dropping a whole index, which is nearly free, whereas deleting documents by query is expensive.
Index lifecycle management automates the whole life of that data:
PUT _ilm/policy/logs-90d
{
"policy": { "phases": {
"hot": { "actions": { "rollover": { "max_primary_shard_size": "40gb", "max_age": "1d" } } },
"warm": { "min_age": "3d", "actions": { "forcemerge": { "max_num_segments": 1 },
"shrink": { "number_of_shards": 1 } } },
"cold": { "min_age": "15d", "actions": { "set_priority": { "priority": 0 } } },
"delete": { "min_age": "90d", "actions": { "delete": {} } }
}}
}
Hot nodes take the writes on fast disks; warm nodes hold recent, rarely written data; cold nodes hold what is queried occasionally. Set the retention per dataset — audit events and debug logs rarely justify the same window — and remember that the cheapest gigabyte is the one you never indexed. Dropping TRACE events and health-check requests at the shipper often halves volume with no investigative loss.
Querying and dashboards
In Kibana's Discover view, KQL covers most day-to-day filtering: service.name:"payment-api" and http.response.status_code >= 500 and not url.path:"/health". The older Lucene syntax is still available for regular expressions and fuzzy matching.
ES|QL, Elastic's piped query language, is the bigger change of the current generation. It expresses filtering, transformation and aggregation in one readable chain:
FROM logs-payment-*
| WHERE http.response.status_code >= 500 AND @timestamp > NOW() - 1 hour
| STATS failures = COUNT(*) BY service.name, error.type
| SORT failures DESC
| LIMIT 20
Build dashboards around questions rather than around available fields: error rate per service, latency percentiles, top failing endpoints, deploy markers. Kibana alerting rules turn any of those into a notification when a threshold is crossed. And use runtime fields for the occasional attribute you forgot to extract — they are computed at query time, so they cost nothing until used and save a reindex.
Sizing and operations
Most unhealthy log clusters suffer from the same disease: too many shards. Every shard is a Lucene index with memory and file-handle overhead, and the cluster state tracking them must be replicated to every node.
- Target 10–50 GB per shard, and roughly 20 shards or fewer per gigabyte of heap on a node.
- Set the JVM heap to about half the machine's RAM and never above ~31 GB, so pointers stay compressed; the rest of the RAM serves the page cache that Lucene depends on.
- Separate node roles once the cluster grows: dedicated master-eligible nodes (three, for quorum) keep cluster state stable when data nodes are busy.
- Watch the disk watermarks. At 85% Elasticsearch stops allocating new shards to a node; at 95% it makes indices read-only. Both surprise teams at three in the morning.
- Snapshot to object storage. Replication protects against a lost node, not against a bad ingest pipeline or a mistaken delete.
Two alternatives deserve a fair mention. OpenSearch is an Apache-licensed fork with a compatible API and its own dashboards, and it is the natural choice where the Elastic licence is a blocker. Grafana Loki takes the opposite architectural bet: index only a small set of labels, keep raw log content compressed in object storage, and grep at query time. That is much cheaper at high volume and much weaker at full-text search across everything. If your queries are always "this service, this time window", Loki fits; if they are "find this token anywhere", it does not. Where log volume grows past what a search cluster should hold, the storage and processing patterns in the big data architecture guide become the alternative.
Key takeaways
- Structured JSON logs with a consistent schema such as ECS remove parsing from the pipeline and make every field queryable and chartable.
- Elastic Agent with Fleet integrations is the default shipper for new deployments; Beats stay simple and Logstash earns its place for buffering, fan-out and heavy transformation.
- Prefer dissect over grok when the format is fixed, anchor grok patterns, and always add an
on_failurehandler so bad lines are tagged rather than dropped. - Control mappings with index and component templates, distinguish
keywordfromtext, and prevent mapping explosions from unbounded field names. - Write to data streams, roll over on size or age, and express retention as an ILM policy with hot, warm, cold and delete phases per dataset.
- Keep shards in the 10–50 GB range, heap under about 31 GB, watch the disk watermarks, and snapshot to object storage because replicas are not backups.
Sources and further reading
Frequently asked questions
- Do I still need Logstash if ingest pipelines exist?
- Often not. Elasticsearch ingest pipelines handle parsing, enrichment and date extraction inside the cluster, and Elastic Agent integrations ship with pipelines already written. Logstash earns its place when you need buffering ahead of the cluster, protocol translation from sources Beats cannot read, fan-out to several destinations, or heavy transformation you would rather not run on data nodes.
- How long should I keep logs?
- Long enough to investigate the incidents you actually investigate, plus whatever compliance requires — for most teams that is days of hot searchable data and weeks or months of colder storage. Encode the answer in an ILM policy rather than a deletion script, and set the retention per data type: request logs, audit records and debug traces rarely deserve the same window.
- Why is my cluster yellow or red?
- Yellow means some replica shards are unassigned, so data is intact but not redundant — common on a single-node development cluster where replicas can never be allocated. Red means a primary shard is unassigned and some data is currently unavailable. The allocation explain API reports the precise reason, which is usually disk watermarks, a node that left, or too few nodes for the requested replica count.
- Is the Elastic stack the only option for logs?
- No. OpenSearch is an Apache-licensed fork of Elasticsearch with a compatible API and its own dashboard, and Grafana Loki takes a different approach by indexing only labels and storing raw log lines in object storage, which is far cheaper at volume but weaker at full-text search. Choose on query patterns, cost per gigabyte and the tooling your team already runs.