Software Architecture

Application Performance: Latency, Percentiles, Profiling and the JVM

Advanced10 min readUpdated

Performance work fails when it starts with opinions about what is slow. This guide sets out the measurements that describe a system honestly — latency versus throughput, percentiles, utilisation and queueing — then works through the places time is actually lost in a JVM service, and closes with an optimisation loop that produces evidence rather than folklore.

What you will learn

  • Distinguish latency, throughput and concurrency and relate them with Little's law
  • Read percentile distributions and explain why averages hide user-visible problems
  • Profile a running service to find where time and allocation actually go
  • Diagnose the common causes of slowness — N+1 queries, pool exhaustion, GC pressure, lock contention
  • Apply caching and concurrency limits without introducing correctness or stability problems

Almost every performance investigation that goes badly starts the same way: someone says the application is slow, someone else has a theory, and a week disappears optimising something that was never on the critical path. The discipline that avoids this is unglamorous. Measure what users experience, find where the time goes with evidence, change one thing, measure again, and keep or revert based on the number rather than the theory.

This guide is about that loop and about the places time is actually lost in a typical JVM service. It concerns diagnosis and optimisation of a system you can observe; constructing the load that reveals problems before production does is covered in load testing methodology.

The three numbers and how they relate

Latency is how long one operation takes. Throughput is how many complete per unit time. Concurrency is how many are in flight at once. They are not independent, and Little's law relates them exactly:

L = λ × W

L = average number of requests in the system (concurrency)
λ = average arrival rate (throughput)
W = average time in the system (latency)

The law holds for any stable system regardless of its internals, which makes it a remarkably good sanity check. A service handling 500 requests per second with 200 ms average latency has, on average, 100 requests in flight — so a thread pool of 50 is a hard ceiling and the queue in front of it is where the time is going. Turn it around and it sizes things for you: to serve 2 000 requests per second at 50 ms, you need 100 concurrent slots, and if each holds a database connection you now know something important about the pool.

The second relationship worth internalising is what happens as utilisation rises. Queueing theory gives, for a simple system, a wait time proportional to ρ / (1 − ρ) where ρ is utilisation. At 50 % utilisation the queue adds about as much as the service time itself; at 90 % it adds nine times; at 95 %, nineteen. This is why a resource at 80 % looks healthy on a dashboard and behaves terribly, and why capacity plans that target high utilisation produce systems that fall over at the smallest traffic spike.

The practical consequence: find the bottleneck, the one resource nearest saturation, because nothing else matters until it moves. Optimising anything other than the bottleneck changes nothing measurable, and after you fix it the bottleneck moves somewhere else — which is the normal, expected outcome, not a failure.

Measure percentiles, never averages

Response times are right-skewed with a long tail, so summary statistics chosen carelessly describe a system nobody uses.

StatisticWhat it tells you
MeanAlmost nothing; hides tails and bimodality entirely
Median (p50)The typical experience
p90 / p95Where users start noticing
p99The tail your most active users hit constantly
p99.9Where infrastructure faults and GC pauses show
MaxOne data point; useful only as an alarm

A user whose page issues twenty backend calls encounters your p95 with probability close to two-thirds. Tail latency is therefore not an edge case; for composite pages it is the common case, which is why "tail latency dominates" is the single most useful phrase in performance work.

Percentiles must be computed from a histogram, not from a sample of "slow requests", and they must be computed over a stated window — a p99 over 24 hours conceals an hour of misery. Aggregate by summing histogram buckets across instances; never average pre-computed quantiles. And beware coordinated omission, the measurement bug where a client that waits for a response before sending the next one stops sampling exactly when the system is slowest, making the reported percentiles far better than reality.

Finally, tie the numbers to something meaningful: a service level objective stated in user terms, such as "99 % of checkout requests complete within 800 ms measured over 28 days". That converts performance from an opinion into a budget you can spend deliberately.

Finding where the time goes

With a target defined, the question becomes where the milliseconds are.

Start at the edge and work inward. A distributed trace of a slow request shows which hop consumed the time — application, database, downstream service, or the network between them. Without tracing, hours vanish in the wrong service. Instrumentation for this is described in Actuator and observability.

Then profile the process. Two profiler types answer different questions and both are needed:

  • A CPU profiler samples stacks to show where compute time is spent. Modern sampling profilers use async stack walking, which avoids the safepoint bias that made older tools report the wrong methods entirely. Read the result as a flame graph: width is total time, so look for wide plateaus, not tall towers.
  • An allocation profiler shows which call sites create objects. Allocation drives garbage collection, and GC time is frequently misattributed to whatever code happened to be running.

Java Flight Recorder is the pragmatic default on the JVM: it is built in, its overhead is around 1 % with the default profile, and it can be enabled on a production process for a bounded window.

# Attach to a running service and record for two minutes.
jcmd <pid> JFR.start name=diag settings=profile duration=120s filename=/tmp/diag.jfr

# Cheap first look while you wait: what are the threads doing?
jcmd <pid> Thread.print | grep -A 5 'java.lang.Thread.State: BLOCKED'

# GC behaviour at a glance.
jstat -gcutil <pid> 1000 30

Read thread dumps for the things profilers miss. Three dumps ten seconds apart tell you far more than one. Threads consistently BLOCKED on the same monitor mean lock contention; threads WAITING on a pool's getConnection mean the pool is exhausted; threads in Socket.read mean you are waiting on someone else and the problem is not here.

Where the time usually is

Across a lot of JVM services, the same handful of causes recur.

Database access

Almost always the first place to look, and usually the largest single win.

N+1 queries. An ORM loads 200 orders and then issues one query per order for its lines. In a test with three rows this is invisible; in production it is 201 round trips. The fix is a join fetch, an entity graph or a batched second query — and the way to catch it is to count statements per request in tests, not to hope. The mechanics in JPA are covered in Spring Data JPA.

Missing or unusable indexes. Read the execution plan rather than guessing. A sequential scan on a large table in an OLTP path is a defect; so is an index that cannot be used because the predicate wraps the column in a function.

EXPLAIN (ANALYZE, BUFFERS)
SELECT o.id, o.total
FROM   orders o
WHERE  o.customer_id = 4711
  AND  o.created_at >= now() - interval '90 days'
ORDER  BY o.created_at DESC
LIMIT  50;
-- Look for: Seq Scan on a large table, rows removed by filter, an estimate
-- far from the actual count, and sorts spilling to disk.

Over-fetching. Selecting every column of every row to display four fields, or paging by offset deep into a large result set. Project only what you need.

Pool exhaustion. Threads waiting for a connection show up as latency in the application with an idle-looking database. Watch pool wait time as a first-class metric, and remember that a bigger pool usually makes this worse, not better.

Transactions held open across remote calls. A @Transactional method that calls an HTTP API holds a database connection and its locks for the duration of that call. Do the remote work outside the transaction.

Memory and garbage collection

The JVM heap has generations because most objects die young. Young collections are cheap and frequent; old-generation work is what causes noticeable pauses.

Diagnose before tuning. Enable GC logging and look at three things: the percentage of wall-clock time spent in GC (above roughly 5 % is worth investigating), the pause-time distribution against your latency target, and whether live heap after each old collection is trending upward — the signature of a leak.

-Xlog:gc*,safepoint:file=/var/log/app/gc.log:time,uptime:filecount=5,filesize=20M
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/app/

Most GC problems are allocation problems. A service that allocates 2 GB per second to build strings and intermediate collections will collect constantly whatever collector you choose, and reducing allocation fixes it permanently. When a collector choice is genuinely the issue: G1 is a sound default balancing throughput and pause time; ZGC and Shenandoah target sub-millisecond pauses at some throughput cost and larger footprint, and are worth it when p99.9 latency is dominated by pauses. Sizing the heap generously reduces GC frequency but lengthens individual collections and increases page-cache competition — and in a container, always set the heap explicitly or verify the JVM's container awareness, because a limit the JVM does not see becomes an out-of-memory kill with no heap dump.

Concurrency and contention

Lock contention appears as latency that grows with concurrency while CPU stays flat. Look for synchronized blocks around shared caches or counters, single shared instances of expensive objects, and logging configured synchronously to a slow appender. Fixes in order of preference: remove the sharing, shrink the critical section, replace the lock with a concurrent data structure or an atomic, or partition the data so threads contend on different stripes.

The opposite failure is unbounded concurrency: an executor with an unbounded queue accepts work indefinitely, so a burst becomes a memory problem and every queued request eventually times out anyway. Bound every queue and reject early.

Serialisation, network and the work you did not need to do

Chatty remote calls, JSON serialisation of enormous payloads, compression settings, and TLS handshakes on connections that should have been pooled all appear in profiles more often than people expect. So does redundant work: computing the same value five times per request, logging at debug in a hot loop, or building an object graph the caller discards.

Caching, carefully

Caching is the highest-leverage optimisation and the one with the worst failure modes, because a cache converts a performance problem into a correctness problem.

Layer by layer: the browser and CDN handle static assets and cacheable GET responses using the validators discussed in REST API design; a distributed cache such as Redis holds shared computed results; a local in-process cache holds hot, small, read-mostly data with the lowest possible latency; and the database's own buffer pool caches pages beneath all of it.

The decisions that determine whether it helps:

  • What is the acceptable staleness? If the answer is "none", you do not want a cache, you want a faster query.
  • Expiry strategy. Time-based expiry is simple and predictable; explicit invalidation on write is precise and easy to get wrong; a short TTL combined with invalidation is a robust compromise.
  • Bounded size with a sensible eviction policy, because an unbounded local cache is a memory leak with a friendly name.
  • Stampede protection. When a popular key expires, hundreds of requests miss simultaneously and all recompute. Use a per-key lock, or serve stale content while one caller refreshes.
  • Measure the hit ratio. A cache below roughly 80 % hits is often adding latency and complexity for very little, and you should know that rather than assume.

Spring's cache abstraction and its providers are covered in caching.

The optimisation loop

  1. Define the target

    Pick one user-visible operation and one number: "p95 of GET /orders under 300 ms at 400 requests per second". Without this you cannot tell when to stop.
  2. Reproduce under representative load

    A workload that mirrors production's mix, data volume and concurrency. Optimising against an empty database teaches you nothing.
  3. Measure and locate

    Trace to find the slow hop, profile to find the slow code, and identify the single bottleneck resource.
  4. Form a hypothesis with a predicted effect

    "Batching these 40 queries into one should remove about 120 ms." A prediction that turns out wrong is information; a change with no prediction is a guess.
  5. Change one thing

    One change per measurement, or you will not know which one mattered — and one of them may have made things worse.
  6. Measure again and decide

    Compare against the baseline on the same workload. Keep the change if it moved the number, revert it if it did not, and record both outcomes.
  7. Repeat until the target is met, then stop

    The bottleneck will have moved. Continue only while the target is unmet; optimisation beyond a met objective is cost without benefit.

Two habits make this durable. Keep the benchmark in the pipeline so a regression is caught by the build rather than by a customer. And write down what you tried and what happened, including the failures — the next person to look at this system will otherwise repeat every one of them.

Key takeaways

  • Latency, throughput and concurrency are linked by Little's law, and queueing effects mean a resource above 80 % utilisation is already in trouble.
  • Report percentiles from merged histograms over a stated window, never averages and never averaged quantiles; for multi-call pages the tail is the common experience.
  • Trace first to find the slow hop, then profile CPU and allocation, and read several thread dumps for contention and pool exhaustion; database access — N+1 queries, missing indexes, over-fetching, exhausted connection pools — is the most common bottleneck and the cheapest to fix.
  • Most GC problems are allocation problems; diagnose from GC logs and reduce garbage before changing collectors, and always set the heap explicitly in containers.
  • Cache only after measuring, with a stated staleness tolerance, a bounded size, stampede protection and a monitored hit ratio.
  • Optimise in a loop: define the target, reproduce, locate, predict, change one thing, measure, and stop when the objective is met.

Sources and further reading

Frequently asked questions

Why is the average response time misleading?
Because response times are not symmetrically distributed. A service with a 40 ms mean can easily serve one request in twenty at over a second, and those slow requests are concentrated on the users doing the most valuable things — the largest baskets, the biggest portfolios. The mean also hides bimodal behaviour completely: cache hits at 5 ms and misses at 900 ms average out to something no request ever experiences.
Can I average percentiles from several servers?
No. The mean of each node's p99 is not the p99 of the whole service and can be badly wrong. Export histogram buckets from each instance, sum the buckets centrally, and compute the percentile from the merged histogram. Most metric systems support this; if yours reports pre-computed quantiles per instance, they cannot be aggregated meaningfully.
How large should a database connection pool be?
Much smaller than most people assume. Connections beyond what the database can execute concurrently just queue, adding latency without throughput, and a large pool lets the application overwhelm the database. Start from the database's parallel capacity — roughly the core count plus effective spindle count — size the pool near that, and verify by raising it and observing that throughput does not improve while latency worsens.
When should I add a cache?
After you have measured that the same expensive result is being computed repeatedly and you can state an acceptable staleness. A cache is a correctness risk and an operational dependency, so it should follow a measurement, not a hunch. Fixing an N+1 query or adding an index frequently removes the need for the cache entirely, and costs nothing in consistency.