Software Testing & Quality
Performance Testing as a Discipline: SLOs, JMH Benchmarks and CI Regression Gates
Intermediate9 min readUpdated
Performance work fails when it is a phase at the end of a project. This guide treats it as part of the test strategy: writing non-functional requirements that can fail a build, measuring code correctly with JMH, profiling a slow test instead of guessing, and running regression gates in a pipeline where the numbers are noisy.
What you will learn
- Turn a vague performance expectation into an SLO that a test can pass or fail
- Place performance checks at the right level, from microbenchmark to full load test
- Write JMH benchmarks that survive JIT warm-up and dead-code elimination
- Profile a slow test or endpoint with sampling profilers and JFR rather than guessing
- Automate performance regression detection in CI despite noisy measurements
Performance problems have a characteristic shape: nobody notices for months, then a release makes a page four times slower and the cause is buried in three hundred commits. The reason is almost never that the team lacked a load-testing tool. It is that performance was treated as an activity that happens near the end, on a different environment, run by different people, against requirements written as adjectives.
Treating it as part of the test strategy changes that. Performance expectations become numbers that can fail a check, measurements happen at several levels of granularity, and a regression is caught by the pipeline that produced it. This guide covers the discipline side — objectives, levels, benchmarking, profiling, and CI gates. For workload modelling, ramp profiles and the mechanics of running a load test at scale, see the load testing methodology guide.
Requirements you can fail a build on
"The system must be fast" cannot be tested. A statement that can needs four parts: which operation, which statistic, which threshold, and under what conditions.
The
POST /ordersendpoint responds in under 400 ms at the 95th percentile and under 900 ms at the 99th, at a sustained 150 requests per second, on two 2-vCPU instances, with a database holding 5 million orders.
That is a service level objective, and it is falsifiable. Note what each part contributes. Naming the operation stops an average across all endpoints from hiding one slow one. Choosing a percentile acknowledges that users experience the tail: an average of 120 ms is entirely compatible with one request in fifty taking four seconds. Stating the load makes the number reproducible. Stating the environment and data volume prevents the classic embarrassment of a system that is fast against ten thousand rows and quadratic against ten million.
Three further quantities belong alongside latency, and are often the ones that actually break: throughput (requests handled per second before latency degrades), resource headroom (CPU, memory, connection pool utilisation at the target load), and error rate under stress (what fraction of requests fail when the system is pushed past its comfortable point, and whether it degrades or collapses).
Where performance checks sit in the strategy
Performance is not one activity but a family, at different scopes and costs.
| Level | What it measures | Cost | Runs |
|---|---|---|---|
| Microbenchmark (JMH) | One method or algorithm, nanoseconds to microseconds | seconds to minutes | On demand, or per commit for hot paths |
| Component benchmark | One service call end to end, in-process | seconds | Per commit |
| Assertion on a functional test | Query count, allocation count, wall time bound | free | Every build |
| Load test | The deployed system under a modelled workload | minutes to hours | Nightly, per release |
| Soak and stress | Stability over hours, behaviour past the limit | hours | Per release |
| Production observation | Real traffic, real data | continuous | Always |
The cheapest and most under-used entry in that table is the third. Many performance regressions are not subtle: a query inside a loop, a full table scan, an accidental O(n²) comparison. Those can be caught by a functional test that asserts on a countable property rather than a duration:
@Test
void loadingAnOrderPageIssuesAFixedNumberOfQueries() {
QueryCounter counter = QueryCounter.attachTo(dataSource);
orderPageService.render(customerWith(50).orders());
assertThat(counter.count())
.as("N+1 regression: one query per order has crept back in")
.isLessThanOrEqualTo(3);
}
Counting queries, allocations or HTTP calls is deterministic, so the assertion never flakes — unlike a wall-clock bound, which will fail on a loaded CI runner sooner or later. Any time you fix an N+1 problem, leave a counting test behind so it cannot return silently.
Microbenchmarks with JMH
When the question is "is this parsing routine faster than that one", you need a benchmark harness, because a hand-written timing loop measures the JIT compiler's opinion of your code rather than the code.
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@State(Scope.Benchmark)
@Fork(value = 2, jvmArgsAppend = {"-Xms1g", "-Xmx1g"})
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 10, time = 1)
public class IbanValidationBenchmark {
@Param({"FR7630006000011234567890189", "DE89370400440532013000"})
public String iban;
private IbanValidator regexValidator;
private IbanValidator arithmeticValidator;
@Setup(Level.Trial)
public void setUp() {
regexValidator = new RegexIbanValidator();
arithmeticValidator = new ArithmeticIbanValidator();
}
@Benchmark
public boolean regex() {
return regexValidator.isValid(iban);
}
@Benchmark
public boolean arithmetic() {
return arithmeticValidator.isValid(iban);
}
@Benchmark
public void consumesExplicitly(Blackhole blackhole) {
blackhole.consume(arithmeticValidator.normalise(iban));
}
}
The annotations are not ceremony; each one defends against a specific way the measurement can lie.
@Warmup discards the iterations during which the method still runs interpreted or is being recompiled. Without it, you measure start-up.
@Fork runs the benchmark in fresh JVMs. Profile pollution is real: running benchmark A before benchmark B in the same JVM can leave the compiler with a polymorphic call site that makes B look slower than it is. Two forks also expose run-to-run variance that a single JVM hides.
Returning a value or using Blackhole prevents dead-code elimination. A benchmark whose result is unused can be optimised away entirely, and the classic symptom is a routine that appears to take 0.3 nanoseconds — less time than a single memory access, which is a physical impossibility rather than a great optimisation.
@State controls what is shared and when it is created. Inputs must not be compile-time constants, or the optimiser will constant-fold the whole computation; @Param supplies them at runtime and, usefully, runs the whole matrix.
Run it from a separate module (mvn clean verify then java -jar target/benchmarks.jar) rather than from an IDE, and read the output including the error margin — JMH reports a confidence interval, and an improvement smaller than that interval is not an improvement.
Even a correct benchmark can mislead. It measures a method in isolation, warmed up, with a hot cache and no contention. In production that method may run once per request on a cold cache while forty threads compete for memory bandwidth. A 40% improvement in a routine that accounts for 0.2% of request time is worth nothing at all. Always check the profile before optimising, and re-check the end-to-end number afterwards.
Profiling instead of guessing
When something is slow, the instinct is to inspect the code and form a theory. Theories about performance are usually wrong, because the cost sits somewhere unglamorous — a logging call that formats a string on every iteration, a regular expression recompiled per request, a synchronized block that serialises eight threads.
Sampling profilers interrupt threads periodically and record stacks. They have low overhead and give you a flame graph where width is time. This is the right first tool for "where does the time go" and it works on a running production process as well as a test.
JDK Flight Recorder is built into the JVM and can be enabled on a single test run:
java -XX:StartFlightRecording=duration=60s,filename=slow-test.jfr,settings=profile \
-jar target/app.jar
The recording contains allocation sites, garbage collection pauses, lock contention, exceptions thrown, thread states and JIT activity. Opening it shows things a stack profiler alone will not: that 30% of the wall time is a GC pause, or that a lock is held for milliseconds in a hot path.
Allocation profiling is the one most often skipped and most often decisive. Excessive allocation does not look like a bottleneck in a CPU profile; it shows up as garbage collection pressure and cache misses. If a request allocates two hundred megabytes, no amount of algorithmic tuning will fix the symptom.
For a slow test, the same tools apply, plus a simpler question first: is the test slow because the code is slow, or because the test starts a container, builds an application context, or sleeps? In an integration suite the answer is almost always the second, and it is a suite problem rather than a performance problem — see the cost-management section of the integration testing guide.
Memory, leaks and long-running behaviour
Some failures only appear over time. A cache without an eviction policy, a thread-local never cleared, a listener list that only ever grows, a connection pool leaking under a specific error path — none of these fail a functional test, and all of them fail a production deployment on day four.
Two automatable checks catch the majority:
@Test
void repeatedRenderingDoesNotRetainDocuments() {
ReferenceQueue<RenderedDocument> queue = new ReferenceQueue<>();
List<PhantomReference<RenderedDocument>> refs = new ArrayList<>();
for (int i = 0; i < 200; i++) {
RenderedDocument doc = renderer.render(sampleInvoice());
refs.add(new PhantomReference<>(doc, queue));
}
System.gc();
await().atMost(Duration.ofSeconds(5)).until(() -> drain(queue) >= 190);
}
A soak test — the system held at a moderate load for several hours while heap use, thread count, file descriptors and connection counts are recorded — catches the rest. The signal is not the absolute value but the slope: a heap floor that rises steadily after each garbage collection is a leak, whatever the peak looks like. Heap dumps taken at two points and compared by class histogram will normally name the culprit in minutes.
Automating regression detection in CI
The difficulty with performance in a pipeline is that the measurement is noisy. Shared runners have neighbours, CPU frequencies vary, container limits differ. A naive gate — "fail if slower than 400 ms" — either fires constantly or is set so loose it never fires.
Techniques that make it workable:
Compare, do not assert absolutes. Run the baseline and the candidate in the same job, interleaved, and compare them. Shared noise affects both, so the ratio is far more stable than either number.
Gate on a wide threshold, alert on a narrow one. Fail the build only on a large regression — say 25% — and raise a non-blocking report for anything above 10%. This keeps the pipeline usable while making drift visible.
Track the trend, not the run. Store each measurement with the commit and plot it. Performance rarely regresses in one dramatic step; it degrades 2% at a time over forty commits, and only a trend line shows that. A step change on the chart names the commit that caused it.
Prefer deterministic proxies where you can. Query counts, allocation counts and object sizes do not vary with machine load, so they can be gated tightly without flakiness.
Isolate what you can. Pin CPUs, disable frequency scaling, use a dedicated runner for benchmarks. Where that is impossible, increase the number of repetitions and use the median rather than the mean.
performance:
stage: verify
script:
- java -jar target/benchmarks.jar -rf json -rff target/jmh-result.json
- ./scripts/compare-benchmarks.sh baseline/jmh-result.json target/jmh-result.json --fail-over 25 --warn-over 10
artifacts:
paths: ["target/jmh-result.json"]
Whatever the gate, record the result even when it passes. The archive of measurements is what lets you answer "when did this get slow", and that question is asked far more often than "is this build slow", as the continuous integration guide argues for build duration generally.
When the numbers mislead
A short list of ways performance results deceive people, most of which have cost somebody a week:
- Measuring a warmed-up JVM and deploying a cold one. If instances restart frequently, start-up and warm-up time is part of the user experience.
- Testing with a tiny dataset. Everything is fast against a thousand rows. Index behaviour, join plans and cache hit rates all change with volume.
- Load generated from one machine that saturates first. If the generator's CPU is at 100%, you are measuring the generator.
- Ignoring coordinated omission. A load tool that waits for a response before sending the next request under-reports latency exactly when the system is struggling, because the requests that would have been slow were never sent.
- Optimising the wrong 3%. Profile first, always, and confirm the improvement end to end afterwards.
- A single run. One number is not a measurement. Repeat, report variance, and be suspicious of any effect smaller than it.
Key takeaways
- Performance requirements only become testable as SLOs: named operation, percentile, threshold, load and environment.
- Put performance checks at several levels — deterministic query and allocation counts in functional tests, JMH for hot code, load tests for the system.
- JMH's warm-up, forking and blackholes each defend against a specific measurement lie; a hand-rolled timing loop reproduces none of them.
- Profile before optimising with a sampling profiler and JDK Flight Recorder, and look at allocation as well as CPU.
- Catch leaks with soak runs and watch the post-GC heap floor rather than the peak.
- In CI, compare against a baseline in the same job, gate wide, alert narrow, and store every measurement so the trend can name the commit that regressed.
Sources and further reading
Frequently asked questions
- Why can I not just time a loop with System.nanoTime?
- Because the JVM will outwit you. The first thousand iterations run interpreted before the JIT compiles the method, so an untimed warm-up is mandatory. Once compiled, the optimiser may hoist your call out of the loop, constant-fold it, or delete it entirely because the result is unused. Garbage collection and CPU frequency scaling add noise larger than most effects you are trying to measure. JMH exists to handle all of that.
- Should performance tests run on every commit?
- Full load tests, no — they need a production-like environment and take too long for a pull request. A small set of targeted benchmarks on hot code paths can run per commit if the runner is stable enough, and a nightly load run against a dedicated environment catches the rest. The important thing is that some performance signal arrives within a day of the change that caused a regression, while the diff is still small.
- What is the difference between an average and a p99, and which should I use?
- The average is dominated by the common case and hides the tail; p99 means 1% of requests were slower than that value. Users experience the tail, and in a page that makes twenty backend calls a p99 on each call affects a large share of page loads. Set objectives on percentiles — typically p95 and p99 — plus a maximum, and treat the average as a sanity check rather than a target.
- My benchmark says the new code is 15% faster. Should I believe it?
- Only if the difference is large relative to the run-to-run variance and you can reproduce it on repeated runs, ideally on a quiet machine or bare metal. Report the confidence interval JMH gives you, run both versions interleaved rather than one after the other, and be sceptical of any improvement under about 10% on a shared CI runner. And confirm the microbenchmark reflects a path that actually matters in production.