Software Architecture

Load Testing Methodology: Workload Models, Test Types and Reading Results

Intermediate10 min readUpdated

A load test is an experiment, and most load tests fail as experiments long before they fail as software. This guide covers the distinct test types and what each answers, how to build a workload model from real traffic, how much environment fidelity you actually need, which metrics to collect on both sides, and how to read a result to find the saturation point rather than a number to put in a slide.

What you will learn

  • Choose the right test type — load, stress, soak, spike, capacity — for the question being asked
  • Build a workload model from production evidence rather than from assumptions
  • Design ramp-up, think time and test data so that the test resembles real traffic
  • Collect the client-side and server-side metrics needed to interpret a run
  • Identify the saturation point and distinguish a real bottleneck from a test artefact

Most load testing produces a number rather than an answer. A team runs a script, records that the system handled 1 200 requests per second, puts it in a report, and learns nothing that changes a decision. The problem is rarely the tool; it is that the test was never framed as a question. "Can we survive the sale on Friday?" and "why does p99 degrade after two hours?" require completely different experiments, and neither is answered by a single throughput figure.

Treat a performance test as an experiment: a hypothesis, controlled conditions, measurements on both sides of the wire, and an analysis that either supports or refutes the hypothesis. This guide covers building that experiment. Diagnosing what the results reveal — profiling, GC, database access — is the subject of application performance.

Test types and the questions they answer

The types differ in load shape and duration, and using the wrong one produces a confident wrong answer.

TypeLoad shapeQuestion it answers
SmokeVery low, minutesDoes the script work and the environment respond at all?
LoadExpected peak, steadyDo we meet our objectives at the traffic we expect?
StressIncreasing until failureWhere does it break, and how does it break?
CapacityStepped increasesWhat is the maximum sustainable throughput, and what limits it?
Soak (endurance)Moderate, hours to daysDoes anything degrade over time — leaks, fragmentation, log growth?
SpikeSudden jump and dropDoes it survive a flash crowd, and does it recover?
ScalabilityLoad per unit of resourceDoes doubling instances double throughput?

Two are consistently undervalued. Soak tests find the failures that only appear with time: a slowly growing collection, connection leaks, a cache that never evicts, disks filling with logs. These are the failures that take production down at 3 a.m. on day four, and no short test can find them. Stress tests matter less for the breaking number than for the breaking behaviour — a system that sheds excess load with fast 429 responses and recovers when the load subsides is enormously better than one that queues until it exhausts memory, even if both break at the same throughput.

Building the workload model

The single largest determinant of whether a test predicts anything is whether the modelled workload resembles the real one. Uniform traffic against one endpoint is easy to script and tells you nothing.

Start from evidence. Access logs, gateway analytics and traces give you the real distribution: which operations, in what proportion, at what rate, with what parameter variety. Take a representative peak hour rather than a daily average, since capacity is decided by peaks.

Model business processes, not endpoints. Real users follow paths — browse, search, view a product, add to basket, check out — with realistic drop-off between steps. A model where 100 % of virtual users check out overstates write load enormously and understates search.

Scenario mix for a retail peak hour (from 30 days of access logs):

  Browse-only ........ 62%   home → category → 3 × product
  Search-driven ...... 24%   search → 2 × product → add to basket (40% of these)
  Checkout ............ 9%   basket → address → payment → confirmation
  Account admin ....... 4%   login → orders → order detail
  API partners ........ 1%   steady 25 req/s, no think time

Peak measured arrival rate: 340 sessions/minute
Read:write ratio: 94:6

Think time is not optional. Real users pause between actions, which changes concurrency for a given arrival rate — and by Little's law, changes how many sessions are needed to produce a given throughput. Removing think time turns 5 000 realistic users into the load of perhaps 200, and produces a test that hammers one code path with unrealistic cache locality. Use a distribution, not a constant: a randomised pause between 3 and 12 seconds is far better than a fixed 5.

Vary the data. Every virtual user requesting the same product ID measures your cache, not your system. Use a realistic key distribution — usually a long tail with some genuinely hot items — and unique data where uniqueness matters, so that the test does not accidentally hit unique constraints or fill one partition.

Model arrival, not just concurrency. Closed-model tools hold N virtual users, each waiting for a response before continuing, which means the offered load automatically falls when the system slows — the coordinated-omission problem, and a flattering one. Open-model tools inject at a specified arrival rate regardless of response time, which is how real internet traffic behaves and how queues actually build. Prefer arrival-rate injection for anything public-facing.

Environment and test data

Perfect fidelity is unaffordable; the useful question is which differences distort the result.

FactorMattersCan often be scaled
Data volume and distributionCritically — plans change with cardinalityRarely; use production-sized data
Application topologyYes — number of hops, load balancing behaviourSame shape, fewer instances
Database sizingYes; it is usually the bottleneckOnly with careful interpretation
Network path and latencyYes for chatty designsSimulate added latency
Third-party dependenciesYesStub with realistic latency, including its tail
CPU generation, disk typeYes for absolute figuresCompare relative results only

Data volume deserves the emphasis. A query plan that uses an index against 10 000 rows may switch to a sequential scan at 10 million, so a test against a small database validates functionality and nothing about performance. If you cannot copy production data, generate a set with the same volume, cardinality and skew — including the customer with 40 000 orders who breaks every pagination assumption.

Third-party stubs must reproduce the dependency's latency distribution, not its mean. A partner API that answers in 80 ms with an occasional 3-second outlier will expose your timeout and retry behaviour; a stub that always answers in 80 ms will not. Where possible, include a fault mode in some runs.

Finally, scale down with honesty. Results from a quarter-size environment predict production only if the system scales linearly and the database is scaled equivalently, which is exactly what a scalability test is for. State the scaling assumption in the report rather than leaving the reader to assume fidelity you did not have.

Writing the test

Three tools dominate, and the choice matters less than the model behind it.

JMeter is mature, GUI-driven with an XML test plan, has an enormous protocol and plugin ecosystem, and is familiar to test teams. Its threads map to virtual users, so very high concurrency needs distributed injectors, and version control of the XML plan is unpleasant but workable.

Gatling expresses scenarios in a Scala or Java DSL, uses an asynchronous engine so a single injector sustains high concurrency, and produces good HTML reports. It suits teams comfortable in a JVM language who want tests reviewed like code.

k6 uses JavaScript, is scriptable and diff-friendly, has first-class support for arrival-rate executors and thresholds that fail a run automatically, and fits pipelines well.

// k6: open model — inject a fixed arrival rate regardless of response time.
import http from 'k6/http';
import { check, sleep } from 'k6';
import { Trend } from 'k6/metrics';

const checkoutLatency = new Trend('checkout_latency', true);

export const options = {
  scenarios: {
    browse: {
      executor: 'ramping-arrival-rate',
      startRate: 20, timeUnit: '1s',
      preAllocatedVUs: 400, maxVUs: 2000,
      stages: [
        { target: 120, duration: '5m' },   // ramp
        { target: 120, duration: '20m' },  // steady state — this is what we measure
        { target: 0,   duration: '2m' },   // ramp down
      ],
    },
  },
  thresholds: {
    'http_req_failed': ['rate<0.005'],                 // fail the run above 0.5% errors
    'checkout_latency': ['p(95)<800', 'p(99)<2000'],
  },
};

const products = JSON.parse(open('./product-ids.json'));   // realistic key distribution

export default function () {
  const id = products[Math.floor(Math.random() * products.length)];

  const home = http.get(`${__ENV.BASE}/`, { tags: { name: 'home' } });
  check(home, { 'home 200': (r) => r.status === 200 });
  sleep(Math.random() * 9 + 3);                             // think time 3–12 s

  const product = http.get(`${__ENV.BASE}/products/${id}`, { tags: { name: 'product' } });
  check(product, { 'product 200': (r) => r.status === 200 });
  sleep(Math.random() * 6 + 2);

  if (Math.random() < 0.09) {
    const res = http.post(`${__ENV.BASE}/orders`,
      JSON.stringify({ sku: id, quantity: 1 }),
      { headers: { 'Content-Type': 'application/json', 'X-Synthetic': 'loadtest' },
        tags: { name: 'checkout' } });
    checkoutLatency.add(res.timings.duration);
    check(res, { 'order created': (r) => r.status === 201 });
  }
}

Note the tags on every request: without per-operation tagging, an aggregate p95 mixes a 20 ms static asset with a 900 ms checkout and describes neither. Note also the X-Synthetic header, which lets the platform exclude test traffic from business metrics.

Whichever tool you use, keep test scripts in the repository beside the application, review them, and parameterise the environment. A load test that only one person can run from their laptop stops being run.

Metrics to collect

Client-side numbers tell you what users would experience; server-side numbers tell you why. You need both, on the same clock.

From the generator: arrival rate achieved versus requested (a gap means the generator is saturated and every other number is suspect), response time percentiles per operation, error rate broken down by status and by cause, connection errors and timeouts separately from HTTP errors, and data transferred.

From the system: the signals exposed by Actuator and observability — request rate, error rate and latency percentiles per service and endpoint; CPU, memory, disk and network per instance; thread pool active counts and queue depths; connection pool usage and acquisition wait time; GC pause time and frequency; database transactions per second, lock waits, slow query counts and cache hit ratios; and queue depths on any broker.

Also check the generator itself. A load injector at 95 % CPU, out of ephemeral ports, or with its own GC pauses reports latency that is entirely its own. Monitor it as carefully as the system under test, and distribute it once one machine is no longer comfortable.

Reading the results

Plot throughput, response time and error rate against offered load on the same time axis, and the shape tells the story.

As load increases, throughput rises roughly linearly while response time stays flat: the system has spare capacity. Then throughput growth slows and response time begins climbing — the knee, where a resource is approaching saturation and queues start forming. Push further and throughput plateaus or falls while response time rises steeply and errors appear: saturation, and beyond it collapse, where throughput drops as the system spends its effort on timeouts, retries and context switching.

The number worth reporting is the highest offered load at which the system still meets its objectives — not the peak throughput observed, which usually occurs at a latency nobody would accept. Then answer the follow-up: what limited it? A saturated CPU, a full connection pool, a lock, a downstream service, or the generator. If you cannot name the limiting resource, the test has produced a number rather than an answer.

A few patterns and their usual meanings:

  • Latency rises while CPU stays low — waiting on something: a pool, a lock, a downstream call.
  • Throughput plateaus well below expectations with a healthy-looking system — an artificial concurrency cap: thread pool size, connection pool, a semaphore, or the generator's own limit.
  • Errors appear only above a threshold — a queue overflowing or a timeout expiring; find which and whether it is the correct behaviour.
  • Response times degrade gradually over hours at constant load — a leak, unbounded growth, or index bloat. Extend to a soak test.
  • A step change after several minutes — a cache filling, a batch job starting, or JIT compilation completing; check what else runs on that schedule.

Compare against a baseline from the same environment rather than against an absolute target from another system. And repeat: three runs of the same configuration establish whether a 7 % difference is a regression or noise.

Common mistakes

Testing a single endpoint. It measures one code path with perfect cache locality and predicts nothing about mixed production traffic.

No think time. Concurrency and cache behaviour are both wrong, in the direction that flatters the system.

Reporting the average. As with any latency work, the average conceals exactly the behaviour that matters.

A tiny or unrealistic dataset. Query plans differ, caches hold everything, and skew is absent.

Ignoring warm-up. Measuring during JIT compilation, cold pools and empty caches produces pessimistic numbers that hide the steady-state problem.

Not resetting between runs. Data grows, caches stay warm, and each run measures a different system.

Ignoring the client. A saturated generator makes an unhealthy system look healthy.

Testing only the happy path. No slow dependency, no partial failure, no retry storm — so the resilience behaviour that decides whether an incident becomes an outage is never exercised.

Running it once, before go-live. Performance regresses with every release. A short automated test in the delivery pipeline plus a scheduled full test is what keeps the result true.

Key takeaways

  • Frame every test as a question, and pick the type — load, stress, capacity, soak, spike — that answers it; soak and stress reveal the failures that hurt most.
  • The workload model decides the value of the result: real operation mix, business scenarios with drop-off, randomised think time, varied data and arrival-rate injection.
  • Data volume and distribution matter more than most other environment differences; stub third parties with a realistic latency distribution, not a constant.
  • Tag every request by operation, keep scripts in version control, and record version, configuration and dataset with every run.
  • Collect client-side and server-side metrics together, and always verify the load generator is not itself the bottleneck.
  • Report the highest load that still meets the objectives and name the limiting resource; a throughput figure without a cause is not a finding.

Sources and further reading

Frequently asked questions

Can I load test in production?
Many organisations do, and it is the only environment with genuine fidelity. It requires safeguards: run during low traffic, mark synthetic requests with a header so they can be excluded from business metrics, cap the injected rate, have an immediate kill switch, and make sure test traffic cannot create real orders or send real emails. Start with a small share of production capacity and increase slowly.
How long should a load test run?
Long enough to pass warm-up and reach steady state, which for a JVM service means at least ten to fifteen minutes of measurement after ramp-up so that JIT compilation, connection pools and caches have settled. Anything shorter measures start-up. Soak tests are a different question entirely and need hours to days to reveal leaks and fragmentation.
Why do my results differ every run?
Usually because something is not controlled. Common culprits are a shared environment with other traffic, a database whose data volume grows across runs, caches warm from a previous test, noisy virtualised neighbours, and a load generator that is itself saturated. Reset data between runs, check generator CPU and network, and repeat each configuration at least three times before believing a difference.
Should load tests run in the CI pipeline?
A short smoke-level performance test on every build is valuable — a few minutes at modest load, failing on a clear regression threshold, which catches the accidental N+1 query before it merges. Full capacity and soak tests belong on a schedule against a dedicated environment, because they need time, stable infrastructure and a human to interpret the result.