Software Testing & Quality
Software Testing Fundamentals: Levels, Techniques and the Vocabulary Teams Argue About
Beginner10 min readReviewed by Mariem Ouled on Updated
Testing is a design activity long before it is a tooling one. This guide sets out what testing can and cannot prove, the levels a change passes through, the formal techniques that turn an infinite input space into a finite set of cases, and the shared vocabulary that keeps discussions about defects from turning into arguments about words.
What you will learn
- Explain what testing can demonstrate and what it can never prove
- Distinguish the unit, integration, system and acceptance levels and what each one is good at
- Apply equivalence partitioning, boundary values, decision tables and state transition testing to derive cases
- Use error, defect and failure, and verification and validation, in their precise senses
- Judge the test pyramid on its merits and recognise where its critics have a point
Every non-trivial program has behaviour its authors never considered. Testing is how a team finds some of that behaviour before customers do, and — more importantly for the people maintaining the code — how it keeps the freedom to change things later. A suite that fails loudly when a refactor breaks a rule is worth more than a suite that merely proves the code worked once, because the first one keeps paying out for years.
The tooling side of testing is easy to pick up and easy to over-value. Knowing how to write a JUnit assertion tells you nothing about which assertions are worth writing. This guide covers the part that comes first: what testing can establish, how work is split across levels, how to derive cases systematically instead of guessing, and the small vocabulary that lets a team discuss a defect without three people meaning three different things.
What a test can and cannot establish
A test executes a program with chosen inputs in a chosen state and compares the observed result with an expected one. Because the input space of even a modest service is astronomically large — every string, every date, every ordering of concurrent requests, every partial failure of a dependency — exhaustive testing is not a thing that exists. What you actually build is a sample, chosen to maximise the chance of exposing failures per unit of effort.
Two practical consequences follow. First, a green suite is not proof of correctness; it means the failures you thought to look for did not happen under the conditions you created. Second, the value of the sample depends almost entirely on how it was chosen. A thousand tests over the same happy path are worth less than twenty tests that each probe a different boundary, an error branch, or an assumption about a collaborator.
This is also why the timing of a test matters as much as its content. The cost of a defect grows with the distance between the moment it is introduced and the moment it is found. A mistake caught by the compiler costs seconds. Caught by a unit test on the developer's machine, minutes. Caught in a pipeline, an hour of context-switching. Caught in production, it costs an incident, a hotfix, a rollback plan, possibly data repair, and the attention of people who were doing something else. The exact multipliers quoted in the literature vary wildly and should be treated with suspicion, but the shape of the curve is not controversial and it justifies pushing detection as early as it can go.
Words that need to mean one thing
Most heated conversations about testing are really conversations about definitions. Four distinctions are worth fixing.
Error, defect, failure. A person makes an error (a mistake in thinking). That error may leave a defect in an artefact — code, a configuration file, a requirement. When the defective code executes under conditions that expose it, you get a failure: observable behaviour that differs from what was expected. Not every defect causes a failure, and a failure can also come from environment or data rather than from a defect at all. Being precise here saves time in incident reviews: "we had a failure" and "we had a defect" call for different investigations.
Verification and validation. Verification checks the product against its specification. Validation checks the specification against the need. Automated tests are overwhelmingly verification; demos, acceptance sessions and usability testing are where validation happens.
Testing and quality assurance. Testing measures a product. Quality assurance is the wider set of practices — reviews, standards, static analysis, definitions of done — that make defects less likely in the first place.
Test case, test suite, test run. A case is one specification of inputs, preconditions and expected result. A suite groups cases. A run is one execution of a suite against one build in one environment. Reporting "the test failed" without saying which run, on which build, is how teams chase ghosts.
The levels a change passes through
Levels describe the scope of what is under test, not the tool used.
| Level | Scope | Typically written by | Answers |
|---|---|---|---|
| Component (unit) | One class or small cluster, isolated from slow collaborators | Developers | Does this logic behave as specified? |
| Integration | Two or more components, or a component and a real external system | Developers | Do these parts agree on their contract? |
| System | The deployed application as a whole | Developers and testers | Does the assembled product meet its requirements? |
| Acceptance | The product in a business context | Product people, users | Is this what we needed, and can we release it? |
The boundaries between them are conventions, not laws, and teams argue endlessly about whether a test that starts a database is "unit" or "integration". The argument is usually pointless; what matters is that each test has a clearly identified subject and a failure that points somewhere specific. A test whose failure could mean any of nine things is a badly scoped test regardless of what you call it. The unit testing guide and the integration testing guide go into how each is built in Java.
Acceptance testing has sub-varieties worth naming: user acceptance testing by the people who will use the system, operational acceptance covering backup, restore, monitoring and runbooks, contractual or regulatory acceptance where an external standard must be demonstrated, and alpha/beta testing where real users exercise a pre-release build.
Functional and everything else
Functional testing asks whether the system does the right thing. Non-functional testing asks how well it does it, and it is where most production incidents actually originate. The main families:
- Performance efficiency — response time, throughput, resource use under load. Covered in the performance testing guide.
- Reliability — behaviour under failure of a dependency, recovery after a crash, data durability.
- Security — authentication, authorisation, injection, secrets handling, dependency vulnerabilities.
- Usability and accessibility — can a person complete the task, including with a screen reader or keyboard only.
- Compatibility — browsers, devices, database versions, older API clients.
- Maintainability and portability — qualities measured on the code and the deployment rather than at runtime.
Non-functional requirements are frequently written as adjectives ("the system must be fast"), which cannot be tested. Turning them into numbers with a context — "95th percentile latency under 300 ms for the search endpoint at 200 concurrent users on the standard production instance size" — is the single highest-leverage thing a team can do for non-functional quality, because it converts an opinion into a pass/fail check.
Black box, white box, and the space between
Black-box (specification-based) testing derives cases from what the system is supposed to do, with no reference to how it is written. It is the only kind possible for a third-party component and it survives refactoring untouched, which is why most valuable tests are black-box with respect to their subject.
White-box (structure-based) testing derives cases from the code: statements, branches, conditions, paths. Its measure is coverage — the fraction of statements or branches executed. White-box thinking is what tells you a test suite never exercises the else branch of a validity check.
Experience-based techniques fill the gaps that neither covers: error guessing (probing the places defects historically live — empty collections, zero, unicode, daylight-saving transitions), exploratory testing (simultaneous learning, design and execution in time-boxed sessions), and checklists built from past incidents.
Deriving cases instead of guessing
Formal test design techniques exist to turn an infinite input space into a small set of cases with an argued justification. Four cover most day-to-day needs.
Equivalence partitioning. Divide each input into classes whose members should be treated identically, then test one member per class. For an age field accepting 18–65, the partitions are "below 18", "18 to 65" and "above 65", plus invalid classes: non-numeric, empty, absent. Three or four values replace millions.
Boundary value analysis. Defects cluster at the edges of partitions, because that is where < and <= get confused. For each boundary, test the value on it and the values immediately either side: 17, 18, 19 and 64, 65, 66. This is the highest yield technique per test written, which is why it appears in almost every specification.
Decision tables. When behaviour depends on a combination of conditions, tabulate them. A shipping rule based on order value, customer tier and destination has a finite set of condition combinations; writing them out exposes the combinations nobody specified.
| Order ≥ 50 | Premium member | Domestic | Shipping |
|---|---|---|---|
| yes | any | yes | free |
| yes | yes | no | free |
| yes | no | no | flat rate |
| no | yes | yes | free |
| no | no | any | standard rate |
A table with four independent binary conditions has sixteen rows, which is usually the point where somebody notices the rule is too complicated. Where the full combinatorial set is impractical, pairwise testing covers every pair of parameter values in far fewer cases and catches the majority of interaction defects.
State transition testing. For anything with a lifecycle — an order, a payment, a subscription — draw the states and the permitted events. Then test each valid transition, and, more importantly, the invalid ones: cancelling a shipped order, paying an already-refunded invoice. Invalid transitions are where the interesting defects hide because the specification rarely mentions them.
The pyramid and its critics
The test pyramid is a heuristic about suite composition: many fast, isolated tests at the bottom, fewer integration tests in the middle, a handful of end-to-end tests at the top. Its justification is economic — tests at the bottom run in milliseconds, fail for one reason and rarely flake, while tests at the top are slow, brittle and vague when they fail.
The critics make two fair points. First, the shape is often applied to the wrong axis: what matters is not the framework used but the isolation and speed of the test, and a "unit" test that mocks every collaborator can pass while the assembled system is broken. Second, in a service that is mostly glue between a database and an HTTP client, there is very little pure logic to unit test, and a middle-heavy shape (sometimes drawn as a trophy or a honeycomb) reflects reality better.
Both critiques are about the specific ratios, not about the underlying trade-off, which nobody disputes: broader tests find more kinds of problem and cost more to run and maintain. The automation strategy guide works through how to choose a shape for a given system rather than inheriting one from a diagram.
From defect found to defect closed
A defect report is a communication artefact, and a bad one wastes more time than the defect. The essentials: what was done, what happened, what was expected, on which build and environment, with logs or a screenshot and the smallest sequence that reproduces it. Two attributes are then assigned and frequently confused. Severity is the technical impact — data loss is high severity regardless of who noticed. Priority is the business urgency of fixing it. A cosmetic misalignment on the sign-up page can be low severity and high priority; a crash in a feature three people use can be the reverse.
The lifecycle is then a small state machine: new, triaged, assigned, fixed, ready for retest, verified, closed — with side branches for rejected, duplicate and deferred. Two rules keep it honest: only the person who reported the defect (or the test that caught it) closes it, and every fixed defect gets a regression test so the same failure cannot return silently.
Knowing when to start and when to stop
Entry criteria state what must be true before a test phase begins — the build deploys, smoke tests pass, test data is loaded, the environment matches the target configuration. Their purpose is to stop a team from burning days testing a build that was never going to work.
Exit criteria state what must be true to declare it finished. Useful ones are specific and measurable: all planned cases executed, no open high-severity defects, coverage on new code above an agreed line, all non-functional thresholds met, known open defects documented and accepted. Useless ones are "testing is complete" or "we ran out of time" dressed up. In a continuous delivery pipeline these criteria stop being a phase gate and become automated quality gates, which is the same idea enforced by a machine rather than a meeting — the continuous integration guide shows what that looks like in practice.
Key takeaways
- Testing samples behaviour; it demonstrates the presence of failures and can never prove their absence, so how cases are chosen matters far more than how many there are.
- Defects get dramatically more expensive the later they are found, which is the entire economic argument for automated tests running early and often.
- Levels describe scope — component, integration, system, acceptance — and each test should have one clearly identified subject so that a failure points somewhere specific.
- Non-functional requirements only become testable when they are written as numbers with a context; adjectives cannot fail a build.
- Equivalence partitioning, boundary values, decision tables and state transitions turn an infinite input space into a defensible set of cases; boundaries repay the effort fastest.
- Precise vocabulary — error versus defect versus failure, verification versus validation, severity versus priority — makes triage faster and arguments shorter.
Sources and further reading
Frequently asked questions
- Can testing prove that software has no bugs?
- No. Testing samples the behaviour of a program on chosen inputs, and the space of possible inputs, states and timings is effectively infinite. A passing suite shows that the failures you looked for did not occur under the conditions you created. That is genuinely valuable evidence, but it is evidence of presence, never proof of absence, which is why coverage numbers and green builds are signals rather than guarantees.
- What is the difference between verification and validation?
- Verification asks whether the product was built according to its specification: does the code do what the design said it would? Validation asks whether that specification solves the user's actual problem. A feature can pass every verification check and still fail validation because the requirement was wrong. Unit and integration tests mostly verify; user acceptance testing, demos and usability sessions validate.
- Are quality assurance and testing the same thing?
- Testing is one activity inside quality assurance. QA covers everything that shapes how defects are prevented and detected: requirements review, coding standards, pair programming, static analysis, definition of done, retrospectives and process improvement. Testing measures the product; QA works on the process that produces it. A team can test heavily and still have poor quality assurance if nothing changes when defects recur.
- How many test cases are enough?
- There is no absolute number. Derive cases from the specification with a technique such as equivalence partitioning and boundary value analysis, add cases for known risky areas and past defects, and stop when new cases stop changing your confidence. Then measure the residual risk rather than the count: which requirements are untested, which code paths never execute, which failure modes were never simulated.