DevOps, CI/CD & Containers

Docker Compose: Multi-Container Apps, Networking and Volumes

Intermediate9 min readUpdated

Compose describes a whole application — the service, its database, a message broker, a reverse proxy — in one file that any developer can start with a single command. This guide covers the file format as it stands in Compose v2, then goes deep on the two areas that cause the most confusion: how containers find each other on a Compose network, and how data survives when containers do not.

What you will learn

  • Write a compose.yaml describing several services with build, image, ports and environment
  • Explain how the project network and embedded DNS let services address each other by name
  • Order startup correctly with healthchecks and depends_on conditions
  • Choose between named volumes, bind mounts and tmpfs, and back a volume up
  • Layer configuration with env files, variable interpolation, profiles and override files
  • Run a Compose stack in a CI job without leaking state between builds

A modern application is rarely one process. A web service needs a database, probably a cache, often a message broker, maybe a reverse proxy and a mail catcher for development. Starting those by hand with docker run means remembering six command lines with the right flags, in the right order, on the right network — which is exactly the kind of knowledge that lives in one person's shell history and nowhere else.

Compose replaces that with a file. compose.yaml describes the services, the networks between them and the storage under them, and docker compose up brings the whole thing to life. This guide assumes you are comfortable with images and containers — the Docker fundamentals guide covers those — and focuses on the file format, the networking model that surprises newcomers, and the storage decisions that determine whether your data survives.

The file, and what changed in v2

Compose v2 is a Go plugin invoked as docker compose (a subcommand, no hyphen). The old Python docker-compose binary is retired, and with it two habits: the top-level version: key is obsolete and produces a warning, and the preferred file name is compose.yaml rather than docker-compose.yml — though all four spellings are still discovered.

Here is a complete stack for a Spring Boot service with PostgreSQL and Redis:

name: shop

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile
      args:
        JAR_FILE: target/app.jar
    image: acme/shop-api:dev
    ports:
      - "8080:8080"
    environment:
      SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/shop
      SPRING_DATASOURCE_USERNAME: shop
      SPRING_DATASOURCE_PASSWORD: ${DB_PASSWORD:?set DB_PASSWORD in .env}
      SPRING_DATA_REDIS_HOST: cache
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/actuator/health"]
      interval: 10s
      timeout: 3s
      retries: 6
      start_period: 40s
    restart: unless-stopped

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: shop
      POSTGRES_USER: shop
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./ops/initdb:/docker-entrypoint-initdb.d:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U shop -d shop"]
      interval: 5s
      timeout: 3s
      retries: 10
    ports:
      - "127.0.0.1:5432:5432"

  cache:
    image: redis:7-alpine
    command: ["redis-server", "--save", "60", "1", "--appendonly", "no"]
    volumes:
      - redisdata:/data

volumes:
  pgdata:
  redisdata:

Start it, watch it, stop it:

docker compose up -d --build
docker compose ps
docker compose logs -f api
docker compose exec db psql -U shop -d shop
docker compose down          # containers and networks; volumes survive
docker compose down -v       # ...and delete the named volumes too

The name: key sets the project name, which prefixes every container, network and volume Compose creates. Without it, Compose uses the directory name — which is why two checkouts of the same repository in differently named folders quietly get separate databases.

Networking: the part that surprises people

When Compose starts a project, it creates a user-defined bridge network called <project>_default and attaches every service to it. Docker runs an embedded DNS server on that network, and each service is resolvable by its service name. So the API reaches PostgreSQL at db:5432 and Redis at cache:6379, with no IP addresses anywhere.

Three consequences follow, and each corresponds to a frequently asked question.

localhost inside a container is the container. An application configured with jdbc:postgresql://localhost:5432/shop will fail, because there is no PostgreSQL in the API container's network namespace. Use the service name.

Container-to-container traffic ignores ports. The ports key publishes a container port on the host, for your browser and your database client. Services on the same network talk on the container port directly, whether or not it is published. This is why the example publishes PostgreSQL as 127.0.0.1:5432:5432 — bound to the loopback interface only, so a laptop on a café network is not offering its development database to the room.

expose publishes nothing. Like EXPOSE in a Dockerfile, it is documentation. Remove it if it confuses people.

You can define additional networks to segment traffic, which is worth doing as soon as the stack has a public edge and a private core:

services:
  proxy:
    image: nginx:1.27-alpine
    ports: ["80:80", "443:443"]
    networks: [edge]
    volumes:
      - ./ops/nginx.conf:/etc/nginx/conf.d/default.conf:ro

  api:
    build: .
    networks: [edge, backend]

  db:
    image: postgres:16-alpine
    networks: [backend]

networks:
  edge:
  backend:
    internal: true      # no route to the outside world

Now the proxy cannot reach the database at all, and the database cannot make outbound connections. That is a real security boundary enforced by the network stack, not a convention.

Two more options come up. aliases gives a service extra DNS names on a network, useful when an application expects a specific hostname such as kafka.internal. And network_mode: host removes network isolation entirely, giving the container the host's interfaces; it is occasionally necessary for network tooling, defeats service discovery, and does not work the same way on Docker Desktop, so avoid it unless you have no choice.

Startup order and healthchecks

depends_on in its short form only controls the order in which containers are created and started. It does not wait for anything inside them to be ready, which is why so many stacks fail on the first up and work on the second.

The long form fixes it, using the dependency's healthcheck:

depends_on:
  db:
    condition: service_healthy
  migrations:
    condition: service_completed_successfully

service_healthy waits until the target's HEALTHCHECK reports healthy; service_completed_successfully waits for a one-shot container — a schema migration, a seed script — to exit zero. A healthcheck therefore does double duty: it tells you the state of a running service in docker compose ps, and it lets other services wait for something meaningful.

Write healthchecks that test the actual dependency, not the process. pg_isready confirms PostgreSQL accepts connections; a TCP check on port 5432 would pass while the database is still recovering. Give slow starters a generous start_period, during which failures do not count towards retries — a JVM service with a 40-second boot will otherwise be marked unhealthy and restarted forever.

Even with all of this, applications should retry their first connection. Compose ordering only covers startup; at runtime a database container can restart at any moment, and a service that dies permanently because its dependency blinked is fragile in every environment, not just this one.

Volumes and where data actually lives

A container's writable layer disappears when the container is removed. Anything that must outlive docker compose down needs a volume. Compose supports three kinds.

A named volume is storage Docker creates and manages under its own directory. It is the right default for databases and other state: it survives container recreation, it can be backed up and moved, and its permissions are handled by the image rather than by your host's user IDs.

volumes:
  - pgdata:/var/lib/postgresql/data

A bind mount maps a host path into the container. Its content is exactly what is on your disk, which makes it ideal for source code you want to edit live, configuration files and read-only seed data:

volumes:
  - ./ops/nginx.conf:/etc/nginx/conf.d/default.conf:ro
  - ./src:/app/src

Add :ro to anything the container has no business writing. Be aware that bind-mounted files keep their host ownership, so a container running as a non-root user may not be able to write to them — the single most common cause of permission errors in a Compose stack. On SELinux systems, append :z (shared) or :Z (private) to relabel the path.

A tmpfs mount puts a directory in host memory, never on disk. Use it for scratch space and for test databases where durability is irrelevant and speed is not:

tmpfs:
  - /tmp
  - /var/lib/postgresql/data:size=512m

Backing up a named volume is a container that mounts it and writes an archive somewhere you control:

docker run --rm \
  -v shop_pgdata:/data:ro \
  -v "$PWD":/backup \
  alpine tar -czf /backup/pgdata-$(date +%F).tar.gz -C /data .

For a database, prefer the database's own dump tool — docker compose exec db pg_dump -U shop shop > shop.sql — because a file-level copy of a running data directory can be inconsistent.

docker compose down deliberately keeps volumes; -v removes them. That flag has deleted a lot of local data that someone assumed was disposable, so know which one your habit is.

Layering configuration

Hard-coding values in compose.yaml makes it useless for anything but one machine. Compose gives you several composable mechanisms.

Variable interpolation reads from the shell and from a .env file next to the Compose file. The syntax supports defaults and required values:

environment:
  LOG_LEVEL: ${LOG_LEVEL:-info}                   # default when unset or empty
  DB_PASSWORD: ${DB_PASSWORD:?DB_PASSWORD is required}
  TAG: ${TAG:-dev}

Do not confuse the .env file (interpolated into the Compose file itself, at parse time) with env_file: on a service (a list of files whose contents become environment variables inside the container). They solve different problems and people reach for the wrong one constantly.

services:
  api:
    env_file:
      - ./config/common.env
      - ./config/local.env      # later files win

Override files let a base definition be specialised. docker compose up automatically merges compose.yaml with compose.override.yaml if it exists. For anything else, name the files explicitly:

docker compose -f compose.yaml -f compose.prod.yaml up -d

The base file describes the application; the override adds development conveniences (bind-mounted source, debug ports, a mail catcher) or production settings (resource limits, restart policies, a pinned image tag instead of a build).

Profiles turn optional services on and off within one file:

services:
  mailhog:
    image: mailhog/mailhog
    profiles: ["dev"]
    ports: ["8025:8025"]
  loadtest:
    image: grafana/k6
    profiles: ["perf"]

Services without a profiles key always start; profiled ones only when requested with --profile dev or COMPOSE_PROFILES=dev. This keeps one file authoritative instead of spawning compose.dev.yaml, compose.demo.yaml and so on.

Finally, YAML anchors and the extends key remove repetition between similar services:

x-service-defaults: &defaults
  restart: unless-stopped
  logging:
    driver: json-file
    options: { max-size: "10m", max-file: "3" }

services:
  api:    { <<: *defaults, image: acme/shop-api:dev }
  worker: { <<: *defaults, image: acme/shop-worker:dev }

The logging options there are worth copying: the default json-file driver has no size limit, and a chatty container on a long-running host will fill the disk with logs.

Compose in a CI job

Compose is an excellent way to stand up integration-test dependencies, provided you deal with two problems: builds running concurrently on the same agent, and leftover state.

Isolate each run with a unique project name, and make clean-up unconditional:

export COMPOSE_PROJECT_NAME="ci-${BUILD_NUMBER}"
trap 'docker compose down -v --remove-orphans' EXIT

docker compose up -d --wait --wait-timeout 120
docker compose run --rm tests

--wait blocks until every service with a healthcheck reports healthy and returns non-zero if one does not, which replaces the sleep loops that used to litter CI scripts. run --rm starts a one-off container using a service definition, attaches to it and returns its exit code — so the CI job's result is the test suite's result.

For a suite defined as a service, --abort-on-container-exit with --exit-code-from achieves the same thing for the whole stack:

docker compose up --abort-on-container-exit --exit-code-from tests

Avoid published host ports in CI entirely: two concurrent builds both binding 8080 will collide. Let the test container reach the service by name on the Compose network instead. The Jenkins pipeline guide shows how the resulting reports get published.

Also consider whether you need Compose at all in the test process itself — for JVM projects, Testcontainers starts the same images programmatically from the test code, which removes the coordination between a shell script and a test suite.

Diagnosing a stack that will not come up

A short sequence resolves most problems:

  1. Validate the merged configuration

    docker compose config prints the fully resolved file with variables substituted and overrides merged. Most "my variable is empty" reports end here.
  2. Look at status and exit codes

    docker compose ps -a shows which services exited and with what code. 137 means the container was killed, usually out of memory; 1 usually means the process failed to start.
  3. Read the logs of the failing service

    docker compose logs --tail=100 db. Database images print clear messages about a data directory that was initialised with different credentials — a classic symptom of a stale named volume.
  4. Test connectivity from inside

    docker compose exec api sh -c 'getent hosts db; nc -zv db 5432' distinguishes a DNS problem from a listener problem from an application configuration problem.
  5. Recreate from scratch

    docker compose down -v && docker compose up --build. If that fixes it, the problem was stale state, and it is worth knowing which volume held it.

Where teams go wrong

  • Assuming depends_on waits for readiness. Add healthchecks and the long form condition.
  • Publishing every port to 0.0.0.0. Bind development databases to 127.0.0.1 and publish only what a browser needs.
  • Editing files inside a running container. The change vanishes on recreation. Change the image or bind-mount the file.
  • Keeping secrets in compose.yaml. The file is committed. Use .env (git-ignored) or Docker secrets, and never reuse production credentials locally.
  • Building images implicitly in production. A production stack should reference an immutable tag built by CI, not build: from whatever source happens to be on the server.
  • Leaving latest in the file. Pin postgres:16-alpine, not postgres. A major version jump on a docker compose pull will not migrate your data for you.
  • Never cleaning up. docker system prune and docker volume ls deserve a place in your routine; orphaned volumes from old project names accumulate silently.

Key takeaways

  • Compose v2 runs as docker compose, prefers compose.yaml and no longer wants a version: key; set name: so the project identity does not depend on the directory name.
  • Every service joins a project network with embedded DNS, so containers address each other by service name and container port; ports is only for host access and expose does nothing.
  • Segment with a second network and internal: true when you want a database that genuinely cannot be reached from the edge.
  • depends_on alone orders startup; healthchecks plus condition: service_healthy or service_completed_successfully make it mean readiness — and applications should still retry.
  • Named volumes for data, bind mounts for code and configuration (with :ro where possible), tmpfs for scratch; back databases up with their own dump tools.
  • Layer configuration with .env interpolation, env_file, override files and profiles instead of copying the whole file per environment.
  • In CI, give each run its own project name, use --wait and run --rm, avoid host port bindings, and tear down with down -v in a trap.

Sources and further reading

Frequently asked questions

Why can't my application connect to localhost:5432 for the database?
Inside a container, localhost is that container's own network namespace, so it refers to the application itself. Other services on the same Compose network are reached by service name and the port the process listens on inside its container — db:5432 — regardless of what you published to the host. Published ports exist for access from your laptop, not for container-to-container traffic.
Is depends_on enough to make the database ready before the app starts?
No. Plain depends_on only orders container creation; the database process inside may still be initialising. Add a healthcheck to the database service and use depends_on with condition service_healthy so Compose waits for the check to pass. Even then, applications should retry their first connection, because containers restart and dependencies go away at runtime too.
What is the difference between a named volume and a bind mount?
A named volume is storage Docker creates and manages, portable across machines and easy to back up or move between containers. A bind mount maps a specific host path into the container, so its contents and permissions come from your filesystem. Use named volumes for data such as database files, and bind mounts for source code you want to edit live during development.
Can I use Compose in production?
For a single host with a handful of services and tolerance for brief downtime during updates, yes — many small deployments run exactly that way. What you do not get is scheduling across machines, rolling updates with health-based rollback, horizontal autoscaling or self-healing across node failures. When those matter, the same service definitions map fairly directly onto Kubernetes objects.