DevOps, CI/CD & Containers

Infrastructure as Code: Terraform, Ansible and Immutable Infrastructure

Advanced11 min readUpdated

Infrastructure as code turns servers, networks and cloud accounts into files you review, test and roll back like application code. This guide separates provisioning from configuration management, works through Terraform's state model and module design, shows the Ansible side, and covers the operational realities — drift, locking, secrets and running plan and apply from a pipeline rather than a laptop.

What you will learn

  • Distinguish declarative provisioning from imperative configuration management and explain idempotence
  • Write Terraform configuration with providers, resources, variables, outputs and modules
  • Explain what state is, why it is locked, and how remote backends protect it
  • Write an idempotent Ansible playbook and know when configuration management still applies
  • Detect and handle drift, and keep secrets out of code and state
  • Run plan on a pull request and apply on merge, with policy checks in the pipeline

Every infrastructure eventually reaches the point where nobody can say exactly what is running. A firewall rule was added during an incident, a server was patched by hand, a load balancer's timeout was raised for a demo three years ago. Rebuilding the environment becomes a research project, and staging diverged from production so long ago that testing there proves very little.

Infrastructure as code is the discipline that prevents this: the infrastructure is described in files, the files live in version control, and changes reach the environment by applying those files rather than by clicking or by logging in. What you gain is not just automation. You gain review, history, the ability to diff two environments, and the ability to rebuild from nothing — which is what turns "our data centre burned down" from an existential event into a long day.

Two jobs, two kinds of tool

The term covers two related activities that people conflate to their cost.

Provisioning creates and destroys resources: virtual networks, subnets, security groups, virtual machines, managed databases, Kubernetes clusters, DNS records, IAM roles. The resources do not exist until the tool makes them. Terraform and its fork OpenTofu, along with cloud-native tools such as CloudFormation, Bicep and Pulumi, live here.

Configuration management takes an existing machine and makes its contents match a description: packages installed, files written, services enabled, users created. Ansible, Chef, Puppet and Salt live here.

The workflows differ too. Provisioning tools are declarative with a persistent record of what they own, so they can compute the difference between your description and reality and destroy things you removed from the file. Configuration managers are typically procedural but idempotent: a playbook is an ordered list of tasks, each written so that running it on an already-correct machine changes nothing.

Idempotence is the property that makes any of this safe. Applying the same configuration twice produces the same result as applying it once, which is why a failed run can simply be re-run and why the tool can run on a schedule to correct drift. Shell scripts are the counter-example: useradd deploy succeeds once and fails afterwards, so every script accumulates conditionals until it becomes an unreliable reimplementation of a configuration manager.

Terraform: the model

Terraform reads .tf files written in HCL, talks to providers (plugins for AWS, Azure, Google Cloud, Kubernetes, GitHub, PostgreSQL and hundreds of others), and maintains state mapping the resources in your configuration to the real objects it created.

terraform {
  required_version = "~> 1.9"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.60"
    }
  }
}

provider "aws" {
  region = var.region
  default_tags {
    tags = {
      Environment = var.environment
      ManagedBy   = "terraform"
      Repository  = "acme/infrastructure"
    }
  }
}

variable "environment" {
  description = "Deployment environment name"
  type        = string
  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "environment must be dev, staging or prod."
  }
}

variable "region" {
  type    = string
  default = "eu-west-3"
}

resource "aws_vpc" "main" {
  cidr_block           = "10.20.0.0/16"
  enable_dns_hostnames = true
  tags = { Name = "${var.environment}-vpc" }
}

resource "aws_subnet" "private" {
  for_each          = { for i, az in local.azs : az => cidrsubnet(aws_vpc.main.cidr_block, 4, i) }
  vpc_id            = aws_vpc.main.id
  availability_zone = each.key
  cidr_block        = each.value
  tags              = { Name = "${var.environment}-private-${each.key}" }
}

locals {
  azs = ["eu-west-3a", "eu-west-3b", "eu-west-3c"]
}

output "vpc_id" {
  value = aws_vpc.main.id
}

Two habits here are worth adopting early. default_tags on the provider labels everything Terraform creates, which makes cost attribution and orphan-hunting possible later. And for_each over a map, rather than count over a number, gives each subnet a stable address keyed by availability zone — with count, removing the middle element renumbers everything after it and Terraform proposes destroying resources that should not have been touched. Pin the Terraform and provider versions too, and commit .terraform.lock.hcl so every machine resolves the same provider builds.

The workflow

terraform init          # download providers and configure the backend
terraform fmt -recursive
terraform validate      # syntax and internal consistency
terraform plan -out=tfplan
terraform apply tfplan
terraform destroy       # tear the environment down

The important step is plan, and the important skill is reading one. A plan lists every action Terraform intends: create, update in place, destroy, or the dreaded "must be replaced". Replacement means destroy-then-create, and on a database that is a data-loss event announced in a line of output that scrolls past quickly.

  # aws_db_instance.main must be replaced
-/+ resource "aws_db_instance" "main" {
      ~ engine_version = "16.3" -> "17.2" # forces replacement
      ~ id             = "shop-prod-db" -> (known after apply)
    }

Always apply a saved plan file rather than running terraform apply on its own. apply without a plan re-plans and asks for confirmation, and the thing it applies may differ from what you reviewed if the world changed in between.

A plan with no changes should print "No changes". If every run instead proposes an update to the same resource, you have a provider bug or a value the cloud normalises differently from what you wrote. Chase it down: if the plan is never clean, you cannot use a clean plan as a signal.

State, and why it needs a backend

State is a JSON document mapping each resource address in your configuration (aws_vpc.main) to the real resource's identifier and attributes. Terraform needs it to know what it created, to detect drift, and to compute dependencies.

A local state file is fine for a tutorial and unworkable for a team: it is not shared, cannot be locked, contains secrets, and losing it means Terraform forgets everything it owns. A remote backend solves all four.

terraform {
  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "prod/network/terraform.tfstate"
    region       = "eu-west-3"
    encrypt      = true
    use_lockfile = true          # native S3 locking, Terraform 1.10+
  }
}

Older configurations use a DynamoDB table for locking instead; Azure and Google Cloud backends lock through their own storage primitives, and Terraform Cloud, Scalr and Spacelift provide managed state with locking and a run history. Whichever you use, enable versioning on the bucket so a corrupted state can be rolled back.

Two consequences follow. First, state is sensitive: passwords, keys and certificates generated by resources are stored in it in plain text. Encrypt the backend, restrict who can read the bucket, and never commit state to Git. Second, state must be split. One enormous state file for an entire organisation means every change plans against everything, applies take twenty minutes, and one broken resource blocks unrelated work. Split by lifecycle and blast radius — network, data stores, cluster, applications — and read across boundaries with data sources or remote state outputs.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-terraform-state"
    key    = "prod/network/terraform.tfstate"
    region = "eu-west-3"
  }
}

resource "aws_instance" "app" {
  subnet_id = data.terraform_remote_state.network.outputs.private_subnet_ids[0]
  # ...
}

When you must intervene in state — a resource renamed, a resource created by hand, a resource that should no longer be managed — use the dedicated commands rather than editing the file: terraform state mv, terraform state rm, and import. Since Terraform 1.5, moved and import blocks let you express those operations declaratively in the configuration, so they go through review like everything else:

moved {
  from = aws_instance.web
  to   = aws_instance.frontend
}

import {
  to = aws_s3_bucket.legacy_assets
  id = "acme-legacy-assets"
}

Modules

A module is a directory of Terraform files with inputs and outputs. The root configuration is itself a module; everything else is a reusable one.

module "service" {
  source  = "git::https://git.example.com/acme/tf-modules.git//service?ref=v2.3.0"

  name        = "shop-api"
  environment = var.environment
  vpc_id      = data.terraform_remote_state.network.outputs.vpc_id
  subnet_ids  = data.terraform_remote_state.network.outputs.private_subnet_ids
  image_tag   = var.image_tag
  min_size    = 3
  max_size    = 12
}

Good modules look like good functions: a clear purpose, well-named typed inputs with sensible defaults, and outputs that expose what callers need rather than everything. Pin the version with ?ref= to a tag — a module referenced by branch changes under you, and the day you discover it is during an incident.

Resist two temptations. Do not wrap a single resource in a module that adds nothing but indirection. And do not build one giant module with forty boolean flags — composition beats configuration, and a module whose behaviour depends on count = var.enabled ? 1 : 0 in twenty places is unreadable.

For environments, prefer separate directories with their own backend keys and .tfvars files over Terraform workspaces. Workspaces share one configuration and one backend key prefix, which makes it far too easy to run a production apply while pointed at the wrong workspace. Separate directories make the target explicit.

Ansible: configuring what exists

Terraform stops at the machine boundary. If you run virtual machines rather than containers, something has to install packages, write configuration files and start services. Ansible does that over SSH with no agent to install.

An inventory lists hosts and groups; a playbook maps groups to roles and tasks; modules perform idempotent operations.

- name: Configure application servers
  hosts: app_servers
  become: true
  vars:
    java_package: openjdk-21-jre-headless
    app_user: shop
  tasks:
    - name: Install runtime packages
      ansible.builtin.apt:
        name:
          - "{{ java_package }}"
          - ca-certificates
        state: present
        update_cache: true
        cache_valid_time: 3600

    - name: Create the service account
      ansible.builtin.user:
        name: "{{ app_user }}"
        system: true
        shell: /usr/sbin/nologin

    - name: Deploy the systemd unit
      ansible.builtin.template:
        src: templates/shop-api.service.j2
        dest: /etc/systemd/system/shop-api.service
        owner: root
        group: root
        mode: "0644"
      notify: Restart shop-api

    - name: Enable and start the service
      ansible.builtin.systemd_service:
        name: shop-api
        enabled: true
        state: started
        daemon_reload: true

  handlers:
    - name: Restart shop-api
      ansible.builtin.systemd_service:
        name: shop-api
        state: restarted

Each module is idempotent: apt with state: present installs only if missing, template rewrites the file only when the rendered content differs, and the handler restarts the service only when something notified it. Run it with --check --diff first to see what would change without changing anything — the equivalent of a Terraform plan.

ansible-playbook -i inventories/prod site.yml --check --diff --limit app_servers
ansible-playbook -i inventories/prod site.yml --limit app_servers

Package tasks into roles (roles/app-server/{tasks,templates,handlers,defaults}) so playbooks stay readable, and keep them under the same review process as everything else. The task names appear in the output, so write them as descriptions a colleague can follow.

Immutable infrastructure

Configuration management assumes long-lived servers you keep correcting. The alternative is to stop correcting and start replacing: build a machine image or container image containing the fully configured system, deploy new instances from it, and never modify a running one.

The pipeline becomes: build an image (Packer for VM images, a Dockerfile for containers), test it, then have the provisioning tool replace instances with ones based on the new image. Nothing is patched in place, so there is no drift to detect, no partially applied change, and rollback is deploying the previous image.

source "amazon-ebs" "app" {
  region        = "eu-west-3"
  source_ami_filter {
    filters     = { name = "ubuntu/images/*24.04-amd64-server-*" }
    most_recent = true
    owners      = ["099720109477"]
  }
  instance_type = "t3.small"
  ssh_username  = "ubuntu"
  ami_name      = "shop-api-{{timestamp}}"
}

build {
  sources = ["source.amazon-ebs.app"]
  provisioner "ansible" {
    playbook_file = "./ansible/site.yml"
  }
}

Note that Ansible has not disappeared — it now runs once at image-build time rather than continuously against production. That is the pragmatic middle ground most teams reach, and it is exactly the same argument that containers make: the delivery pipeline builds an immutable artifact once and promotes it unchanged.

Immutability has limits: databases hold state that cannot be replaced casually. Apply the pattern where it fits and keep configuration management for the rest.

Drift, and what to do about it

Drift is any difference between the code and reality: a console change during an incident, an autoscaler adjusting a value, a cloud provider changing a default, another team's automation touching a shared resource.

Detect it on a schedule rather than during a deployment:

terraform plan -detailed-exitcode -lock=false
# exit 0 = no changes, 1 = error, 2 = changes pending

A nightly pipeline job running that and alerting on exit code 2 turns drift into a notification instead of a surprise. Ansible's --check mode gives the equivalent signal for configured hosts.

Handling drift is a judgement call. Usually you reconcile by re-applying the code, which undoes the manual change — safe when the change was accidental, dangerous when it was an urgent fix nobody has told you about. Sometimes you adopt the change into the code because it was correct. Occasionally you accept that a field is managed elsewhere and tell Terraform to stop caring:

lifecycle {
  ignore_changes  = [tags["LastScanned"], desired_count]
  prevent_destroy = true
}

prevent_destroy on databases and state buckets is cheap insurance against a plan nobody read carefully.

Secrets

Never put a credential in a .tf or .yml file. The realistic options, in increasing order of maturity: environment variables supplied by the pipeline (TF_VAR_db_password); a secret manager referenced by a data source at plan time; and resources that never hold the secret at all, such as a database whose password is generated straight into the cloud provider's secret manager.

Ansible has Vault for encrypting variable files in place, which keeps secrets in Git in encrypted form:

ansible-vault encrypt group_vars/prod/secrets.yml
ansible-playbook site.yml --vault-password-file ~/.ansible-vault-pass

And remember the constraint from earlier: a secret Terraform creates or reads lands in state in plain text. Marking a variable sensitive = true only suppresses console output. If a value must never sit in state, the resource that owns it must not be managed by Terraform.

Running it from a pipeline

Applying from laptops is where most IaC incidents originate: different tool versions, stale providers, no lock, no audit trail, and credentials sitting in someone's shell profile. Move it into CI.

The standard shape is plan on the pull request, apply on merge:

  1. On every pull request

    Run fmt -check, validate, a security scan (tfsec, Checkov or Trivy's IaC mode) and plan -out=tfplan. Post the plan as a comment so reviewers see the infrastructure diff next to the code diff.
  2. Gate on policy, not on vigilance

    Add policy-as-code — OPA/Conftest, Sentinel or the scanner's rules — for the invariants that must always hold: no public storage buckets, encryption enabled, mandatory tags, no security group open to the world on port 22.
  3. Apply the reviewed plan on merge

    Apply the saved plan artifact from the pull-request run, not a freshly computed one, so what ships is what was approved. Use short-lived credentials from OIDC federation rather than long-lived access keys.
  4. Promote through environments

    The same module versions applied to dev, then staging, then production with different variable files. Differences between environments should be values, never separate code.
  5. Detect drift nightly

    A scheduled plan -detailed-exitcode per state, alerting when reality has moved away from the code.

The self-service extension is a platform: the same modules exposed to product teams behind a pull request, so a team gets a queue, a bucket or a database by adding ten lines to a repository rather than raising a ticket — the platform-team model described in the DevOps introduction.

Where it goes wrong

  • State in Git. It contains secrets, it conflicts on every merge, and it will be lost. Use a remote backend from day one.
  • One state for everything. Slow plans, huge blast radius, constant lock contention. Split by lifecycle.
  • Modules referenced by branch. An upstream commit changes every consumer's plan without warning. Pin tags.
  • count where for_each belongs. Index-based addressing turns a deletion in the middle of a list into a cascade of replacements.
  • Ignoring the plan. "Must be replaced" on a database is not a formality. Read every destroy line before typing yes.
  • Half-migrated infrastructure. Some resources in code, some clicked in the console, nobody sure which. Import the strays or delete them, and put a tag policy in place so unmanaged resources are visible.
  • Treating IaC as write-only. Code nobody has re-applied for a year is a hypothesis. Rebuild an ephemeral environment from scratch regularly to prove it still works.

Key takeaways

  • Provisioning tools create resources and track them in state; configuration managers make existing machines match a description. Most estates need both, or an image build that folds the second into the first.
  • Idempotence is what makes re-running safe, which is what makes automated correction and unattended pipelines possible.
  • Terraform's plan is the review artifact: read it, save it, apply the saved file, and never let "must be replaced" scroll past unexamined.
  • State is shared, sensitive and lockable — put it in an encrypted remote backend with versioning, split it by blast radius, and change it with moved, import and the state commands rather than by hand.
  • Write modules like functions, pin them by tag, prefer for_each to count, and use separate directories rather than workspaces for environments.
  • Immutable images remove drift by replacing servers instead of correcting them; keep configuration management for the parts that cannot be rebuilt.
  • Detect drift on a schedule with plan -detailed-exitcode, and decide deliberately whether to reconcile, adopt or ignore each difference.
  • Run plan on pull requests with policy checks, apply the approved plan on merge with short-lived credentials, and promote the same modules across environments by changing values only.

Sources and further reading

Frequently asked questions

Terraform or Ansible — which do I need?
They solve different problems and are frequently used together. Terraform is a provisioner: it creates and destroys cloud resources such as networks, databases, clusters and DNS records, tracking what it owns in state. Ansible is a configuration manager: it makes an existing machine match a described configuration, installing packages and writing files over SSH. Provision with one, configure with the other, or bake the configuration into an image and skip the second step.
Can I edit infrastructure in the cloud console after applying?
You can, and the next plan will propose undoing it. That is drift, and it is the mechanism working as designed. Emergency console changes are sometimes unavoidable; the rule is to reproduce the change in code the same day. If a resource must be managed outside Terraform, exclude it explicitly rather than letting the two owners fight over it on every apply.
What happens if two people run apply at the same time?
With a backend that supports locking, the second run fails immediately with a message naming who holds the lock. Without locking, both runs read the same state, both write it, and one set of changes is lost — leaving resources that exist in the cloud but not in state. This is the single strongest reason to use a remote backend and to run apply from a pipeline rather than from laptops.
Are secrets safe in Terraform state?
No. Values such as generated passwords and access keys are written to state in plain text even when the variable is marked sensitive, which only hides them from console output. Treat state as a secret: store it in an encrypted backend with tight access control, never commit it, and prefer resources that reference a secret manager rather than resources that create and hold the secret themselves.