DevOps, CI/CD & Containers
Kubernetes Patterns: Probes, Resources, Scaling, Storage and Security
Advanced10 min readUpdated
A Deployment and a Service will run your application; they will not make it survive a bad release, a noisy neighbour or a node failure. This guide covers the patterns that separate a demo cluster from a production one — probes that tell the truth, resource requests that make scheduling predictable, safe rollouts, autoscaling, persistent state, least-privilege access and a sane way to package manifests per environment.
What you will learn
- Configure liveness, readiness and startup probes that reflect real application state
- Set requests and limits deliberately and predict the resulting QoS class and eviction behaviour
- Control rollouts with surge settings, disruption budgets and rollback
- Scale workloads horizontally on CPU and custom metrics
- Run stateful workloads with StatefulSets, PersistentVolumeClaims and StorageClasses
- Apply RBAC, security contexts and network policy, and package manifests with Helm or Kustomize
The gap between "it runs on the cluster" and "it runs in production" is filled by decisions that are easy to postpone: how the cluster decides your container is healthy, what happens when it uses more memory than expected, how a bad release is contained, what breaks when a node is drained for maintenance. Skip them and the symptoms are familiar — restarts during traffic peaks, deploys that drop requests, one greedy service degrading everything on a node.
This guide works through those decisions as a set of patterns, each with the manifest fragment that implements it. It assumes the objects from the Kubernetes fundamentals guide are familiar and targets clusters on the 1.3x line.
Probes that tell the truth
Kubernetes asks each container three separate questions, and conflating them causes more production incidents than any other misconfiguration.
Liveness asks: is this process wedged beyond recovery? A failure restarts the container. Readiness asks: can this instance serve traffic right now? A failure removes the Pod from Service endpoints but leaves it running. Startup asks: has the process finished booting? While it is failing, liveness and readiness are suspended, which lets a slow starter avoid a restart loop without loosening the liveness timings for the rest of its life.
startupProbe:
httpGet: { path: /actuator/health/liveness, port: http }
failureThreshold: 30
periodSeconds: 5 # allows up to 150s to boot
livenessProbe:
httpGet: { path: /actuator/health/liveness, port: http }
periodSeconds: 10
failureThreshold: 3
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: http }
periodSeconds: 5
failureThreshold: 2
The rules that follow from the definitions:
- The liveness endpoint must not check dependencies. It should return 200 if the process can still handle requests at all. An endpoint that pings the database turns a database blip into a cluster-wide restart storm.
- The readiness endpoint should check dependencies that the instance genuinely needs, and should start reporting failure as soon as the application receives
SIGTERM, so traffic drains before the process exits. - Probe timings must fit the application. A JVM service that takes 90 seconds to warm up needs a startup probe, not a 120-second
initialDelaySecondson liveness.
Frameworks help here: Spring Boot exposes separate liveness and readiness groups through Actuator, described in the observability guide, and most ecosystems have an equivalent.
Pair readiness with graceful shutdown. When a Pod is deleted, Kubernetes removes it from endpoints and sends SIGTERM at roughly the same moment, so in-flight requests can still arrive for a second or two. A preStop sleep bridges the gap:
lifecycle:
preStop:
exec: { command: ["sh", "-c", "sleep 5"] }
terminationGracePeriodSeconds: 45
Requests, limits and what the scheduler does with them
A request is what the scheduler reserves; a limit is what the runtime enforces. The scheduler places a Pod only on a node whose unreserved capacity covers the sum of its containers' requests — actual usage is irrelevant to that decision.
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
memory: "512Mi"
The two resources behave completely differently. CPU is compressible: exceeding the limit means the kernel throttles the cgroup, adding latency in bursts that look like mysterious tail-latency spikes. Memory is incompressible: exceeding the limit means the container is killed with exit code 137 and OOMKilled in its status. That asymmetry is why the common advice is to always set memory limits, set memory requests equal to them for anything important, set CPU requests carefully, and be sceptical of CPU limits.
Requests and limits together determine the Pod's QoS class, which decides who dies first when a node runs out of memory:
| QoS class | Condition | Eviction order |
|---|---|---|
Guaranteed | Every container sets requests equal to limits, for both CPU and memory | Last |
Burstable | At least one request set, but not matching limits | Middle |
BestEffort | No requests or limits at all | First |
Give production workloads Guaranteed or a well-tuned Burstable; leave BestEffort for things you are happy to lose.
Two namespace-level objects keep this honest across a shared cluster. A LimitRange supplies defaults and caps so that a Pod without resources cannot be created, and a ResourceQuota bounds the namespace's total consumption:
apiVersion: v1
kind: ResourceQuota
metadata: { name: team-quota, namespace: staging }
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.memory: 60Gi
pods: "80"
For JVM workloads, remember the container needs to know about the limit. Modern JVMs read cgroup limits automatically, but set -XX:MaxRAMPercentage=75 so the heap leaves room for metaspace, thread stacks and native buffers — a heap sized at 100 % of the memory limit is an OOMKilled waiting to happen.
Rollouts you can trust
A Deployment's strategy governs how it replaces Pods:
spec:
minReadySeconds: 15
progressDeadlineSeconds: 600
revisionHistoryLimit: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # may create one extra Pod above replicas
maxUnavailable: 0 # never drop below the desired count
maxUnavailable: 0 with maxSurge: 1 is the conservative setting: capacity never dips, at the cost of a slower rollout and one extra Pod's resources. minReadySeconds stops a Pod that passes readiness and then crashes from being counted as a success, which is what makes a bad release stall rather than propagate.
progressDeadlineSeconds marks a stuck rollout as failed but does not roll it back. Make your pipeline do that:
kubectl apply -f k8s/
if ! kubectl rollout status deploy/shop-api --timeout=10m; then
kubectl rollout undo deploy/shop-api
exit 1
fi
A PodDisruptionBudget protects availability during voluntary disruptions — node drains, cluster upgrades, autoscaler scale-down. It does nothing for a node that crashes, but it stops an administrator from evicting all your replicas at once:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: shop-api }
spec:
minAvailable: 2
selector:
matchLabels: { app: shop-api }
Set minAvailable below your replica count or the budget blocks drains entirely and someone will delete it at 3 a.m. Blue-green and canary strategies, and the trade-offs between them, are covered in the delivery pipeline guide; on Kubernetes they are usually implemented with a progressive-delivery controller rather than by hand.
Scaling
The HorizontalPodAutoscaler adjusts a Deployment's replica count towards a metric target. It needs metrics-server installed and resource requests set — CPU utilisation is expressed as a percentage of the request, so a Pod with no request has no denominator.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: shop-api }
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: shop-api
minReplicas: 3
maxReplicas: 20
metrics:
- type: Resource
resource:
name: cpu
target: { type: Utilization, averageUtilization: 70 }
behavior:
scaleDown:
stabilizationWindowSeconds: 300
policies:
- { type: Percent, value: 25, periodSeconds: 60 }
scaleUp:
policies:
- { type: Percent, value: 100, periodSeconds: 30 }
The behavior block is what makes autoscaling calm: scale up quickly when load arrives, scale down slowly with a five-minute stabilisation window so a brief dip does not remove capacity you will need again in a minute. For workloads where CPU is not the bottleneck — queue consumers, for instance — scale on a custom or external metric such as queue depth; KEDA is the usual way to plumb those in.
Two companions: the VerticalPodAutoscaler recommends or applies better requests and limits, useful in recommendation mode even if you never let it act; and the Cluster Autoscaler (or Karpenter on AWS) adds and removes nodes when Pods cannot be scheduled. Horizontal scaling of Pods is pointless if the cluster has no room to put them.
Getting traffic in
A LoadBalancer Service per application means a cloud load balancer and a public IP per application. Ingress consolidates them: one controller — ingress-nginx, Traefik, HAProxy, or a cloud-native one — behind a single load balancer, routing by host and path and terminating TLS.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
nginx.ingress.kubernetes.io/proxy-body-size: "10m"
spec:
ingressClassName: nginx
tls:
- hosts: [shop.example.com]
secretName: shop-tls
rules:
- host: shop.example.com
http:
paths:
- path: /api
pathType: Prefix
backend:
service:
name: shop-api
port: { name: http }
Ingress's weakness is that anything beyond host/path routing lives in controller-specific annotations, so manifests stop being portable. The Gateway API is the successor: GatewayClass, Gateway and HTTPRoute are first-class typed resources supporting header matching, traffic splitting by weight and cross-namespace delegation, with a clean split between the platform team owning the Gateway and application teams owning routes. It is the direction of travel for new clusters; Ingress remains supported and ubiquitous.
Stateful workloads
Storage is expressed through three objects. A PersistentVolumeClaim is a request for storage; a StorageClass describes a kind of storage and the provisioner that creates it on demand; a PersistentVolume is the resulting piece of storage. In a cloud cluster you write claims and reference a class, and the rest is automatic.
A StatefulSet differs from a Deployment in ways that matter for databases and brokers: Pods get stable ordinal names (kafka-0, kafka-1), a stable DNS name via a headless Service, their own PersistentVolumeClaim created from a template and retained across restarts, and ordered, one-at-a-time rollouts.
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: postgres }
spec:
serviceName: postgres-headless
replicas: 3
selector: { matchLabels: { app: postgres } }
template:
metadata: { labels: { app: postgres } }
spec:
terminationGracePeriodSeconds: 60
containers:
- name: postgres
image: postgres:16
volumeMounts:
- { name: data, mountPath: /var/lib/postgresql/data }
volumeClaimTemplates:
- metadata: { name: data }
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: fast-ssd
resources: { requests: { storage: 100Gi } }
Note ReadWriteOnce: most block storage can be mounted by one node at a time, so two Pods cannot share such a volume. Shared read-write access needs a file storage class supporting ReadWriteMany.
A StatefulSet gives you identity and storage. It does not give you replication, failover or backups — those come from the database software and, in practice, from an operator: a controller that encodes the operational knowledge for a specific system, watching custom resources and performing backups, failovers and version upgrades. Running a serious database on Kubernetes means running its operator, or using a managed service and pointing the cluster at it.
Batch work
A Job runs Pods until a specified number complete successfully. A CronJob creates Jobs on a schedule.
apiVersion: batch/v1
kind: CronJob
metadata: { name: nightly-reconcile }
spec:
schedule: "17 2 * * *"
timeZone: "Europe/Paris"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 3
jobTemplate:
spec:
backoffLimit: 2
activeDeadlineSeconds: 3600
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: OnFailure
containers:
- name: job
image: registry.example.com/acme/reconcile:1.4.2
concurrencyPolicy: Forbid prevents a slow run from overlapping the next schedule — the default Allow has caused a lot of duplicated processing. ttlSecondsAfterFinished cleans up completed Jobs, without which the namespace accumulates thousands of finished Pods. And activeDeadlineSeconds stops a hung job from running forever.
Init containers and sidecars
Init containers run to completion, in order, before the application containers start. They are the right place for waiting on a dependency, running a schema migration, or fetching a file into a shared emptyDir.
Sidecars run alongside the application for the Pod's lifetime. Since 1.29 they have first-class support: declare them as init containers with restartPolicy: Always, and the kubelet starts them before the application containers and stops them after — which finally fixes the old problem of a log shipper dying before it flushed, or a sidecar keeping a Job Pod alive forever.
initContainers:
- name: wait-for-db
image: busybox:1.36
command: ["sh", "-c", "until nc -z postgres 5432; do sleep 2; done"]
- name: log-shipper # a native sidecar
image: fluent-bit:3.1
restartPolicy: Always
volumeMounts:
- { name: logs, mountPath: /var/log/app }
Adapter and ambassador variants of the same idea — a container that reshapes the application's output, or that proxies its outbound calls — are what service meshes install automatically.
Least privilege
Four mechanisms, each closing a different door.
Security context constrains the container itself:
securityContext:
runAsNonRoot: true
runAsUser: 10001
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: ["ALL"] }
seccompProfile: { type: RuntimeDefault }
A read-only root filesystem forces you to mount emptyDir volumes wherever the application genuinely writes, which is a useful audit in itself. Pod Security Admission enforces these settings namespace-wide with a label such as pod-security.kubernetes.io/enforce: restricted, so a manifest that omits them is rejected rather than quietly insecure.
RBAC governs what a subject may do through the API. Give each workload its own ServiceAccount, and grant only what it needs:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { namespace: staging, name: config-reader }
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { namespace: staging, name: shop-api-config-reader }
subjects:
- kind: ServiceAccount
name: shop-api
roleRef: { kind: Role, name: config-reader, apiGroup: rbac.authorization.k8s.io }
Role and RoleBinding are namespaced; ClusterRole and ClusterRoleBinding apply cluster-wide. Set automountServiceAccountToken: false on workloads that never call the API at all. kubectl auth can-i --list --as=system:serviceaccount:staging:shop-api tells you what a service account can actually do, which is usually more than its owner assumed.
NetworkPolicy restricts Pod-to-Pod traffic, which is otherwise completely open. The pattern is to default-deny a namespace and then allow specific flows:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny-ingress, namespace: staging }
spec:
podSelector: {}
policyTypes: ["Ingress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-from-ingress, namespace: staging }
spec:
podSelector: { matchLabels: { app: shop-api } }
policyTypes: ["Ingress"]
ingress:
- from:
- namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: ingress-nginx } }
ports:
- { protocol: TCP, port: 8080 }
Policies are enforced by the CNI plugin, so confirm yours supports them — with some plugins the objects are accepted and silently ignored, which is worse than not having them.
Packaging manifests
Copying YAML per environment does not survive contact with a third environment. Two tools dominate.
Kustomize is built into kubectl. A base directory holds plain manifests; overlays patch them:
# overlays/prod/kustomization.yaml
resources:
- ../../base
namespace: prod
images:
- name: registry.example.com/acme/shop-api
newTag: "1.4.2"
replicas:
- { name: shop-api, count: 6 }
patches:
- path: resources-patch.yaml
target: { kind: Deployment, name: shop-api }
configMapGenerator:
- name: shop-api-config
files: [application.yaml]
Nothing is templated, so every file is valid YAML you can read, and kubectl kustomize overlays/prod prints exactly what will be applied. The generators add a content hash to ConfigMap names, which means changing configuration automatically triggers a rollout — the problem the checksum annotation solves by hand.
Helm packages manifests as a versioned chart with Go templates and a values.yaml. It is the distribution format for practically all third-party components — helm install ingress-nginx ingress-nginx/ingress-nginx -f values.yaml — and it tracks releases so helm rollback restores the previous manifests. The cost is that templated YAML is awkward to read and debug; helm template before helm install should be a reflex.
Whichever you choose, the manifests belong in Git and should be applied by an automated process rather than from a laptop. GitOps controllers such as Argo CD and Flux take this to its conclusion: the repository is the desired state, the controller continuously reconciles the cluster with it, and drift is reported instead of discovered.
Key takeaways
- Liveness restarts, readiness drains, startup protects slow boots — never let a liveness probe check a dependency, and add a
preStopdelay so connections drain. - Requests drive scheduling and limits enforce; always set memory limits, treat CPU limits with suspicion, and know which QoS class your Pods land in.
- Rolling updates with
maxUnavailable: 0andminReadySecondsfail safely, but rollback is your pipeline's job; a PodDisruptionBudget protects you during drains. - Autoscale on CPU relative to requests, tune the
behaviorblock so scale-down is slow, and remember the cluster autoscaler has to supply the nodes. - Consolidate entry points behind an Ingress controller today and plan for the Gateway API; keep controller-specific behaviour in annotations you can find later.
- StatefulSets give identity and per-Pod storage; replication, failover and backups come from the software or its operator, not from Kubernetes.
- Lock workloads down with a restrictive security context, per-workload ServiceAccounts with minimal RBAC, and a default-deny NetworkPolicy your CNI actually enforces.
- Use Kustomize overlays for your own services and Helm charts for third-party software, keep both in Git, and let a GitOps controller apply them.
Sources and further reading
Frequently asked questions
- Should I set CPU limits?
- Often not. CPU is compressible: exceeding a limit throttles the container rather than killing it, and aggressive throttling produces latency spikes that are hard to diagnose. Setting a CPU request guarantees a share and drives scheduling, which is the part that matters. Memory is different — it is not compressible, so a memory limit is essential to stop one container taking down a node, and it should equal the request for critical workloads.
- Why does my Pod restart even though the application is fine?
- Usually a liveness probe that is too strict. If the endpoint checks a database connection, a slow query or a brief outage marks the container dead and Kubernetes restarts it — which does not fix the database and does drop in-flight requests. Liveness should test only whether the process is wedged. Dependency health belongs in the readiness probe, which removes the Pod from load balancing without killing it.
- What actually happens during a rolling update?
- The Deployment creates a new ReplicaSet and scales it up while scaling the old one down, respecting maxSurge and maxUnavailable. A new Pod only counts as available once its readiness probe passes and minReadySeconds has elapsed. If new Pods never become ready, progressDeadlineSeconds eventually marks the rollout failed — it does not roll back automatically, so either watch rollout status in your pipeline or use a controller that does.
- Helm or Kustomize?
- Kustomize suits your own applications: you keep plain YAML and layer environment-specific patches over a base, with no templating language to debug. Helm suits software you distribute or consume, because a chart is a versioned package with values, dependencies and release history. Many teams use both — Helm for third-party components, Kustomize for their own services — and that combination is entirely reasonable.