DevOps, CI/CD & Containers

Jenkins Pipelines and the Jenkinsfile: Declarative Syntax in Depth

Intermediate8 min readUpdated

A Jenkinsfile turns your build into reviewed, versioned code that lives with the project. This guide works through declarative pipeline syntax section by section — agent selection, environment and parameters, conditional and parallel stages, retries and timeouts, post conditions — then covers shared libraries, the scripted escape hatch and how to debug a pipeline without twenty trial commits.

What you will learn

  • Write a declarative pipeline with agent, stages, steps, options, environment and post blocks
  • Select where each stage runs, including per-stage Docker agents
  • Parameterise builds and bind credentials without exposing secrets in the log
  • Run stages conditionally and in parallel, and apply timeouts, retries and concurrency limits
  • Extract repeated logic into a shared library and know when scripted Groovy is justified

A pipeline defined in the Jenkins UI is invisible to the people who care about it. Nobody reviews it, nobody can diff it against last month, and it disappears with the controller. Moving that definition into a Jenkinsfile at the root of the repository fixes all of that at once: the build becomes part of the codebase, changes go through review with the code they affect, every branch carries the pipeline appropriate to it, and a rebuilt controller picks the definition back up from the repository.

This guide assumes you have a working controller with agents and credentials — the Jenkins setup guide covers that — and concentrates on the language. Declarative syntax is the focus because it is what you should write; scripted Groovy appears at the end, where it belongs.

The shape of a declarative pipeline

Every declarative pipeline is a pipeline block containing a small number of named sections, in a fixed grammar that Jenkins validates before the build starts.

pipeline {
    agent { label 'linux && jdk21' }

    options {
        timeout(time: 30, unit: 'MINUTES')
        buildDiscarder(logRotator(numToKeepStr: '30'))
        disableConcurrentBuilds(abortPrevious: true)
        timestamps()
    }

    environment {
        MAVEN_OPTS = '-Xmx1g -Djava.awt.headless=true'
        APP_VERSION = "1.4.${env.BUILD_NUMBER}"
    }

    stages {
        stage('Build') {
            steps {
                sh 'mvn -B -ntp clean compile'
            }
        }
        stage('Test') {
            steps {
                sh 'mvn -B -ntp verify'
            }
        }
    }

    post {
        always  { junit testResults: '**/target/surefire-reports/*.xml', allowEmptyResults: true }
        failure { mail to: '[email protected]', subject: "Build ${env.JOB_NAME} #${env.BUILD_NUMBER} failed", body: env.BUILD_URL }
        cleanup { cleanWs() }
    }
}

Read it top to bottom. agent decides where the work runs. options configures the job itself — how long it may take, how much history to keep, whether two builds may run at once. environment defines variables visible to every step. stages holds the ordered list of stage blocks, each containing steps. post runs at the end regardless of the outcome, with sub-blocks selected by result.

The grammar is strict on purpose. Sections must appear in this order, steps may contain only steps, and a typo produces a clear parse error before an agent is even allocated. That strictness is what makes declarative pipelines readable a year later.

Choosing where a stage runs

agent appears at the top level and, optionally, again inside individual stages. The common forms:

agent any                              // any agent with a free executor
agent none                             // allocate nothing at the top level
agent { label 'docker && linux' }      // a label expression
agent { node { label 'build'; customWorkspace '/mnt/fast/ws' } }
agent { docker { image 'maven:3.9-eclipse-temurin-21'; args '-v $HOME/.m2:/root/.m2' } }
agent { dockerfile { dir 'ci'; filename 'Dockerfile.build' } }
agent { kubernetes { yamlFile 'ci/pod.yaml' } }

The docker agent is the one that changes how builds feel. The stage runs inside a container started from that image on an agent that has Docker available, so the toolchain comes from the image rather than from whatever happens to be installed on the machine. Two projects can use different JDK versions with no coordination, and reproducing a CI failure locally is a matter of running the same image.

agent none at the top with per-stage agents is the pattern for pipelines whose stages need different environments. It also avoids holding an executor while waiting for an approval:

pipeline {
    agent none
    stages {
        stage('Build') {
            agent { docker { image 'maven:3.9-eclipse-temurin-21' } }
            steps { sh 'mvn -B -ntp package' }
        }
        stage('Frontend') {
            agent { docker { image 'node:22-alpine' } }
            steps { sh 'npm ci && npm run build' }
        }
    }
}

Be aware that each agent gets its own workspace. Files produced in one stage are not visible in the next unless you carry them across with stash/unstash or archive them.

Environment, parameters and credentials

environment sets variables at pipeline or stage level. Stage-level entries override pipeline-level ones, and both are ordinary shell environment variables inside sh.

Jenkins injects a large set of its own: BUILD_NUMBER, BUILD_URL, JOB_NAME, WORKSPACE, BRANCH_NAME and CHANGE_ID in multibranch jobs, plus GIT_COMMIT and GIT_BRANCH from the Git plugin. Reference them as env.BUILD_NUMBER in Groovy and $BUILD_NUMBER inside a shell step.

Parameters make a job interactive. Declare them in the pipeline; on the first run Jenkins registers them and the job gains a "Build with Parameters" button.

parameters {
    choice(name: 'TARGET_ENV', choices: ['dev', 'staging', 'prod'], description: 'Deployment target')
    booleanParam(name: 'RUN_INTEGRATION', defaultValue: true, description: 'Run the slow suite')
    string(name: 'RELEASE_VERSION', defaultValue: '', description: 'Leave blank for a snapshot build')
}

Read them as params.TARGET_ENV. Note the chicken-and-egg: the first build after adding a parameter runs with defaults, because Jenkins only learns the definition by executing the file.

Credentials come from the store, never from the file. Two mechanisms exist. The credentials() helper in an environment block handles the common kinds:

environment {
    // secret text -> one variable
    SONAR_TOKEN = credentials('sonar-token')
    // username/password -> NEXUS, NEXUS_USR and NEXUS_PSW
    NEXUS = credentials('nexus-deploy')
}

withCredentials is the general form and is required for SSH keys, files and certificates:

steps {
    withCredentials([
        sshUserPrivateKey(credentialsId: 'deploy-key', keyFileVariable: 'SSH_KEY'),
        file(credentialsId: 'kubeconfig-staging', variable: 'KUBECONFIG')
    ]) {
        sh 'ssh -i "$SSH_KEY" deploy@app-01 /opt/deploy.sh'
        sh 'kubectl get pods -n staging'
    }
}

Quote the variables in shell, keep the block as small as possible, and remember that masking hides exact matches only. A secret that a script transforms — base64, URL-encoded, split across lines — will appear in the log unmasked.

Conditional stages with when

when decides whether a stage's steps run at all. It is evaluated after the agent is entered by default, which matters for conditions that inspect the workspace.

stage('Deploy to staging') {
    when {
        allOf {
            branch 'main'
            not { changeRequest() }
            expression { currentBuild.result == null }
        }
    }
    steps { sh './deploy.sh staging' }
}

stage('Publish release') {
    when {
        beforeAgent true
        tag pattern: 'v\\d+\\.\\d+\\.\\d+', comparator: 'REGEXP'
    }
    steps { sh 'mvn -B -ntp deploy -DskipTests' }
}

Useful conditions include branch, tag, changeRequest(), environment name: 'X', value: 'y', changeset '**/*.sql' for path-based triggering, and expression { } for arbitrary Groovy. Combine them with allOf, anyOf and not. Add beforeAgent true whenever the condition does not need a workspace — it avoids allocating an executor for a stage that will be skipped, which on a busy controller is a meaningful saving.

For a human gate, input pauses the build:

stage('Approve production') {
    steps {
        timeout(time: 2, unit: 'HOURS') {
            input message: 'Deploy to production?', ok: 'Deploy', submitter: 'release-managers'
        }
    }
}

Put input in a stage with agent none, or the build holds an executor for the entire waiting period. The delivery pipeline guide discusses when a manual gate is genuinely useful and when it is theatre.

Running work in parallel

Long pipelines are usually long because independent work runs in sequence. parallel fixes that.

stage('Verification') {
    parallel {
        stage('Unit tests') {
            agent { label 'linux' }
            steps { sh 'mvn -B -ntp test' }
        }
        stage('Integration tests') {
            agent { label 'linux && docker' }
            steps { sh 'mvn -B -ntp verify -Pintegration' }
        }
        stage('Static analysis') {
            agent { label 'linux' }
            steps { sh 'mvn -B -ntp checkstyle:check spotbugs:check' }
        }
    }
}

Each branch may have its own agent, so the three run on different machines. failFast true inside the parallel block aborts the siblings as soon as one fails, which saves executor time when a failure is decisive.

matrix expands one stage definition across combinations of axis values — the natural fit for testing several JDK or database versions:

stage('Compatibility') {
    matrix {
        axes {
            axis { name 'JDK'; values '17', '21' }
            axis { name 'DB';  values 'postgres', 'mysql' }
        }
        excludes {
            exclude { axis { name 'JDK'; values '17' }; axis { name 'DB'; values 'mysql' } }
        }
        agent { label 'linux' }
        stages {
            stage('Test') { steps { sh "./ci/test.sh --jdk ${JDK} --db ${DB}" } }
        }
    }
}

Four combinations minus one exclusion means three cells, run concurrently as executors allow.

Robustness: options, retry and timeout

Builds fail for reasons that have nothing to do with the code, and a pipeline that treats every transient failure as a defect trains people to ignore red.

options {
    timeout(time: 45, unit: 'MINUTES')            // whole pipeline
    retry(2)                                       // retry the whole pipeline on failure
    skipStagesAfterUnstable()
    disableConcurrentBuilds(abortPrevious: true)   // newer push cancels the older run
    quietPeriod(15)
    preserveStashes(buildCount: 5)
}

At step level, retry and timeout wrap individual actions, which is far more precise than retrying a whole build:

steps {
    timeout(time: 5, unit: 'MINUTES') {
        retry(3) { sh 'curl -fsSL https://artifacts.example.com/toolchain.tgz -o toolchain.tgz' }
    }
}

Use retries for network and infrastructure operations only. Retrying a flaky test hides the flakiness instead of fixing it, and a suite with hidden flakiness eventually fails in a way retries cannot rescue.

The post block

post is where reporting and clean-up belong, because it runs whether the build passed, failed or was aborted. The conditions run in a defined order: always, then changed, then fixed, regression, aborted, failure, success, unstable, and finally cleanup.

post {
    always    { archiveArtifacts artifacts: 'target/*.jar', fingerprint: true, allowEmptyArchive: true }
    unstable  { slackSend channel: '#ci', color: 'warning', message: "Tests failing on ${env.JOB_NAME}" }
    fixed     { slackSend channel: '#ci', color: 'good',    message: "Back to green: ${env.JOB_NAME}" }
    regression{ slackSend channel: '#ci', color: 'danger',  message: "Broken by ${env.GIT_COMMIT}" }
    cleanup   { cleanWs() }
}

fixed and regression fire only on a change of state, which is the difference between a notification channel people read and one they mute. cleanup always runs last and is the right place for cleanWs(); deleting the workspace in always can race with steps that still need it.

post blocks may also be attached to an individual stage, where they apply to that stage only — handy for publishing a stage's reports next to the code that produced them.

Shared libraries

Once three repositories contain the same forty lines of pipeline, extract them. A shared library is a Git repository with a fixed structure:

(root)
├── vars/
│   ├── buildJavaService.groovy      # a global step: buildJavaService(...)
│   └── buildJavaService.txt         # help text shown in the snippet generator
├── src/
│   └── com/example/ci/Versioning.groovy
└── resources/
    └── com/example/ci/pod-template.yaml

Files in vars/ become steps callable by name; classes in src/ are ordinary Groovy you can unit test; resources/ holds files loaded with libraryResource. Register the library once under Manage Jenkins → System → Global Pipeline Libraries, then use it:

@Library('ci-lib@v3') _

standardJavaPipeline(
    jdk: '21',
    sonarProject: 'shop-api',
    deployTo: ['staging']
)

A vars/ file that defines def call(Map config) can encapsulate an entire pipeline, reducing each project's Jenkinsfile to a handful of declarative lines. Version the library with tags and reference a tag rather than @main for anything important — otherwise a library change silently alters every build in the organisation at once.

Scripted pipelines and the script block

Scripted syntax came first: a Groovy program with node, stage and full imperative control.

node('linux') {
    stage('Checkout') { checkout scm }
    stage('Build') {
        def modules = sh(script: 'ls -d modules/*', returnStdout: true).trim().split('\n')
        for (m in modules) {
            sh "mvn -B -ntp -f ${m}/pom.xml package"
        }
    }
}

It is more flexible and much easier to write badly. There is no validation before the build starts, the stage view degrades, and the file drifts towards a program that only its author understands. Reserve it for pipelines whose structure genuinely cannot be known in advance.

The pragmatic middle ground is a script block inside a declarative stage, which allows arbitrary Groovy just where you need it:

stage('Version') {
    steps {
        script {
            def pom = readMavenPom file: 'pom.xml'
            env.RELEASE_VERSION = pom.version.replace('-SNAPSHOT', '')
            currentBuild.displayName = "#${env.BUILD_NUMBER} ${env.RELEASE_VERSION}"
        }
    }
}

Keep those blocks to a few lines. When one grows past a screen, it belongs in a shared library.

Debugging without twenty commits

Three techniques save an enormous amount of time.

Replay. Open a finished build, click Replay, edit the pipeline text and run it again. It uses the edited script without touching the repository. Once it works, commit the change.

The linter. Validate syntax before pushing:

curl -s -X POST -F "jenkinsfile=<Jenkinsfile" \
  https://jenkins.example.com/pipeline-model-converter/validate

Add it as a pre-commit hook or a CI check and the class of "typo discovered after the build queued for eight minutes" disappears.

The snippet generator. Every Jenkins instance serves /pipeline-syntax/, which builds a correctly formed step invocation from a form, using the plugins actually installed on that controller. It is the fastest way to find the parameter names of an unfamiliar step.

Beyond those, keep pipelines short. If your Jenkinsfile is 400 lines, the logic belongs in a shell script or a library that you can run outside Jenkins — which also means a developer can reproduce the build on a laptop.

Key takeaways

  • Declarative syntax validates up front, renders well and reads clearly; write it by default and drop into script only for the lines that need it.
  • agent can be set per stage, and a Docker agent takes the toolchain from an image, which makes builds reproducible and lets projects differ without coordinating on agent packages.
  • Parameters, environment and injected Jenkins variables configure a build; credentials always come from the store through credentials() or withCredentials, in single-quoted shell strings.
  • when with beforeAgent true skips work cheaply; parallel and matrix cut wall-clock time by fanning independent stages across agents.
  • options, step-level timeout and retry make pipelines resilient to infrastructure noise without hiding real test failures.
  • post handles reporting and clean-up on every outcome, and fixed/regression produce notifications people actually read.
  • Extract repetition into a versioned shared library, pin it by tag, and unit test the code in src/ like any other Groovy.
  • Use Replay, the declarative linter and /pipeline-syntax/ instead of pushing speculative commits to a branch.

Sources and further reading

Frequently asked questions

Declarative or scripted — which should I write?
Declarative, unless you have a specific reason not to. It validates before the build starts, gives clear error messages, renders properly in Blue Ocean and the stage view, and supports options, when and post directives that scripted pipelines have to hand-roll. When you hit a genuine limit, drop into a script block for those few lines rather than rewriting the whole file as scripted.
Why does my pipeline fail with a serialisation error?
Pipeline code runs on the controller and is checkpointed after every step so a build survives a restart. Local variables must therefore be serialisable, and non-serialisable objects such as a JsonSlurper result or a matcher cannot stay in scope across a step. Wrap that logic in a method annotated @NonCPS, or set the variable to null before the next step runs.
How do I stop a stage from failing the whole build?
Wrap it in catchError with a build result, for example catchError(buildResult: 'UNSTABLE', stageResult: 'FAILURE'). The stage is marked failed in the visualisation, the overall build becomes unstable rather than failed, and later stages still run. Use it sparingly — a stage that is allowed to fail every time is a stage nobody reads.
Where should heavy Groovy logic live?
In a shared library, not in the Jenkinsfile. Code in src/ is compiled Groovy you can unit test with JUnit and the pipeline test harness, whereas a hundred lines of logic inside a script block is untestable and duplicated across repositories. Keep the Jenkinsfile declarative and readable — it should describe what the build does, not implement it.