Archiboard

Set C · Cloud & Operations

Bicep and the environments ladder

One module, four parameter files, and a what-if that a human actually reads. Everything else is how environments drift apart.

In the spring of 2016 the acceptance environment of a product I was responsible for had been different from production for eight months and nobody knew. Someone had added a firewall rule on the acceptance SQL server by hand during a debugging session, and the ARM template, all 2,400 lines of JSON, was deployed in incremental mode, so the rule survived every release. The customers doing acceptance testing were testing something production would never be. We found out when a release that passed acceptance failed in production on exactly that rule, at nine on a Tuesday morning.

That is the problem the environments ladder is meant to solve, and no tool solves it on its own. Bicep, when it arrived, made the templates readable, and that was worth the migration by itself. Readable is not the same as identical. What kept the environments the same was a rule we adopted in 2016 and have not changed since: one template, one parameter file per rung, and nothing touched by hand above dev. What-if came later and turned the third part of that rule into something a pipeline could enforce rather than something I had to believe.

Four rungs and what each one is for

Dev is for developers. It is cheap, it scales to zero, it may be destroyed any night, and it is where the module is first deployed after a change. Test is for the pipeline: it runs the integration tests against real Azure resources and nothing else, and it is allowed to be broken for an hour. Acceptance is production-shaped. Same SKUs, same networking, same stamp module, smaller instance counts, and it is where customers try a release before it reaches them. Production is two stamps, and nothing is deployed there that did not go through the three rungs below.

The rungs differ in size and in nothing else. That sentence is the whole discipline. If acceptance has a public SQL endpoint and production has a private one, acceptance is not testing production, it is testing a different system that happens to run the same code.

The ladder of environments with the same Bicep module deployed at each rung modules/stamp.bicep one module, four parameter files dev dev.bicepparam B1, one stamp, min 0 test test.bicepparam P0v3, one stamp what-if, approval acceptance acc.bicepparam P1v3, one stamp, prod SKUs what-if, approval production prod.bicepparam P1v3 x3, two stamps done by hand, dated: DNS, Entra apps, secrets the same module at every rung; only the parameter file changes
Fig. 1. Four rungs, one module. The parameter file is the only thing that changes between them, and the two upper gates are where a human reads the what-if before anything runs.

Modules per stamp

The stamp module composes everything a stamp needs: the plan and the app, the workers, the SQL server and pool, the Service Bus namespace, Key Vault, Application Insights. Each of those is a smaller module in the same repository, and the stamp module is the only thing the environment files ever reference. The main file sits at subscription scope, creates a resource group per stamp and deploys the module into it, looping over an array that in dev has one entry and in production has two.

targetScope = 'subscription'

param env string
param stamps array
param sku object

resource rg 'Microsoft.Resources/resourceGroups@2024-03-01' = [for s in stamps: {
  name: 'rg-app-${env}-${s.name}'
  location: s.location
}]

module stamp 'modules/stamp.bicep' = [for (s, i) in stamps: {
  name: 'stamp-${env}-${s.name}'
  scope: rg[i]
  params: {
    env: env
    stampName: s.name
    location: s.location
    planSku: sku
  }
}]

Once a module has stopped changing weekly, it goes into a private registry in Azure Container Registry with a version tag, and the environments reference br/platform:stamp:1.14.0 instead of a relative path. That is what lets production run 1.14 while dev runs 1.15 for a day, and it is also what makes a rollback a one-line change in a parameter file rather than a git archaeology exercise.

Parameters versus environment files

Anything that differs by environment is a parameter of the main file, and the values live in one .bicepparam per rung: SKU names, instance counts, the stamps array, the location, the log retention. Nothing else. If you find yourself adding an if (env == 'prod') inside a module, the thing that differs should have been a parameter, and the condition is a second environment hiding inside the first. The 2016 version used JSON parameter files, which could not hold expressions and could not be validated against the template until the deployment failed; the Bicep format links to the template with using and the editor tells you when a parameter no longer exists.

using '../main.bicep'

param env = 'acc'
param sku = { name: 'P1v3', capacity: 1 }
param stamps = [
  { name: 'eu1', location: 'westeurope', capacity: 250 }
]

Secrets are not in these files. A parameter file is plain text in git; connection strings and keys are read from Key Vault with getSecret at deployment time or, better, never exist because the apps use managed identity. The one secret I could not remove in 2025 was a third-party payment gateway's API key, and it is a Key Vault reference in the file, not a value.

What stays manual

Not everything belongs in the module, and pretending otherwise produces modules that only work once. The subscription itself. The DNS delegation at the registrar. The TXT record that Front Door needs to validate a custom domain the first time. The Microsoft Entra application registrations, which Bicep can now do through the Graph extension and which I still do by hand because they are created once and changed never, and I would rather a human do the one-time thing than maintain automation for it. The break-glass account. The first secret in the first Key Vault.

Each of these has a line in a file called manual.md at the root of the infrastructure folder, with the date it was done, who did it, and what to look at if it needs doing again. The file is short, and the rule is that anything on it that gets done a third time gets automated.

The pipeline and the what-if gate

The GitHub Actions workflow has five jobs. Build runs bicep build and the linter on every pull request. What-if runs against dev on every pull request and posts the summary as a comment, which is where most drift is caught, because a developer sees a change they did not expect while the context is fresh. Deploy-dev runs on merge, then the integration tests against test. Acceptance and production are GitHub environments with required reviewers, and the job that asks for the review shows the what-if for that rung and refuses to continue if it contains a deletion.

  whatif-acceptance:
    runs-on: ubuntu-latest
    environment: acceptance
    permissions: { id-token: write, contents: read }
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: what-if
        run: |
          az deployment sub what-if \
            --location westeurope \
            --parameters infra/env/acc.bicepparam \
            --no-pretty-print > whatif.json
          if jq -e '.changes[] | select(.changeType == "Delete")' whatif.json > /dev/null; then
            echo "what-if reports a Delete; refusing to continue"; exit 1
          fi

What-if has known noise: properties that the service defaults and the template omits show up as changes, and anything that depends on reference() or a secure parameter cannot be evaluated. Live with it. A reviewer who has read forty what-ifs knows the noise by shape and sees the real change in seconds. A reviewer who has read none will approve anything, and that is a process problem the tool cannot fix. Azure DevOps pipelines do the same thing with a stage approval; we moved for the OIDC login and the pull request comments, not because one is better at Bicep than the other.

The trade-off I would make today

I would keep the four rungs even for a small product, and pay for an acceptance environment that looks like production, because the alternative is discovering the difference in production. I would put every environment-specific value in a .bicepparam and refuse conditions on the environment name inside modules. And I would keep the Delete gate strict even though it blocks a legitimate teardown twice a year, because both times someone read the what-if, said "yes, that one is intended", and typed the override on purpose.

Drawn from