Deployment stamps: cloning the whole cell

A product I was responsible for reached 340 tenants in a single deployment in the spring of 2021. The elastic pool was at its ceiling, the Service Bus namespace had been bumped to Premium for the wrong reason, and a Thursday afternoon deploy with a bad migration took every one of those 340 tenants offline for fifty minutes. The migration was my fault. The fact that one mistake reached everybody was the architecture's fault, and the architecture was also mine.

So we did what the Azure Architecture Center calls the Deployment Stamps pattern, and what I had been calling, less elegantly, "just clone the whole thing". A stamp is a complete copy of the application: compute, queues, databases, storage, secrets, monitoring. Each stamp serves a fixed group of tenants and knows nothing about the others. You scale by adding stamps, and you limit blast radius the same way. That second property turned out to be worth more than the first.

What goes inside the cell

The rule I draw at the top of every stamp sheet is this: a stamp must be able to serve its tenants if every other stamp in the world is switched off. Everything a request needs sits inside. The API and workers, the Service Bus namespace, the SQL elastic pool with one database per tenant, the storage account, the Key Vault, and its own Application Insights resource so that a noisy stamp cannot blind the others.

What stays outside is short and deliberate. The tenant directory, which maps a tenant to a stamp. Identity, because a user signs in once regardless of where their data lives. Billing. The router. These are the only components with a global view, and they are the only ones you cannot clone your way out of, so they get the most care and the least code.

The stamp is one Bicep module. The main file loops over a list and deploys the module into a resource group per stamp. That loop is the whole pattern from the infrastructure side; the first version was an ARM template with a copy loop and a two-hundred-line parameters file, and I do not miss it.

targetScope = 'subscription'

param stamps array = [
  { name: 'eu1', location: 'westeurope', capacity: 250 }
  { name: 'eu2', location: 'westeurope', capacity: 250 }
  { name: 'ch1', location: 'switzerlandnorth', capacity: 250 }
]

module stamp 'modules/stamp.bicep' = [for s in stamps: {
  name: 'stamp-${s.name}'
  scope: resourceGroup('rg-app-${s.name}')
  params: {
    stampName: s.name
    location: s.location
    designCapacity: s.capacity
  }
}]

The router in front

In 2021 the router was Traffic Manager and a DNS record per tenant, which worked and was miserable to operate. Today it is one Front Door Standard profile with a wildcard custom domain per stamp: *.eu1.example.com routes to the eu1 origin group, *.ch1.example.com to ch1. A managed wildcard certificate per stamp, one route per stamp, and onboarding a tenant touches nothing in Front Door at all. The Architecture Center calls this the stamp-based subdomain scenario and it is the one I would pick every time, unless tenants insist on vanity domains, in which case you add a custom domain per tenant and watch the profile limits.

The application still checks. Every request carries a tenant, resolved in middleware, and the tenant record says which stamp it belongs to. If that is not the stamp we are running in, we refuse, because a misrouted request that silently succeeds against an empty database is the worst kind of bug. In .NET 9 that check is eight lines.

app.Use(async (ctx, next) =>
{
    var tenant = ctx.RequestServices.GetRequiredService<ITenantContext>();
    var stamp = ctx.RequestServices.GetRequiredService<IOptions<StampOptions>>().Value;

    if (!string.Equals(tenant.StampId, stamp.Id, StringComparison.Ordinal))
    {
        ctx.Response.StatusCode = StatusCodes.Status421MisdirectedRequest;
        ctx.Response.Headers["X-Expected-Stamp"] = tenant.StampId;
        return;
    }
    await next(ctx);
});

How big is a stamp

The pattern documentation says deploy at least two stamps from day one, and it is right, because a single stamp lets you hardcode assumptions you will not find until the second one exists. We found four of them in the first week: a queue name without a stamp prefix, a storage container shared by accident, an App Insights key in a config file, and a cron job that assumed it was the only one.

Capacity is a proxy number, and ours was tenants. We load-tested one stamp until the pool's DTU graph stopped being funny and called it 250 tenants, then set a rule that the directory stops placing new tenants on a stamp at 240 and a new stamp is cut at that point. Bin packing, not round robin, because a half-empty stamp costs the same as a full one and the fixed floor of a stamp is real money; the cost sheet on this site (C-05) goes into that.

Moving a tenant between stamps

This is the part I described in 2021 and had not done. We have now done it twenty-one times, and it is a runbook, not a feature. Freeze writes for the tenant with a flag in the directory, which the middleware honours by returning 503 with a Retry-After. Export the tenant database as a bacpac, import it into the target pool, copy the blob container, replay any queue messages that arrived during the freeze. Flip the stamp id in the directory. Unfreeze. Verify with a smoke test that reads back the tenant's three most recent documents. Delete the source database a week later, not the same day.

For a tenant with three years of history the whole thing takes about forty minutes, and thirty of those are the bacpac. We tried to make it faster with geo-replication into the target pool and it worked, but the runbook grew to two pages and the gain was fifteen minutes on an operation we do twice a month. I put the simple version back.

Regional stamps for residency

Most customers never asked where West Europe was. One did, and their contract said the data stays in Switzerland. Under this pattern that requirement costs a parameter: one more entry in the stamps array with a different location, a resource group in Switzerland North, and a Front Door origin group pointing at it. The Bicep module is the same file.

What takes care is everything that leaves a stamp without asking. Backups default to geo-redundant storage, which would put a copy of Swiss data in a paired region, so the Swiss stamp uses zone-redundant backups. Application Insights resources are regional and stay put, but the Log Analytics workspace behind them must also be in the region, not the shared one. Front Door terminates TLS at an edge outside Switzerland; the contract lawyers accepted that after a conversation I would rather not repeat. Draw every arrow that leaves the stamp and ask where it lands.

What I would draw today

Two stamps on day one, sized by a number you measured, with the tenant directory as the only global state and a router that never needs touching to add a tenant. I would resist the temptation to build a backplane between stamps; a runbook and a script have moved twenty-one tenants without one. And I would put the residency stamp in the plan before the first customer asks for it, because by the time they ask, the answer needs to be a parameter and not a project.

Drawn from

• Deployment Stamps patternlearn.microsoft.com
• Use Azure Front Door in a multitenant solutionlearn.microsoft.com
• Tenancy models for a multitenant solutionlearn.microsoft.com
• Azure Front Door overviewlearn.microsoft.com