Archiboard

Set T · Tenancy & Data

Onboarding a tenant is a saga

Six steps across five systems that share no transaction. Treat it as a saga with compensations, make every step idempotent, and send the mail last.

On a Tuesday in January 2021 an onboarding failed at the custom domain step. The database had been created, the identity tenant had been created, and then the DNS verification timed out, the queue handler threw, and Service Bus did what Service Bus does: it delivered the message again. The handler started from the top. It created a second database, with a fresh GUID in the name, and a second identity tenant, and failed again at DNS. By Friday there were three databases, two welcome mails in the customer's inbox, and one customer who could not log in to any of them.

Nothing in that story was a bug in the ordinary sense. Every step worked. What we did not have was a model for what onboarding is: a sequence of writes across Azure SQL, an identity directory, DNS, a certificate service and a mail provider, none of which share a transaction with any of the others. That is a distributed transaction without a coordinator, and the pattern for it has had a name since 1987. Onboarding a tenant is a saga.

Six steps, five systems

The order matters, because each step depends on the last and because the cost of undoing them is not the same. Mine goes like this:

  1. Write the catalog row in state Provisioning, so the tenant exists and resolves to nothing.
  2. Create the database in the right elastic pool and run migrations to the current version.
  3. Create the identity tenant, the application registration and the invitation for the first administrator.
  4. Bind the custom domain: publish the verification record on our side, request the hostname binding and the managed certificate.
  5. Seed the data the tenant cannot function without: default roles, the chart of accounts, a first fiscal year.
  6. Set the catalog row to Online and send the welcome mail.

Steps one to five can be undone. Step six cannot. A mail, once sent, is in someone's inbox, and a tenant that has been Online for thirty seconds may already have a user in it. That line between five and six is the pivot of the saga, and where it sits is the single most important decision in the design.

The onboarding saga with its compensating steps and the point of no return orchestrator (Durable Functions or state row) register in catalog provision database identity tenant DNS and domain seed data online and welcome mail on failure mark failed drop database delete identity unbind domain nothing: db is gone retry until sent left of the line: undo. right of it: retry until done
Fig. 1. Every step left of the dashed line has a compensation that runs in reverse order when a later step fails. The last step has none; once the tenant is Online and the mail is out, the only correct move is forward.

Compensation, and the step you cannot undo

A compensation is not a rollback. It is a new action that puts the world back into a state you can live with, and it is allowed to be lossy. Dropping the database is a compensation for creating it. Deleting the identity tenant is a compensation for creating it, and it may leave an invitation mail in somebody's inbox, which you accept. Marking the catalog row Failed is a compensation for creating it, and it is deliberately not a delete, because a failed onboarding is something support will want to look at on Monday.

The step with no compensation defines the pivot. Everything before the pivot must be reversible; everything after it must be retryable, forever if necessary, because the alternative is a tenant that is half online. That is why I moved the Online transition and the mail into the same last step in 2023. In the 2021 version the tenant went Online after seeding and the mail was a separate step, which meant a mail failure left an online tenant nobody had told about. Now a mail failure retries with backoff until the provider recovers, and the tenant is never Online without a mail on its way.

DNS deserves its own sentence, because domain verification depends on a customer's DNS administrator, who is not in the room. The step in the saga ends when the binding is requested and the verification record is published on our side. The wait for the customer's record is a durable timer with a 72 hour timeout, and when it expires the tenant goes Online on the default subdomain and a follow-up saga picks up the custom domain later. The tenant should not fail to exist because someone in their IT department is on holiday.

Idempotent steps

The three databases in January 2021 were a naming bug. The database name was tenant-{Guid.NewGuid()}, so every attempt created a fresh one, and no attempt could tell that the previous one existed. The fix was to derive every resource name from the tenant id and to check before creating. An activity that runs twice must produce the same world as an activity that runs once, and the easiest way to get there is to make it look first.

[Function(nameof(ProvisionDatabase))]
public async Task ProvisionDatabase([ActivityTrigger] OnboardingOrder order)
{
    var name = $"t-{order.TenantId:N}";   // deterministic: a rerun finds what the first run made

    if (await sql.DatabaseExistsAsync(order.Server, name) is false)
        await sql.CreateInPoolAsync(order.Server, name, order.PoolName);

    await migrator.MigrateAsync(order.Server, name);   // EF migrations are idempotent by design
}

The same rule applies to the compensations. Dropping a database that is already gone is a success, not an error, and deleting an identity tenant that was never created is a no-op. An orchestrator that retries a compensation and gets a 404 must treat it as done, or the saga will sit in a Compensating state for the rest of its life. The seed step is idempotent by construction if seeding is a migration; if it is a script, give it a marker row and check it.

Durable Functions or a state row

There are two honest ways to run the orchestrator. The first is Durable Functions in the isolated worker model, which gives you a checkpointed orchestration, retry policies per activity, durable timers for the DNS wait, and a history you can query when a customer asks why their onboarding is stuck. The orchestrator function must be deterministic, so all the real work lives in activities, and the compensation logic is a stack you unwind in a catch.

[Function(nameof(OnboardTenant))]
public static async Task OnboardTenant([OrchestrationTrigger] TaskOrchestrationContext ctx)
{
    var order = ctx.GetInput<OnboardingOrder>()!;
    var undo = new Stack<Func<Task>>();
    var retry = TaskOptions.FromRetryPolicy(new RetryPolicy(5, TimeSpan.FromSeconds(10), 2.0));
    try
    {
        await ctx.CallActivityAsync(nameof(RegisterInCatalog), order, retry);
        undo.Push(() => ctx.CallActivityAsync(nameof(MarkCatalogFailed), order.TenantId));
        await ctx.CallActivityAsync(nameof(ProvisionDatabase), order, retry);
        undo.Push(() => ctx.CallActivityAsync(nameof(DropDatabase), order.TenantId));
        await ctx.CallActivityAsync(nameof(CreateIdentityTenant), order, retry);
        undo.Push(() => ctx.CallActivityAsync(nameof(DeleteIdentityTenant), order.TenantId));
        await ctx.CallActivityAsync(nameof(BindCustomDomain), order, retry);
        undo.Push(() => ctx.CallActivityAsync(nameof(UnbindCustomDomain), order.TenantId));
        await ctx.CallActivityAsync(nameof(SeedData), order, retry);
        await ctx.CallActivityAsync(nameof(GoOnlineAndWelcome), order, retry);
    }
    catch (TaskFailedException)
    {
        while (undo.Count > 0) await undo.Pop()();
        throw;
    }
}

The second way is a state row in the catalog and a worker that advances it one step per run. It is less code than people expect, it keeps onboarding inside the same deployable as everything else, and it is what I ran from 2021 until early 2024. The row carries the last completed step, an attempt counter, a status, and the last error, and the worker is a switch on the step with the same compensations in the same order. It loses the durable timer and the history, and you write your own backoff. It is the right answer for a team that does not already run Functions and has fewer than four steps.

CREATE TABLE catalog.Onboarding (
    TenantId   uniqueidentifier NOT NULL PRIMARY KEY,
    Step       tinyint       NOT NULL,   -- last step completed, 0 to 6
    Attempt    int           NOT NULL DEFAULT 0,
    Status     tinyint       NOT NULL,   -- 0 running, 1 done, 2 compensating, 3 failed
    LastError  nvarchar(max) NULL,
    UpdatedAt  datetime2     NOT NULL
);

Both need the same idempotent activities, and moving from one to the other in 2024 was a week of work precisely because the activities did not change. The 2021 version was written on .NET 5 against the in-process Durable Functions model with IDurableOrchestrationContext; that model leaves support in November 2026, and the isolated worker's TaskOrchestrationContext above is the one to write against now.

What I ship now

An orchestration in Durable Functions on the isolated worker, six activities that derive every resource name from the tenant id and check before they create, a compensation stack unwound in reverse when anything left of the pivot fails, a durable timer for the DNS wait with a fallback to the default subdomain, and the Online transition fused to the welcome mail as the one step with no way back. Three databases for one customer was the cost of learning that. It has not happened since.

Drawn from