Background work: hosted services, queues and the job you must not run twice

We scaled a product I was responsible for from one instance to two on a Tuesday afternoon in the autumn of 2018, for entirely sensible reasons, and on Wednesday morning eleven customers received their monthly invoice twice. The invoice run was a BackgroundService with a Task.Delay loop, ported a few months earlier from the scheduled task it used to be. It had been correct for as long as it had existed, because there had only ever been one of it. Nobody wrote down that assumption anywhere, least of all in the class that depended on it.

What makes this failure worth a whole sheet is that it is invisible in every environment where you would notice it. On your machine there is one instance. In the test environment there is one instance, because nobody scales test. It appears the first time you scale, or the first time a rolling deployment briefly runs the old and new versions together, and by then the job has been running unattended for long enough that everyone trusts it.

So the rule I now draw on every worker sheet: a scheduled job in a horizontally scaled process needs an answer to "what happens when two of these wake up at the same instant", and "we only run one instance" is not that answer. It is a note in the margin waiting to become an incident.

The two-instance problem

BackgroundService itself is fine. It is the right base class, it starts with the host, and since .NET 6 the default behaviour when ExecuteAsync throws is to stop the host, which is what you want. In .NET Core 2.1, where this sheet started, an unhandled exception in a hosted service killed that service silently while the web application carried on serving traffic, and a job could be dead for a week before anyone noticed. We lost about that long to it.

The modern shape uses PeriodicTimer, which does not drift the way a Task.Delay loop does and which cooperates properly with the stopping token.

public sealed class InvoiceRunService(
    IServiceScopeFactory scopes,
    ILogger<InvoiceRunService> log) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stopping)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromMinutes(5));

        while (await timer.WaitForNextTickAsync(stopping))
        {
            using var scope = scopes.CreateScope();
            var runner = scope.ServiceProvider.GetRequiredService<InvoiceRunner>();
            await runner.RunDueAsync(stopping);
        }
    }
}

Note the scope. A hosted service is a singleton, so it cannot take a DbContext in its constructor, and every version of this bug I have seen in a code review has been someone injecting a scoped service into a singleton and getting away with it until the second request. Create a scope per tick.

What that class still does not know is whether another copy of it is running fifty milliseconds away on another instance.

Prefer a queue to a timer

The cleanest fix is to stop having a timer. If the work is triggered by something that happened, put a message on a queue when it happens and have the workers compete for messages. Azure Service Bus and Storage queues both give you at-least-once delivery with a lock on the message while one consumer works on it, which means the "only one instance does this" property comes from the queue rather than from your code. Two instances then become a feature: the second one doubles throughput instead of doubling invoices.

This works for most background work. Send the email, generate the PDF, reindex the tenant, recalculate the balance. All of it is a reaction to an event, and all of it belongs on a queue with an idempotent handler, because at-least-once means you will get a duplicate eventually and the handler has to survive it. Idempotency is not optional here; it is the price of the queue solving your concurrency problem.

What stays on a timer is genuinely periodic work with no triggering event. Nightly invoicing on the first of the month. Purging expired sessions. Refreshing a cache of exchange rates at six in the morning. For those you still need to elect one instance, and that is what the lease is for.

The blob lease that settles it

A lease on an Azure Storage blob is an exclusive write lock with a timeout, and it is the cheapest distributed mutex you can get on Azure. The Architecture Center documents exactly this as the Leader Election pattern, with a blob lease as the sample implementation. You create a zero-byte blob whose only job is to be leased, and the first instance to acquire it is the one that runs.

var blob = container.GetBlobClient("invoice-run");
var lease = blob.GetBlobLeaseClient();

try
{
    await lease.AcquireAsync(TimeSpan.FromSeconds(45), cancellationToken: ct);
}
catch (RequestFailedException e) when (e.Status == 409)
{
    return; // another instance holds it, nothing to do this tick
}

await using var renewal = StartRenewing(lease, TimeSpan.FromSeconds(20), ct);
await runner.RunDueAsync(ct);
await lease.ReleaseAsync(cancellationToken: ct);

Two numbers matter. A lease duration must be between 15 and 60 seconds, or infinite, and you renew it while the work runs. I use 45 seconds with a renewal every 20, which tolerates one missed renewal without losing the lock. If the instance dies, the lease expires 45 seconds later and the other instance picks up the next tick, which is the whole point.

Do not use the infinite lease. It looks convenient and it means that an instance which dies holding it leaves a lock nobody can clear without a manual break, which you will discover at two in the morning on the one night the job matters. I have broken exactly one production lease by hand and the memory is enough.

Shutting down without losing work

The other half of running a job correctly is stopping one correctly. When the host shuts down, the token passed to ExecuteAsync is cancelled and the host waits for the shutdown timeout before killing the process. That timeout defaults to five seconds. Five seconds is a long time in a web request and nothing at all in a job that is halfway through a batch of two thousand invoices.

builder.Services.Configure<HostOptions>(o =>
{
    o.ShutdownTimeout = TimeSpan.FromSeconds(45);
    o.BackgroundServiceExceptionBehavior =
        BackgroundServiceExceptionBehavior.StopHost;
});

Raising the timeout only helps if the work cooperates. Pass the stopping token down through every await, check it between units of work rather than only at the top of the loop, and commit progress in batches so that being killed loses one batch and not the whole run. The pattern that has served me best is a job that records what it has already done in the database and starts by asking what is left, which makes a restart a resumption instead of a repeat. That property is worth more than any lock, because it also covers the case where the platform gives you no notice at all.

Also make sure the container has time to receive the signal. A deployment that removes the instance from the load balancer and then sends SIGTERM two seconds later has already decided the outcome, whatever your HostOptions say.

Where I would put the job today

Out of the web application. In 2018 I put the workers in the same process as the API because it was one deployment and one set of configuration, and that convenience cost us in every direction afterwards: the API scales on request rate, the workers need memory, and a job that pins a core makes p95 worse for every tenant on that instance. The two workloads have different shapes and want different scaling rules.

For periodic work, an Azure Container Apps job with the Schedule trigger and a cron expression, which runs a container, does the work and stops. No lease, because there is one execution. For work triggered by messages, an Event job with a queue scale rule, or an Azure Functions app with a queue trigger if the handler is small. The scheduled job on its own removes most of what this sheet is about, and it costs nothing when it is not running.

Where I still keep a BackgroundService in the API process is for things that belong to that process: flushing a metrics buffer, refreshing an in-memory cache of the tenant catalog, draining a channel of low-value writes. Those are per-instance by design, which means the two-instance question has an answer, and the answer is that both of them should run.

Drawn from

• Background tasks with hosted services in ASP.NET Corelearn.microsoft.com
• Leader Election patternlearn.microsoft.com
• Lease Blob (REST API)learn.microsoft.com
• Jobs in Azure Container Appslearn.microsoft.com