Archiboard

Set B · Services & Boundaries

The transactional outbox with EF Core and Azure Service Bus

You cannot commit a row and publish a message in one transaction. Write the message to a table instead and let a relay carry it to the bus.

On a Thursday in November 2019 we issued sixty-one invoices and sent nobody an email about them. The invoices were in the database, correct and numbered. The InvoiceIssued messages were nowhere, because the Service Bus namespace had had a bad ninety seconds and the send had thrown after the row was already committed. We found out on the Monday, from a customer.

The code that did this was the obvious code. Save the entity, then publish the event. Two writes, to two systems, with no transaction over both of them, and a window between them in which the process can die, the network can drop, or the broker can refuse. That window is not rare. Ours was open for about four milliseconds per invoice and it caught sixty-one of them in a minute and a half.

You cannot close the window by reordering the two writes. Publish first and you get messages about invoices that do not exist, which is worse, because a consumer will act on them. You cannot close it with a distributed transaction either; Service Bus is not going to enlist in your SQL transaction, and you would not want it to if it could. The only fix is to stop having two writes.

The dual write and the gap between the two writes POST /invoices invoice handler invoices 1 service bus topic 2 step 1 committed, step 2 threw, and nothing will retry it
Fig. 1. The dual write. Two systems, two writes, no transaction across them. The hatched band is the window in which the process can die with the invoice saved and nobody told.

One transaction, two tables

The outbox pattern replaces the second write with a first-class row in your own database. The handler inserts the invoice and inserts a message, both through the same DbContext, and one call to SaveChangesAsync puts them in the same transaction. EF Core wraps a single SaveChanges in a transaction by default, so you do not need BeginTransaction for this and I would rather you did not use one; the fewer moving parts around the commit, the better.

public sealed class OutboxMessage
{
    public long Id { get; init; }                 // bigint identity, and the message id
    public Guid TenantId { get; init; }
    public required string Type { get; init; }    // "billing.invoice-issued.v1"
    public required string Payload { get; init; } // JSON
    public DateTimeOffset OccurredAt { get; init; }
    public DateTimeOffset? SentAt { get; set; }
}

// In the handler: one SaveChanges, one transaction, two tables.
db.Invoices.Add(invoice);
db.Outbox.Add(OutboxMessage.For(invoice.TenantId, new InvoiceIssued(invoice.Id, invoice.Total)));
await db.SaveChangesAsync(ct);

Either both rows land or neither does. There is no state of the world in which the invoice exists and the intention to announce it does not. That is the entire trick, and everything else on this sheet is plumbing around it.

If your handlers raise domain events on entities, put the translation from event to outbox row in a SaveChangesInterceptor rather than in each handler. It runs inside the same transaction, it cannot be forgotten, and it keeps the business code free of the word outbox. We moved ours there in 2021 after the third handler shipped without an event.

The relay

A hosted service polls the table, sends what it finds, and marks the rows. It is about forty lines and it is the only place in the system that talks to the sender.

public sealed class OutboxRelay(IServiceScopeFactory scopes, ServiceBusSender sender,
    TimeProvider clock) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken ct)
    {
        using var timer = new PeriodicTimer(TimeSpan.FromSeconds(2), clock);
        while (await timer.WaitForNextTickAsync(ct))
        {
            await using var scope = scopes.CreateAsyncScope();
            var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
            var batch = await db.Outbox.Where(m => m.SentAt == null).OrderBy(m => m.Id).Take(100).ToListAsync(ct);
            foreach (var m in batch)
            {
                await sender.SendMessageAsync(new ServiceBusMessage(m.Payload)
                {
                    MessageId = m.Id.ToString(),      // the deduplication key
                    Subject = m.Type,
                    SessionId = m.TenantId.ToString() // ordering, per tenant
                }, ct);
                m.SentAt = clock.GetUtcNow();
            }
            await db.SaveChangesAsync(ct);
        }
    }
}

Two details in there matter more than the rest. MessageId is the outbox row id, which makes the send repeatable: if the relay crashes after sending and before marking, the next pass sends the same message with the same id, and Service Bus duplicate detection drops it if the window has not passed. SessionId is the tenant, because a topic without sessions gives you no ordering at all and a topic with one session per tenant gives you the ordering you actually care about. In 2019 we had no SessionId and spent a fortnight chasing an InvoiceVoided that arrived before its InvoiceIssued.

Sequence from the command to the outbox row to the bus handler sql database relay service bus insert invoice + outbox row one commit, or none select where SentAt is null a batch of 100 send, MessageId = row id set SentAt crash here and the row is sent again the broker never appears inside the database transaction
Fig. 2. The relay carries the row to the bus after the commit. The hatched gap is the reason delivery is at-least-once: the send has happened, the row has not yet been marked.

At-least-once, and what that costs the consumer

Look at the hatched gap in the second drawing. The send succeeded and the row is not yet marked. If the relay dies there, the message goes out twice, and no amount of care in the relay removes that possibility, only shrinks it. This is the trade you are making: the outbox converts "might never be sent" into "might be sent more than once", and the second problem has a known answer while the first does not.

Duplicate detection on the queue or topic helps, because the relay resends with the same MessageId and the broker drops the copy. It is not the whole answer, though. The detection window is bounded, ten minutes by default and seven days at most, and it does nothing about a consumer that processes a message and then fails before settling it. Every consumer on the other side has to be idempotent regardless. Sheet B-05 is about that side of the wire.

Cleaning up

The outbox is a log, and logs grow. We keep sent rows for seven days, because that is longer than any redelivery window we have configured and long enough to answer "did we really send that", and we delete in batches on a schedule so that a single statement never takes a long lock.

DELETE TOP (5000) FROM dbo.Outbox
WHERE SentAt IS NOT NULL
  AND SentAt < DATEADD(day, -7, SYSUTCDATETIME());

Run that in a loop until it affects zero rows, every night, and watch the unsent count as a metric instead of the table size. The number that tells you something is broken is the age of the oldest row where SentAt is null. If that goes above thirty seconds, something is wrong with the relay, the namespace or the database, and you would like to know before a customer does.

What I would build today

The same thing, with one change of mind: in 2019 I wrote our own relay because the libraries felt heavy, and in 2026 I would look hard at NServiceBus or MassTransit first, because both ship an outbox that has survived more edge cases than mine has. If the answer is still to build it, keep it to a table, an interceptor, a hosted service and a filtered index, and resist every suggestion to make the relay clever. Its whole job is to be boring at two-second intervals for years.

Drawn from