Idempotent consumers and the duplicate message you will receive

A bank transfer of 4,182 euro was applied twice to the same invoice on a Tuesday in January 2022, and the tenant's account went into credit by exactly that amount. Nobody had sent the message twice. Our consumer had received it once, written the payment, taken longer than sixty seconds over a slow third-party call, lost its lock, and been handed the same message again by a broker doing precisely what it promises to do.

That is the sentence to keep. The broker is not broken when it delivers twice. Service Bus, Event Hubs, Kafka and every queue you are likely to put in front of a .NET worker offer at-least-once delivery, and at-least-once is a guarantee about the floor, not the ceiling. Exactly-once across a broker and your database is not something anyone can sell you, because the broker cannot see inside your transaction and your transaction cannot see inside the broker.

So you get to choose where the duplicate is absorbed. It is either absorbed in the consumer, deliberately, or it is absorbed by a customer noticing their balance is wrong.

Where the second copy comes from

There are three sources and it is worth knowing which one you are looking at, because they need different answers.

The producer sends, the acknowledgement is lost on the network, and the producer sends again. Two copies now sit in the queue with the same intent. This is the one that broker-side duplicate detection is designed for, and it works.

The consumer receives, processes, and then fails to settle: it crashes, or its lock expires because the work took longer than the lock duration, which defaults to one minute and can be set to five. The message returns to the front of the queue and another instance picks it up. Nothing on the send side can help here, because the send was fine.

The producer's relay resends after a crash between "sent" and "marked as sent". That is the hatched gap on sheet B-03, and it is the price of the outbox.

Try natural idempotency before you build anything

The best consumer is one where processing twice cannot do damage, and a surprising number of operations can be written that way if you decide early.

An update that sets an absolute value is repeatable. An update that adds to a number is not. A message that carries the resulting state ("this invoice is now Paid, at version 9") lets the consumer write an upsert; a message that carries a delta ("add 4,182 to the paid amount") forces you into bookkeeping. When I get to influence the message contract, I ask for the state and a version, and about half of the consumers on that product then need no inbox at all.

// Naturally idempotent: an absolute value guarded by a version. Runs twice, same result.
await db.Database.ExecuteSqlAsync($"""
    UPDATE billing.Invoice
    SET Status = {status}, StatusVersion = {version}
    WHERE Id = {invoiceId} AND StatusVersion < {version}
    """, ct);

Creating a row with a business key you already know is idempotent too, as long as there is a unique constraint on that key and you treat the violation as success. The pattern is the same as the inbox below; you have just stored the marker on the business data instead of next to it.

The inbox table, and why the constraint does the work

For everything else, the consumer keeps a record of what it has processed. A table with two columns that matter and a primary key across both.

CREATE TABLE billing.Inbox (
    ConsumerId  varchar(64)  NOT NULL,   -- which consumer, not which message type
    MessageId   varchar(128) NOT NULL,
    ProcessedAt datetime2(3) NOT NULL,
    CONSTRAINT PK_Inbox PRIMARY KEY CLUSTERED (ConsumerId, MessageId)
);

ConsumerId is in the key because a topic has several subscribers and each one has to track its own progress. We shipped without it in 2022 and the first consumer to process a message suppressed it for the other two, which is a bug that looks exactly like a missing subscription and took an afternoon to find.

The important part is what you do with the key, and this is where the 2022 version of the design was wrong. Checking whether the row exists and then processing is a race: two instances holding two copies of the same message can both pass the check before either commits. Insert the row and let the unique constraint arbitrate, in the same transaction as the side effect.

public async Task HandleAsync(ServiceBusReceivedMessage msg, CancellationToken ct)
{
    await using var tx = await db.Database.BeginTransactionAsync(ct);

    db.Inbox.Add(new InboxMessage("billing.payment-applier", msg.MessageId, clock.GetUtcNow()));
    await ApplyPaymentAsync(msg.Body.ToObjectFromJson<PaymentReceived>(), ct);

    try
    {
        await db.SaveChangesAsync(ct);        // marker and side effect together
        await tx.CommitAsync(ct);
    }
    catch (DbUpdateException e) when (e.InnerException is SqlException { Number: 2627 or 2601 })
    {
        await tx.RollbackAsync(ct);           // somebody already did this work
        logger.LogInformation("Duplicate {MessageId} ignored", msg.MessageId);
    }
}

Either both the marker and the payment land, or neither does. A crash halfway through leaves no marker, so the redelivery reprocesses cleanly. A concurrent duplicate hits error 2627 and rolls back. There is no window in which the work is done and the marker is missing, which was the flaw in the check-then-write version and the reason we double-applied a payment for a second time in 2023, on a different consumer, after I had already written this sheet once.

Duplicate detection is a filter, not a solution

Service Bus will discard a message whose MessageId it has seen recently on the same queue or topic. The default history window is ten minutes, the minimum is twenty seconds and the maximum is seven days, and a larger window costs throughput because every send is matched against the retained ids. It is available on Standard and Premium, not Basic.

Turn it on. It removes the producer-retry duplicates cheaply and it costs you nothing but a property on the entity. Then remember what it does not do: it operates on the send side, so a redelivery after a lock expiry sails straight past it, and it forgets everything older than the window. A message that dead-letters on Monday and gets resubmitted by an operator on Thursday is, as far as the broker is concerned, brand new.

So both mechanisms, and they answer different questions. The broker stops the same send being accepted twice. The inbox stops the same work being done twice.

Which leaves the housekeeping, and it is not optional. Inbox rows have to outlive every path by which the original message could come back: the lock duration times the maximum delivery count, plus the message time to live, plus whatever margin you want for a human resubmitting from the dead-letter queue. We keep thirty days, which is generous, and delete in batches nightly the way the outbox is trimmed on sheet B-03.

Watch two numbers. The duplicate rate, because a rise in it usually means lock durations are too short for the work you are doing rather than that anything is failing. And the row count, because an inbox that grows without bound will eventually make the insert slow, and the insert is on the hot path of every message you process.

What I would do on a new consumer

Ask first whether the operation can be made naturally repeatable, because that is free and needs no table. If it cannot, the inbox insert goes in the same transaction as the work from the first commit, never as a check before it. Enable duplicate detection with a window of an hour and stop thinking about it. And if the project is greenfield and messaging-heavy, look at NServiceBus or MassTransit before writing any of this, because both ship an inbox that has met more failure modes than yours will in its first two years.

Drawn from

• Idempotent Consumer patternlearn.microsoft.com
• Azure Service Bus duplicate detectionlearn.microsoft.com
• Message transfers, locks and settlementlearn.microsoft.com