Rate limiting the noisy neighbour

Early in 2022 a tenant of a regulatory news publisher I was working with wrote a synchronisation script that pulled their full document index every thirty seconds. Not maliciously; someone had set a cron expression to */30 * * * * * instead of */30 * * * * and nobody looked at it again. It ran for five days. Their own experience of the product was fine, because their requests were fast and cached. Everyone else on that elastic pool spent five days with reports that took nine seconds instead of two, and we spent three of those days looking for a memory leak that did not exist.

That is the noisy neighbour in its most common form, and it is worth saying plainly: the tenant was not attacking us and did not know. The Architecture Center puts it the same way, that individual tenants rarely intend to cause the problem, which means the fix is not detection and blocking, it is governance you apply to everyone all the time. A shared system without limits is not generous, it is just undecided about who gets to suffer.

Since .NET 7 the mechanism has been in the framework. Microsoft.AspNetCore.RateLimiting gives you four algorithms and, more usefully, partitioned limiters, which is the part that matters when the thing you are limiting is a tenant rather than an IP address.

Partition by tenant, not by IP

The default examples partition by IP address or by user identity. Neither is right for a multi-tenant web application sold to businesses. IP is wrong because a customer with two hundred employees behind one NAT gateway looks like one very busy client, and because a customer with a mobile app looks like two hundred quiet ones. User identity is wrong because the noisy neighbour is usually a service account running an integration, and because the resource you are protecting is shared per tenant, not per user.

So the partition key is the tenant id, resolved in middleware before the limiter runs (sheet T-02 covers where that resolution belongs). The limit itself comes from the tenant's plan, which means the free tier and the enterprise tier get different buckets from the same code path.

builder.Services.AddRateLimiter(options =>
{
    options.AddPolicy("per-tenant", httpContext =>
    {
        var tenant = httpContext.RequestServices
            .GetRequiredService<ITenantContext>();

        return RateLimitPartition.GetTokenBucketLimiter(
            partitionKey: tenant.TenantId,
            factory: _ => new TokenBucketRateLimiterOptions
            {
                TokenLimit = tenant.Plan is Plan.Pro ? 200 : 20,
                TokensPerPeriod = tenant.Plan is Plan.Pro ? 100 : 10,
                ReplenishmentPeriod = TimeSpan.FromSeconds(10),
                QueueLimit = 0
            });
    });
});

One warning that the documentation gives and that I have seen ignored twice: a partition key that comes from unbounded user input is a memory exhaustion bug. Each distinct key creates and caches a limiter. A tenant id from your own database is bounded and safe. A header value that a client can set to anything is not.

Token bucket or sliding window

Four algorithms ship in the box: fixed window, sliding window, token bucket and concurrency. In practice you will use two of them.

Token bucket is the one I reach for on public APIs. A bucket holds up to TokenLimit tokens, each request takes one, and TokensPerPeriod are added back every ReplenishmentPeriod. It expresses the thing customers actually want, which is a sustained rate plus a burst allowance: 100 requests per 10 seconds sustained, with a bucket of 200 so a page that fires twenty parallel calls on load does not fail. Sliding window gives you a smoother count over a period without the edge effect of a fixed window, where a client can send a full window's worth at 11:59:59 and another full window at 12:00:00. It is the better choice when you are limiting something whose cost per call is uniform and where bursts are not legitimate.

Concurrency is the underrated one. It limits requests in flight rather than requests per period, and for an endpoint that generates a PDF or runs a report it is a far better fit than any rate, because what exhausts the database is four simultaneous heavy queries, not forty light ones spread over a minute. We run a token bucket globally per tenant and a concurrency limiter of two on the report endpoints, and the second of those has prevented more incidents than the first.

Saying no properly

A 429 with no Retry-After header is a rude answer to a reasonable question. The client cannot tell whether to come back in one second or in five minutes, so it guesses, usually badly, and the guess is what turns your throttle into a retry storm. The limiter knows the answer; the lease carries the metadata and you have to copy it onto the response.

options.OnRejected = async (context, ct) =>
{
    if (context.Lease.TryGetMetadata(MetadataName.RetryAfter, out var retryAfter))
    {
        context.HttpContext.Response.Headers.RetryAfter =
            ((int)retryAfter.TotalSeconds).ToString(NumberFormatInfo.InvariantInfo);
    }

    context.HttpContext.Response.StatusCode = StatusCodes.Status429TooManyRequests;
    await context.HttpContext.Response.WriteAsync(
        "Rate limit for this tenant reached.", ct);
};

Log the rejection with the tenant id and the endpoint, then put it on a dashboard. A tenant hitting their limit occasionally is the system working. The same tenant hitting it for six hours is a sales conversation or a bug in their integration, and either way somebody should call them before they call you.

The middleware counts per instance

Here is the part the 2022 version of this sheet did not say. The limiter lives in the process. Run six instances behind Front Door and a tenant with a 100 per 10 second limit can send roughly 600, because each instance keeps its own bucket and none of them talks to the others. The limits are therefore soft, and their real value is cutting the worst case by a factor rather than enforcing a contract to the request.

For most products that is enough, and I would leave it there. A tenant that meant to send 100 and sends 600 is still not sending 20,000. If you genuinely need a hard number, the counter has to live somewhere shared, which means Redis and a round trip on every request, and the round trip has to be on the fast path of everything you serve. I have done it once, for a metered API where the number was the invoice, and I would not do it for anything else. The cheaper approximation is to divide the intended limit by the instance count and accept that a scale-out event changes your effective limits, which is ugly but honest.

Azure API Management sits in front of some products and does enforce limits centrally, which is a fine answer if you already run it. Buying a gateway to solve this problem alone is not.

What the limit is really protecting

Not the web tier. The web tier scales out in ninety seconds and a request that costs three milliseconds of CPU is not what hurt anyone. What hurt everyone in the story at the top of this sheet was a shared elastic pool going to its DTU ceiling, and the useful limits are the ones expressed in terms of that resource: how many report queries per tenant, how many bulk imports at once, how many rows a single API call may return.

So I size limits backwards from the database, not forwards from the API. Measure what one call of each expensive kind costs in DTU or vCore seconds, decide what share of the pool a single tenant may hold at peak, and set the numbers so that the worst tenant at their limit still leaves room for everyone else. That measurement is also what tells you your cost to serve per tenant, which the Architecture Center guidance on measuring consumption goes into properly and which most teams do not do until pricing becomes painful.

Today I ship three things on any shared-pool product: a per-tenant token bucket sized from the plan, a concurrency limiter of two or three on the endpoints that generate documents or reports, and a hard cap on result set size. The third one is not rate limiting at all, and it has saved more pools than the other two combined.

Drawn from

• Rate limiting middleware in ASP.NET Corelearn.microsoft.com
• Rate Limiting patternlearn.microsoft.com
• Noisy Neighbor antipatternlearn.microsoft.com
• Measure consumption in a multitenant solutionlearn.microsoft.com