Per-tenant configuration and entitlements

In February 2019 a support engineer switched on bulk export for a demo. Bulk export was behind a feature flag, the flag was global, and for eleven days every tenant on the free plan had a feature that the pricing page said cost forty euros a month. Nobody noticed until we turned it off, at which point forty-one tenants noticed at once. The engineer had done nothing wrong. He had asked for a switch, and we had given him the only switch we had.

The mistake was mine, and it was a modelling mistake. A feature flag and an entitlement look identical from inside an if statement and are completely different objects from everywhere else. They have different owners, different lifetimes, different storage and different failure modes. This sheet is about keeping them apart, about where each layer of per-tenant configuration lives, and about how to get the result into an IOptionsSnapshot without querying the catalog on every call.

A flag is not an entitlement

A feature flag belongs to engineering. It exists so that code can ship before it is ready to be seen, it is switched per deployment or per percentage of traffic, and it is deleted once the feature is stable. The .NET feature management library and Azure App Configuration do this well, and nothing in this sheet replaces them. An entitlement belongs to the commercial side. It says what a specific tenant is allowed to use, and how much of it, because of the plan they are on and the exceptions someone agreed to. It is permanent, it is per tenant, it is edited by support without a deployment, and it includes numbers: 500 invoice lines, 3 users, 10 GB.

The relationship between the two runs one way. A flag can gate an entitlement, so that bulk export stays invisible for everyone until engineering says it works. An entitlement never gates a flag. And support edits entitlements; nobody outside engineering touches flags. Writing those two sentences on the wall in 2019 did more for the product than the caching layer that followed.

Three layers and where they live

Configuration for a tenant resolves through three layers, and each lives somewhere different. Product defaults live in appsettings.json with the code, because they are part of the product. Plan definitions also ship with the code, as a dictionary of plan code to entitlements, because a plan is a contract that changes with releases and deserves a test. The tenant's plan code and its overrides live in the catalog row, next to the shard name from sheet T-04, because you need them before you have opened the tenant's database and because support needs to edit them from one place.

public sealed record Entitlement(bool Enabled, int? Limit);

public sealed record Plan(string Code, IReadOnlyDictionary<string, Entitlement> Entitlements);

public sealed record TenantSettings(
    Guid TenantId,
    string PlanCode,
    int Version,
    IReadOnlyDictionary<string, Entitlement> Overrides);

What does not live in the catalog is usage. The number of invoice lines a tenant has created this month is data, it changes on every write, and it belongs in the tenant database next to the invoices. The entitlement says 500; the counter says 487; the handler compares the two. Putting the counter in the catalog was the second mistake of 2019, and it turned a tiny read-mostly table into the hottest row in the system.

Resolution and caching

The resolver is a small loop: look in the overrides, then in the plan, then in the defaults, and return the first hit. It runs once per request, in the tenant resolution middleware from sheet T-02, right after the tenant is found by host. The catalog row it needs is already in hand at that point, so the settings blob is loaded with HybridCache under a key that includes the tenant id and the row's Version, with a long expiry. When support changes an override, the catalog bumps Version, the next request builds a new key, and the old entry ages out on its own.

public async ValueTask<TenantSettings> LoadAsync(TenantRow row, CancellationToken ct)
    => await cache.GetOrCreateAsync(
        $"tenant:{row.TenantId}:settings:v{row.SettingsVersion}",
        async token => await catalog.ReadSettingsAsync(row.TenantId, token),
        new HybridCacheEntryOptions { Expiration = TimeSpan.FromHours(6) },
        tags: [$"tenant:{row.TenantId}"],
        cancellationToken: ct);

The reason I key on the version rather than evicting by tag is a detail of how HybridCache invalidation works: RemoveByTagAsync clears the distributed layer and the local memory of the server that called it, and the other servers keep their in-memory copy until it expires. With four instances behind a load balancer that means a change is visible on one server and invisible on three for as long as the local expiry. Keying on the version sidesteps the problem entirely, at the cost of one integer in the catalog row that was already being read anyway. In 2019 I had a two-level cache of my own and exactly the same bug, found by a customer who saw their new limit on every third page load.

IOptions per tenant

The last step is getting the resolved value into the shape the rest of the application expects, which is a typed options class. The trick is that IOptionsSnapshot<T> is scoped and builds its value from every IConfigureOptions<T> it can resolve from the scope, so a scoped IConfigureOptions<T> that reads ITenantContext is enough. This has worked for as long as IOptionsSnapshot has existed, which by 2019 was already three years, a fact I discovered after writing a custom options factory that did the same thing worse.

public sealed class TenantInvoiceOptionsSetup(ITenantContext tenant, PlanCatalog plans)
    : IConfigureOptions<InvoiceOptions>
{
    public void Configure(InvoiceOptions options)
    {
        var limit = plans.Resolve(tenant.Settings, "invoices.max-lines").Limit;
        if (limit is int max) options.MaxLines = max;
    }
}

// Program.cs
builder.Services.Configure<InvoiceOptions>(builder.Configuration.GetSection("Invoices"));
builder.Services.AddScoped<IConfigureOptions<InvoiceOptions>, TenantInvoiceOptionsSetup>();
// Consumers inject IOptionsSnapshot<InvoiceOptions>. Never IOptions<T>: that one is a singleton.

The Configure call binds the product defaults first, the scoped setup overlays the tenant value, and a handler that injects IOptionsSnapshot<InvoiceOptions> sees the right number for the right tenant with no idea where it came from. The one rule to enforce in review is the last comment in that snippet: IOptions<T> and IOptionsMonitor<T> are singletons, they are built once for the process, and a service that injects either of them will get the first tenant's limit for everyone. I have an analyzer for that too.

The rule I apply

Flags in feature management, owned by engineering, deleted when done. Entitlements in the catalog row, owned by the commercial side, resolved override then plan then default, cached under a versioned key, and delivered through a scoped IConfigureOptions into IOptionsSnapshot. Usage counters in the tenant database, next to the data they count. And a switch that a support engineer can flip is an entitlement by definition, whatever the code calls it, so model it as one before the demo rather than after.

Drawn from

• Options pattern in ASP.NET Corelearn.microsoft.com
• HybridCache library in ASP.NET Corelearn.microsoft.com
• Pricing models for a multitenant solutionlearn.microsoft.com
• .NET feature management referencelearn.microsoft.com