Archiboard

Set R · Resilience & Scale

Caching layers for a web application: in-process, Redis, and HybridCache

Two layers, one key format. If the tenant id is not in the key, the cache is a data leak with good response times.

In October 2015 a support ticket arrived that said, more or less, "these are not our numbers". One tenant was reading another tenant's price list. It lasted about ninety seconds, it affected two users, and it was entirely my fault: the cache key was pricelist_7, the repository behind it filtered by tenant, and the MemoryCache.Default entry in front of it did not. Two tenants happened to have a price list with id 7, and the first one to ask warmed the entry for both.

Nothing was lost, nothing was written, and the customer was gracious about it. But I have never since drawn a caching layer without writing the key format on the sheet first, in mono, with the tenant segment at the front. The cache is the one component in a multi-tenant system that sits outside your data access layer and therefore outside every safety net you built there. Query filters do not reach it. Row-level security does not reach it. The key is the whole fence.

The rest of this sheet is about the two layers you actually need, why the in-process one is both the fastest and the most annoying, and how much of the code I wrote in 2015 has since been deleted because the framework grew a better version of it.

The two layers and what each is for

The first layer is IMemoryCache, in the process, holding real objects with no serialisation. A hit costs a dictionary lookup, somewhere around fifty nanoseconds. It is the only layer fast enough to sit in front of something you read on every request, like the tenant record itself or a feature flag set. Its problem is that it is per instance. Run six instances and you have six caches that disagree with each other, and none of them knows when another one changed something.

The second layer is IDistributedCache, which in practice means Redis. A hit costs a network round trip plus deserialisation, so call it half a millisecond inside the same region, which is roughly ten thousand times slower than the memory cache and still roughly forty times faster than the query it replaced. It is shared, so all instances see the same value, and it survives a deployment, which matters more than people expect: a rolling restart with only in-process caches means every instance starts cold and the database takes the full load at exactly the moment you are also swapping containers.

You want both, in that order, and that combination has a name and now an implementation. Before .NET 9 you wrote it yourself. I wrote it three times, for three products, and the third one was not better than the first.

// .NET 10. In 2015 this was MemoryCache.Default in front of a hand-written
// Redis wrapper and a lock. HybridCache arrived in .NET 9.
builder.Services.AddStackExchangeRedisCache(o =>
{
    o.Configuration = builder.Configuration.GetConnectionString("Redis");
    o.InstanceName = "app:";
});

builder.Services.AddHybridCache(o =>
{
    o.MaximumPayloadBytes = 256 * 1024;
    o.DefaultEntryOptions = new HybridCacheEntryOptions
    {
        Expiration = TimeSpan.FromMinutes(10),           // L2, shared
        LocalCacheExpiration = TimeSpan.FromSeconds(30)  // L1, per process
    };
});

The two expirations are the interesting part of that block. The local one is short because it is the one you cannot invalidate across the fleet; the shared one is long because it is the one that protects the database. Thirty seconds of local staleness is a product decision, not a technical one, and it is the number I would argue about in a design review rather than any of the others.

Two cache layers with a tenant-prefixed key in front of a shared database instance 1 api L1 IMemoryCache this process only, 30 s instance 2 api L1 IMemoryCache this process only, 30 s the two L1 caches never agree L2 Azure Managed Redis shared by every instance, 10 min miss in both tenant db the key, in both layers t:0042:pricelist:7:v3
Fig. 1. Both layers are addressed by the same string. The hatched prefix is the only thing that stops instance 2 handing tenant 0042 data to tenant 0051.

The key must carry the tenant

Four segments, always in the same order: tenant, entity, identifier, version. t:0042:pricelist:7:v3. The tenant first because it makes a Redis SCAN by prefix possible when you are debugging, and because a key that starts with the tenant is a key whose mistake is visible in the logs. The version last because a deployment that changes the shape of the cached object must not read yesterday's shape back; bumping v3 to v4 is a one-character migration and I have never regretted having the segment there.

Build the key in one place. Not in each service, not with string interpolation scattered over forty call sites, but in a small helper that takes the tenant from the scoped tenant context and cannot be called without it. The compiler is your only real defence here, so make the type system carry the requirement. When we finally did that properly, two years after the ticket at the top of this sheet, the refactor found two more keys with no tenant in them, neither of which had leaked, both of which would have.

public sealed class PriceListService(
    HybridCache cache, ITenantContext tenant, PriceListRepository repo)
{
    public ValueTask<PriceList> GetAsync(int listId, CancellationToken ct) =>
        cache.GetOrCreateAsync(
            $"t:{tenant.TenantId}:pricelist:{listId}:v3",
            (listId, repo),
            static (s, ct) => s.repo.LoadAsync(s.listId, ct),
            tags: [$"t:{tenant.TenantId}"],
            cancellationToken: ct);
}

The tags argument is worth the extra line. Tagging every entry with the tenant means that when a tenant is deleted, or restored, or has its data corrected by a support script, one call to RemoveByTagAsync retires everything belonging to them. Without tags you are back to keeping a set of keys per tenant, which is a second cache with its own bugs.

The stampede

Here is what happens without protection. A popular entry expires. Two hundred requests arrive in the same twenty milliseconds, all miss, all call the repository, and the database receives two hundred identical queries for something that has one answer. On a good day the plan cache absorbs it. On the day the entry was a report that takes 900 milliseconds, we watched an elastic pool go from 30 percent to its ceiling and stay there, because each wave of timeouts produced another wave of misses.

HybridCache handles this for you: for a given key, one concurrent caller runs the factory and the others wait on that result. It is the single best reason to adopt it even if you never configure a second layer, because the in-process cache alone plus stampede protection is already better than what most hand-written wrappers do. The coordination is per instance, though. Six instances means up to six concurrent factory calls, not one, and if that still hurts, the answer is a longer expiry or a background refresh rather than a distributed lock.

The 2015 version of this sheet recommended a SemaphoreSlim keyed by cache key, and I recommended it with some confidence. It worked. It also leaked semaphores for keys that were never requested again, which took me four months and a memory dump to notice.

What HybridCache does not solve

Invalidation across instances. When you call RemoveAsync or RemoveByTagAsync, the entry goes from the local cache of the server handling the call and from the shared store, but the in-memory caches of the other five servers keep their copy until it expires on its own. This is documented, it is a reasonable design, and it means your local expiration window is also your worst-case staleness window after a write. Thirty seconds is fine for a price list. It is not fine for a permission set, which is why permissions on the products I look after are cached at L2 only, with no local copy at all.

It also does not solve the thing people hope it solves, which is that some data should not be cached. Anything a user edits and immediately expects to see is a bad candidate. Anything used in an authorisation decision should be either uncached or cached with an expiry measured in seconds. And anything you cannot afford to serve stale after a support engineer fixes it by hand should not be in a cache whose contents you cannot enumerate.

What I would build today

HybridCache with Redis behind it, a local expiry of thirty seconds, a shared expiry of ten minutes, one key helper that cannot produce a key without a tenant, and tags per tenant. For the tenant record itself, which every single request reads, a longer local expiry and an explicit invalidation path when the record changes, because that one is worth the extra code.

On the Redis side, plan for Azure Managed Redis rather than Azure Cache for Redis on anything new, since the older service has a published retirement path. Put the cache in the same region as the application, use the hostname rather than an IP, and stay off the smallest tiers in production, where you share a core with strangers. A cache that times out under load is worse than no cache, because you pay the round trip and then pay the query anyway.

Drawn from