Load testing a tenant-shaped system

In February 2024 we ran a load test that passed comfortably at 800 requests per second, shipped the release, and watched one customer's dashboard time out for two hours the following Monday. The test had used five hundred virtual users, each with their own tenant, each with the same twelve rows of seeded data, each doing the same four things. Production had forty-three tenants, one of which held two million rows and generated more database work on its own than the other forty-two combined.

The test was not wrong about throughput. It was wrong about shape. A multi-tenant system fails at the shape of its load long before it fails at the volume, because the failure mode you care about is one tenant consuming a share of a shared resource that the others needed. Uniform load can never produce that, by construction. It is the one thing a synthetic test will not show you unless you build it in on purpose.

So the first question on a load test sheet is not how many users. It is which tenants, how big, and doing what.

Load that looks like your tenants

Go and look at the real distribution before writing a line of script. Every B2B product I have worked on has the same broad shape: a small number of tenants generating most of the work, a long tail generating almost none, and one or two outliers that are not really the same product any more. Sixty percent of the load coming from three of forty-three tenants was our number in 2024 and it has not moved much since.

The distribution is not a guess. It is a query.

-- Derive the shape from last month, not from a guess in a planning meeting.
SELECT TOP 20
    tenant_id,
    COUNT(*)                  AS requests,
    SUM(duration_ms) / 1000.0 AS total_seconds,
    CAST(100.0 * COUNT(*) / SUM(COUNT(*)) OVER () AS decimal(5,2)) AS pct
FROM request_log
WHERE occurred_at >= DATEADD(day, -30, SYSUTCDATETIME())
GROUP BY tenant_id
ORDER BY requests DESC;

Then build the test to that shape, with two changes. Weight the virtual users so the big tenants get their real share of the traffic. And seed the test tenants with data volumes that match the real ones, because a tenant with two million rows is slow for reasons that have nothing to do with request rate: index depth, a query plan that flips at scale, a report that scans instead of seeks. Ten thousand rows for everybody tests your web tier and nothing else.

Add one tenant that is deliberately worse than any real one. Twice the data of your largest customer, running your heaviest endpoint. That tenant is the one that tells you what next year looks like, and it is cheap to include once the shaping exists.

What to measure, and per what

Aggregate numbers lie in a multi-tenant system. An overall p95 of 600 milliseconds is compatible with forty tenants at 200 milliseconds and three at nine seconds, and the three are the ones who will call. So every client-side metric is grouped by tenant, which means the test script has to tag its requests with the tenant and the results have to keep that tag.

Four things go on the dashboard, and the useful test is the one where you watch all four together:

• p95 and p99 per tenant, on each of the two or three endpoints that matter.
• Database utilisation: DTU percent for an elastic pool, or CPU and log write percent for vCore. This is usually the first thing to hit its ceiling and the last thing anyone looks at.
• Queue depth and, more tellingly, the age of the oldest message. Depth tells you there is a backlog; age tells you whether the backlog is draining.
• The rate of 429 and 503 per tenant, which is your own rate limiting doing its job or the platform throttling you.

Azure Load Testing gives you the client side and, for Azure-hosted components, pulls the server-side metrics into the same dashboard through Azure Monitor. That correlation is most of the value. A latency spike at minute fourteen means nothing on its own and means a great deal next to a DTU graph that hit 100 percent at minute thirteen.

The failure I look for specifically is asymmetric degradation: the big tenant's p95 stays flat while the small tenants triple. That is the noisy neighbour, found in a test environment, on a Tuesday, by a graph rather than by a customer.

Where the script comes from

Azure Load Testing runs Apache JMeter or Locust, and nothing else. For a URL-shaped smoke test the portal will generate the script for you, which is fine for a health check and useless for this.

I now write Locust first. The script is Python, tenants and weights are ordinary data structures, and a colleague can read it. JMeter earns its place when you need a protocol that is not HTTP, since the service supports every protocol JMeter does, so testing a JDBC connection or a message queue directly is a JMeter job. We keep one of each: a Locust script for the tenant-shaped API load, and a small JMeter plan that hammers the reporting database on its own, to separate "the API is slow" from "the database is slow".

Then it runs in the pipeline, on the release candidate, with fail criteria so the answer is pass or fail rather than a graph somebody interprets on Friday.

az load test-run create \
  --load-test-resource lt-platform \
  --test-id tenant-shape-nightly \
  --test-run-id "run-$(date +%Y%m%d-%H%M%S)" \
  --description "release candidate against stamp eu3"

Set the auto stop and the fail criteria before the first run, not after the first surprise invoice. A misconfigured endpoint that returns 404 in two milliseconds will happily run a full-scale test for an hour and tell you your product is very fast.

Test on a stamp, never on production

The deployment stamps sheet (C-02) argues for stamps as a unit of blast radius. Load testing is where that pays a second time. We keep one stamp with no paying tenants on it, sized identically to a production stamp, deployed by the same Bicep module from the same pipeline. The load test runs there. Nobody has to be awake, nobody has to agree, and the numbers mean something because the pool, the queues and the instance counts are the ones customers actually use.

A staging environment that is a smaller SKU tells you very little. Azure SQL performance is not linear in the way you hope, an S3 does not behave like a quarter of a P2, and a test that fails on a smaller pool teaches you nothing except that the pool was smaller. Either test at production size or accept that you are testing the application code and not the system.

The stamp gets torn down between campaigns. Standing it up is one pipeline run because the module already exists, which is the whole argument for having built the module.

What a passing run looks like now

Two hours at the real shape, plus a thirty minute peak at three times the busiest hour we have ever recorded. p95 per tenant inside the objective for every tenant, not on average. Pool utilisation below seventy percent at sustained load, because the headroom is what absorbs the thing nobody modelled. Queue age never above two minutes. No 503, and 429 only from the tenant we deliberately pushed over its limit.

And one number that is not in any tool: the largest tenant on the stamp costs us a known amount of database time per hour, which is what makes the pricing conversation an arithmetic problem instead of an argument. Measuring per-tenant consumption is on Microsoft's multitenancy checklist under cost, and the load test is the cheapest place to get the first version of that number, long before the tenant exists.

Drawn from

• What is Azure Load Testinglearn.microsoft.com
• Create a JMeter-based load testlearn.microsoft.com
• Create a load test with Locustlearn.microsoft.com
• Multitenancy checklist on Azurelearn.microsoft.com