OpenTelemetry in .NET: traces that carry the tenant id

In January 2018 a customer, one of about three hundred tenants on a product I was responsible for, called to say their overnight document runs had become slow. Not failing, slow. I opened Application Insights, found their tenant's requests in about a minute, and then hit the wall: the worker that rendered the PDFs produced telemetry with no tenant on it at all. The API knew who the tenant was. The message on the queue did not say. The worker was emitting four hundred items a minute from everybody and I could not tell which ones were theirs.

We found the problem eventually (their logo was a 9 MB PNG and the renderer re-decoded it per page), but it took a day that should have taken twenty minutes. Since then the rule on every sheet I draw is short. The tenant id is on every span, in every process, and it survives the queue. Not most spans. Every span.

ActivitySource, not a vendor SDK

The instrumentation side has been stable since ActivitySource arrived in .NET 5. You create one static source per component, you call StartActivity around meaningful work, you set tags. In .NET the Activity is the span and the ActivitySource is the tracer; the names predate OpenTelemetry and were kept. Libraries only need System.Diagnostics.DiagnosticSource, which ships in the runtime, so a class library never has to reference OpenTelemetry at all. In 2018 there was no such thing and every component referenced the Application Insights SDK directly, which is why the telemetry code and the business code were the same code.

The exporting side is what changed most. The 2018 arrangement was an ITelemetryInitializer that read the tenant from HttpContext and stamped it on each item on its way out. That worked in the API and was useless in the worker, which is the whole story above. Today the exporter is the Azure Monitor OpenTelemetry distro, one package and one call, and the enrichment is an OpenTelemetry processor that runs in every process the same way, including the ones that have never heard of HTTP.

using Azure.Monitor.OpenTelemetry.AspNetCore;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .UseAzureMonitor(o =>
    {
        o.SamplingRatio = 0.2F;   // one trace in five, fleet-wide
    })
    .WithTracing(t => t
        .AddSource("App.*")
        .AddProcessor<TenantTagProcessor>());

builder.Services.AddSingleton<TenantTagProcessor>();

The connection string comes from APPLICATIONINSIGHTS_CONNECTION_STRING in the environment, never from code, and the worker's Program.cs is the same block with Host.CreateApplicationBuilder instead of WebApplication.

Tags versus baggage

This is the distinction that took me too long to respect. A tag lives on one span and is exported. Baggage lives on the context, is inherited by every child span in the process, is written into the baggage header on outgoing HTTP calls and read back on the other side, and is not exported anywhere by default. A tag is what you query. Baggage is how the value travels.

So the pattern has two halves. At the edge, once, after the tenant is resolved, set the tenant id as baggage. Everywhere else, a processor copies the baggage value onto each span as it starts. The edge code does not know about the processor and the processor does not know about HTTP.

// At the edge, after tenant resolution.
app.Use(async (ctx, next) =>
{
    var tenant = ctx.RequestServices.GetRequiredService<ITenantContext>();
    Baggage.SetBaggage("tenant.id", tenant.Id);
    await next(ctx);
});

// In every process: baggage becomes a tag on each span.
public sealed class TenantTagProcessor : BaseProcessor<Activity>
{
    public override void OnStart(Activity activity)
    {
        var tenantId = Baggage.GetBaggage("tenant.id");
        if (tenantId is not null)
        {
            activity.SetTag("tenant.id", tenantId);
        }
    }
}

Put the id in baggage, never the tenant name. Baggage goes out in a header to every HTTP dependency you call, including third parties, and a tenant name in a header to a payment provider is a conversation with a privacy officer.

Across the queue

HTTP propagation is free: the instrumentation writes traceparent and baggage headers and reads them on the other side. A queue is not free. The Azure Service Bus SDK writes the trace context into a Diagnostic-Id application property on the message and links the consumer's span to it, so the two halves of the trace connect. It does not carry baggage, and I have stopped hoping that it will.

So the producer sets tenant.id as an application property on the message explicitly, and the consumer reads it and calls Baggage.SetBaggage before starting its own activity. It is four lines on each side and it is the single most important four lines in this article. If you use a different broker, check what its instrumentation carries before assuming; I have not verified every SDK version and I write the property myself regardless.

Sampling per tenant

At two hundred requests per second, exporting everything is a bill, so the distro samples. SamplingRatio keeps a fixed fraction of traces and TracesPerSecond keeps a rate; both are set on UseAzureMonitor and the distro annotates each exported span with the ratio so that Application Insights can scale request counts back up. Start at 5 to 20 percent and alert on metrics, which are never sampled.

The per-tenant part is a small custom Sampler that wraps whatever sampler you would otherwise use and forces RecordAndSample for tenants on a watch list. The tenant id is available to it because baggage was extracted from the incoming request before the sampler runs. When a customer calls, their tenant goes on the list for an hour and every one of their traces lands in the workspace while everyone else stays at 20 percent. Two cautions: replacing the distro's own sampler means the forced traces carry no ratio annotation, so counts for watched tenants read high, and the watch list should live in configuration you can change without a deploy.

public sealed class TenantWatchSampler(Sampler inner, IWatchList watch) : Sampler
{
    public override SamplingResult ShouldSample(in SamplingParameters p)
    {
        var tenantId = Baggage.GetBaggage("tenant.id");
        if (tenantId is not null && watch.Contains(tenantId))
        {
            return new SamplingResult(SamplingDecision.RecordAndSample);
        }
        return inner.ShouldSample(in p);
    }
}

What to do about logs

Logs are the easy part once traces are right, because ILogger already stamps the trace id and span id on every record and the distro exports them next to the spans. The tenant id gets on logs the same way it gets on spans: a scope opened in the middleware, or a log enricher that reads baggage. Filtering the Log Analytics table by tenant.id and joining to the trace id is then a two-line query.

One default to know about. When sampling is on, logs that belong to unsampled traces are dropped with them, which is correct for information-level noise and wrong for errors. The distro lets you opt out of trace-based sampling for logs, and I do that for warnings and above, so an exception for a tenant at 20 percent sampling is always kept even when its trace is not.

The trade-off I would make today

I would set the tenant id as baggage at the edge and write it onto every message by hand, accept the processor as a permanent part of every service, and sample at 20 percent with a watch list I can flip during an incident. I would give up exact request counts for watched tenants without a second thought. The alternative, which I lived with for a year, is a trace that stops at the queue and a day lost finding a 9 MB logo.

Drawn from

• Add distributed tracing instrumentation in .NETlearn.microsoft.com
• Enable OpenTelemetry in Application Insightslearn.microsoft.com
• Configure OpenTelemetry in Application Insightslearn.microsoft.com
• .NET observability with OpenTelemetrylearn.microsoft.com