Durable Functions Concurrency: Gate the Queue, Not the Host

A batch of customers landed on the queue at once. Every message got picked up, every message started an orchestration, and every orchestration began building a SharePoint site.

Then I watched them all crawl forward together. Site 1 finished step three. Site 2 finished step three. Site 40 finished step three. Nothing was stuck, nothing failed, and nothing finished either. The Durable runtime was faithfully round-robining between dozens of instances, one activity at a time, and the wall-clock distance from “message arrives” to “customer has a site” stretched out to something I’d be embarrassed to put in a status mail.

The fix I reached for first was the obvious one. It’s also the one that doesn’t work.

host.json Limits Are Not a Concurrency Limit

My host.json already had this:

"extensions": {
  "durableTask": {
    "maxConcurrentActivityFunctions": 3,
    "maxConcurrentOrchestratorFunctions": 5
  }
}

Five orchestrators. So at most five sites at a time, right?

Wrong. Those settings cap how many orchestrator and activity functions a worker holds in memory at once. An orchestration only occupies a slot while it’s actually processing an event. The moment it awaits an activity, it unloads and stops counting. Microsoft’s own performance guidance says it in one sentence: an orchestration that’s waiting for an activity to finish doesn’t count toward the throttle, and even millions of instances in Running state can sit there while only the ones in memory count.

My orchestrations spend nearly all their time awaiting activities that call SharePoint. So the effective limit on simultaneous site builds was: however many messages arrive. The throttle throttles episodes, not work.

The second half of the same doc page is worth having tattooed somewhere: concurrency throttles apply per worker and don’t limit total system throughput. Even if the numbers did what I thought, they’d only do it one instance at a time.

The Throttling You Don’t See

Here’s the part that made this more than an impatience problem. Every one of those parallel site builds hits SharePoint as the same app registration, and SharePoint’s application limits are counted per app per tenant across Graph, CSOM and REST together. Forty sites building at once is forty times the request rate against one bucket.

And I never saw a single 429. Not one. CSOM’s retry helpers and the Graph client’s retry handler both catch throttling responses, honour Retry-After, sleep, and try again. From the outside, that isn’t an error, it’s just slow. What I actually observed was retries climbing and durations stretching, which reads like “SharePoint is having a bad day” rather than “I am the bad day”.

That’s the trap: the SDKs are good enough at absorbing throttling that you can be deep into it without a single failed request to point at. Microsoft’s own first line of advice for avoiding it is “reduce the number of concurrent requests”, which is exactly the knob I thought host.json was giving me.

The Durable Functions Concurrency Gate

So the limit has to live where the work is started, not where it runs. The Service Bus subscriber asks a gate for permission before scheduling anything:

internal async Task<bool> HasCapacityAsync(DurableTaskClient client, CancellationToken cancellationToken = default) {
    var query = new OrchestrationQuery {
        Statuses = [OrchestrationRuntimeStatus.Running, OrchestrationRuntimeStatus.Pending],
        InstanceIdPrefix = InstanceIdPrefix,
        CreatedFrom = DateTimeOffset.UtcNow - LookbackWindow,
        FetchInputsAndOutputs = false
    };

    var cap = _settings.MaxConcurrentOrchestrations;
    var active = 0;

    // Stop counting at the cap; never enumerate the full instance history.
    await foreach (var _ in client.GetAllInstancesAsync(query).WithCancellation(cancellationToken)) {
        if (++active >= cap) {
            logger.LogInformation("Provisioning at capacity ({Active}/{Cap} orchestrations active).", active, cap);
            return false;
        }
    }

    return true;
}

What’s happening here?

1. The query counts instances that are actually occupying the system: Running and Pending. Completed, failed and terminated instances are irrelevant to capacity.
2. InstanceIdPrefix is why the deterministic instance IDs from the dedupe design pay off twice. Because every provisioning instance is named provision-customer-{id}, the prefix is a free filter that separates provisioning from every other orchestration in the task hub.
3. CreatedFrom bounds the scan to the last seven days. The Azure Storage backend doesn’t index instances by ID prefix, so without a time bound the count walks more history every week the app lives.
4. The loop returns the moment it reaches the cap. It’s a “do we have room” question, not a “how many are there” question, and the cheap version of that answer is the one that stops counting early.
5. FetchInputsAndOutputs = false keeps the payloads out of it. Nothing here needs to know what the orchestrations are doing.

My cap is 2, in config rather than in code, and two sites in parallel finish in about three minutes. That’s the number that made the difference: two runs that complete beat forty that all progress.

Deferral Is Not Abandonment

Being at capacity doesn’t mean rejecting the message. It means putting it back with a delay:

private static readonly TimeSpan[] DeferDelays = [
    TimeSpan.FromMinutes(1),
    TimeSpan.FromMinutes(2),
    TimeSpan.FromMinutes(5),
    TimeSpan.FromMinutes(10)
];

// Publish-then-complete: caller must schedule the clone BEFORE completing the original.
// A crash between the two leaves a harmless duplicate rather than a lost message.
internal async Task DeferAsync(ServiceBusReceivedMessage original, int deferCount, CancellationToken cancellationToken = default) {
    var delay = DeferDelays[Math.Min(deferCount, DeferDelays.Length - 1)];

    var jitter = TimeSpan.FromSeconds(Random.Shared.Next(_settings.DeferJitterSeconds + 1));
    var scheduledTime = DateTimeOffset.UtcNow.Add(delay).Add(jitter);

    var clone = new ServiceBusMessage(original.Body) {
        ContentType = original.ContentType,
        Subject = original.Subject,
        ScheduledEnqueueTime = scheduledTime
    };
    clone.ApplicationProperties[DeferCountProperty] = deferCount + 1;
    clone.ApplicationProperties[FirstSeenUtcProperty] = GetFirstSeenUtc(original).ToString("o");

    await _sender.ScheduleMessageAsync(clone, scheduledTime, cancellationToken);
}

Why a scheduled clone instead of abandoning the message? Because abandoning gives you Service Bus’s redelivery schedule, which is seconds, and counts toward MaxDeliveryCount. A message that waits an hour for a slot would dead-letter itself on the broker’s timetable rather than mine. Scheduling a fresh copy with ScheduledEnqueueTime means the wait is mine to define, and the delivery count never moves.

The delays escalate 1, 2, 5, 10 minutes and then hold at 10, with up to 30 seconds of jitter on top. The jitter is not decoration: without it, every message deferred in the same second wakes in the same second and stampedes the gate together, which is the same burst I was trying to prevent, just on a timer.

Two application properties ride along. DeferCount picks the next delay. FirstSeenUtc is copied across every bounce so the log line can say how long this customer has genuinely been waiting, not how long since the last hop.

And the ordering is deliberate: schedule the clone, then complete the original. Crash in between and you get a duplicate message, which the dedupe checks make a no-op. Do it the other way and a crash loses a customer’s site request entirely.

Gotchas

• The cap is soft. The instances query is eventually consistent, so two workers can both pass the gate at the same moment. Occasional N+1 is fine and expected. If you need an exact count, a Durable entity acting as a semaphore is the tool, and it costs you a lot more moving parts.
• Gate last, after the dedupe checks. A redelivered message for a customer that already has a site should short-circuit, not queue up for a slot it will never need.
• There is no deadline, on purpose. A message with no free slot waits indefinitely. Any age limit would eventually dead-letter customers whose only crime was arriving late in a backlog, and a large bulk run legitimately takes days at a cap of two.
• A wedged instance eats a slot forever. One orchestration stuck in Running permanently reduces the cap, and with no deadline the queue behind it just waits. Alert on deferral volume and on the oldest FirstSeenUtc, because nothing else will surface it - the same log search alert that catches failed orchestrations is where those queries belong.
• FIFO is gone. A deferred message is completed and republished, so it gets a new sequence number and loses its place in line. Harmless when customers are independent of each other. Not harmless if something downstream assumes order.

Wrapping Up

maxConcurrentOrchestratorFunctions sounds like a concurrency cap and isn’t one: it limits orchestrator episodes in memory per worker, and an orchestration waiting on an activity doesn’t count. If you need “only N of these at a time”, the only place to enforce it is before the orchestration is scheduled. Count the non-terminal instances by prefix, and when you’re full, hand the message back to the queue with a future delivery time and some jitter. Rule of thumb: throttle where work is admitted, not where it runs.

Jeppe Spanggaard

Software developer & co-founder of Pinksky, working in C#/.NET, Azure, Microsoft Graph and SharePoint. I write these posts in Danish and use AI to help with the English - the code, the numbers, and the mistakes are all mine. More about me →