I watched my provisioning orchestration retry an activity ten times over roughly 25 minutes.
The error? An invalid site type string in the provisioning schema. A config mistake. Attempt one failed with “unknown site type”. So did attempt two, five seconds later. And attempt three, ten seconds after that. Exponential backoff dutifully stretched the intervals to five minutes while the outcome stayed exactly as impossible as it was at second zero.
Retrying a throttled request is smart. Retrying a typo is just failing slowly.
TL;DR
Durable Functions’RetryPolicy retries every exception the same way - it has no concept of “this will never work”. Define a dedicated exception type for permanent failures, throw it from activities when the input is wrong rather than the weather, and wrap CallActivityAsync so that exception type short-circuits the retry loop immediately. One catch: the exception gets serialized across the worker boundary, so you must match on FailureDetails.ErrorType as well as the exception type.Retry Policies Don’t Read Exceptions
My orchestrator calls every activity with the same retry options: 10 attempts, exponential backoff from 5 seconds up to 5 minutes. That’s the right shape for what the policy is there for - SharePoint throttling, Graph hiccups, the occasional 503. Transient stuff.
But RetryPolicy is blunt. Timeout? Retry. HTTP 429? Retry. “The site type ‘TeamSiet’ does not exist”? Retry, obviously, maybe the spelling will heal.
The fix has two halves: activities that know when they’ve hit a wall, and an orchestrator that listens.
Half One: Activities Declare Permanence
/// <summary>
/// Thrown by activities to signal a non-transient failure that should NOT be retried.
/// Examples: invalid configuration, illegal site name, missing required fields.
/// The orchestrator inspects this to short-circuit the retry policy.
/// </summary>
public sealed class PermanentProvisioningException : Exception
Activities throw it exactly where a retry can’t help: an unknown site type in the schema, a hub site ID that isn’t a valid GUID, a PnP template file that doesn’t exist at the configured path. These are all “fix the config and start over” failures. No amount of waiting turns a missing file into a present one.
Everything else - network faults, throttling, SharePoint having a moment - throws ordinary exceptions and rides the retry policy as designed.
Half Two: The Orchestrator Listens
Every activity call goes through a wrapper instead of calling CallActivityAsync raw:
/// <summary>
/// Calls an activity with retry options, but short-circuits immediately for permanent failures.
/// Durable Functions' built-in RetryPolicy retries ALL exceptions; this wrapper inspects the
/// inner exception and re-throws immediately if it's a PermanentProvisioningException.
/// </summary>
private async Task CallActivitySafeAsync(string activityName, object input)
{
try
{
await context.CallActivityAsync(activityName, input, ActivityRetryOptions);
}
catch (TaskFailedException ex) when (IsPermanentFailure(ex))
{
throw new PermanentProvisioningException(
$"Activity '{activityName}' failed with a non-retryable error: {ex.InnerException?.Message ?? ex.Message}",
ex);
}
}
And the detection, which is where the real lesson lives:
private static bool IsPermanentFailure(TaskFailedException ex)
{
// Check if the activity explicitly signaled a permanent failure
if (ex.InnerException is PermanentProvisioningException)
return true;
// Also check the failure details for the exception type name, since
// Durable Functions may serialize/deserialize across process boundaries.
var details = ex.FailureDetails;
if (details?.ErrorType?.Contains(nameof(PermanentProvisioningException), StringComparison.Ordinal) == true)
return true;
return false;
}
What’s happening here?
- Activity failures surface in the orchestrator as
TaskFailedException, with the original error wrapped inside. - The first check is the obvious one: is the inner exception our permanent type?
- The second check is the one you’d never write until it bites you. In the isolated worker model, the activity’s exception is serialized when it crosses from the worker process back to the host and into orchestration history. What you get back is not necessarily your exception instance - sometimes all that survives is
FailureDetailswith the type name as a string. So the check falls back to matchingErrorTypetextually. If you seeFailed to deserialize exception from TaskActivityin the logs, that’s the same boundary telling you out loud what it usually does quietly. In the in-process model the shape is the same with different names: the wrapper isFunctionFailedException, and itsInnerExceptionarrives as a plainSystem.Exceptionrather than your type, which is whycatch (MyException)never fires there either. - When either check hits, the wrapper rethrows as a fresh
PermanentProvisioningExceptionand the orchestration fails right now, with a message naming the activity, instead of 25 minutes from now.
If you only take one thing from this post: an exception that crosses the Durable Functions process boundary is not guaranteed to arrive as its original type. Any design that relies on catch (MySpecificException) around CallActivityAsync needs the string-based fallback, or it will silently degrade into retry-everything.
Wait, doesn’t RetryPolicy have a handle-callback for this? The in-process model had RetryOptions.Handle; in the isolated model the equivalent is a retry handler reading RetryContext.LastFailure, where IsCausedBy<T>() does the type test for you. That shape has been shakier across versions, and even where the callback exists you’re inspecting the same serialized FailureDetails anyway. I went with the explicit wrapper: it’s boring, it’s visible in the call path, and it works the same on every version I’ve touched.
Worth naming the in-process spelling too, since most samples you’ll find use it: CallActivityWithRetryAsync is the same call as CallActivityAsync(name, input, retryOptions) here. If your retry policy appears to do nothing at all, this section is usually why - the policy is running fine, your catch just never matches, so every attempt burns.
The Inverse Trick: Throwing to Request a Retry
Once permanent and transient failures are distinct, activities can use exceptions the other way around: throw a deliberately transient exception to ask for a rerun.
My property bag activity writes values to a fresh site, then reads them back, because SharePoint occasionally accepts the write and forgets it. When verification fails:
throw new InvalidOperationException(
$"Property bag verification failed on {siteUrl}. " +
"Values did not persist to the server; retrying the activity.");
Plain InvalidOperationException, on purpose. It doesn’t match the permanent check, so the retry policy picks it up and reruns the activity, which rewrites and reverifies. The exception type has become a control channel: permanent means “stop, a human needs to fix input”, everything else means “run me again”.
That’s the whole taxonomy. Two kinds of failure, one bit of information, and the activity is the only place with enough context to set that bit. The orchestrator can’t know whether a 500 from SharePoint is a fluke or a doomed request - but the activity that built the request usually can.
Gotchas
- Fail fast still means fail loudly. When the wrapper short-circuits, my orchestrator writes the error message to the provisioning status list before rethrowing, so the person who typo’d the schema sees “unknown site type” next to the customer row instead of finding a generic failed instance in the portal a day later.
- Be stingy with permanence. Marking a failure permanent when it’s actually transient is worse than the reverse: the reverse wastes 25 minutes, but a wrong permanent verdict kills runs that would have succeeded. When in doubt, let it retry.
ErrorTypematching should use the exception’s name, vianameof. Hardcoding the string invites drift when the class gets renamed;nameof(PermanentProvisioningException)keeps the match and the type in lockstep.maxNumberOfAttemptscounts the first call. Ten attempts means the original plus nine retries, not ten retries. Set it to 1 and you have disabled retries entirely - which is a perfectly good way to convince yourself the final retry “never executes”.- Don’t forget the sub-orchestration case. If you split into sub-orchestrations later, the same wrapping applies to
CallSubOrchestratorAsync- permanent failures need to propagate up through every layer that has a retry policy, or one layer’s policy will happily retry another layer’s typo.
Wrapping Up
A retry policy without a permanent-failure escape hatch turns config errors into slow-motion failures. Give activities an exception type that means “never retry this”, short-circuit it in the orchestrator, and match on the serialized type name, not just the .NET type, because the process boundary eats exception identity. Retries are for the weather, not for typos.
This one assumes you already have a retry policy worth escaping. Durable Functions: A Function That Sleeps for a Week is the starting point for the rest of this series.
References
