// Outer safety net: retry any activity that throws a transient error (e.g. throttling
// that slips past HTTP-level retry). Exponential backoff: 5s, 10s, 20s, ... up to 5min,
// 10 attempts total. All activities are idempotent, so retrying the full activity is safe.
private static readonly TaskOptions ActivityRetryOptions = new(
    new RetryPolicy(
        maxNumberOfAttempts: 10,
        firstRetryInterval: TimeSpan.FromSeconds(5),
        backoffCoefficient: 2.0,
        maxRetryInterval: TimeSpan.FromMinutes(5)));

Read that comment again: “All activities are idempotent, so retrying the full activity is safe.”

That sentence is either the best thing about the whole engine or a lie that duplicates customer sites at 2am. There’s no middle ground. The retry policy reruns the entire activity, not the line that failed. If your activity created a site and then died resolving its ID, the retry creates the site again - unless the activity was written to survive being run twice.

So this post is a catalog of how I actually make that sentence true. Five patterns, all from real provisioning code.

Pattern 1: Check Before Create

The site creation activity is the highest-stakes one. If a previous attempt created the site but failed a moment later (say, while resolving the Graph site ID), the retry must not create it again. So it probes first:

// Idempotency: if the site already exists (e.g. previous attempt created it but
// failed during Graph ID resolution), skip creation and go straight to Graph lookup.
if (await SiteExistsAsync(newSiteUrl))
{
    logger.LogInformation("Site already exists at {SiteUrl}, skipping creation.", newSiteUrl);
}
else
{
    await CreateSiteAsync(parsedType, site, siteAlias, newSiteUrl);
}

Note what makes the probe possible in the first place: the site URL is deterministic, built from a prefix and alias in the schema. No timestamps, no random suffixes. If the URL contained Guid.NewGuid(), attempt two would probe a different URL, find nothing, and create a second site. Deterministic naming is the quiet prerequisite for check-before-create.

Pattern 2: “Already Exists” Is Success

Sometimes you can’t probe cheaply, so you attempt the operation and translate the failure. Activating a site feature that’s already active comes back from SharePoint as an error - but it means the activity already did its job on a previous run:

catch (InvalidOperationException ex) when (
    // "Feature already activated" comes back as HTTP 500 with odata.error.code
    // "System.Data.DuplicateNameException" - locale-independent, unlike the human message.
    ex.Message.Contains("System.Data.DuplicateNameException", StringComparison.OrdinalIgnoreCase))
{
    logger.LogInformation("{Feature} feature is already active, skipping.", displayName);
}

The comment is a scar. My first version matched on the English error text, which works great until the code runs against a tenant in another language. Match on error codes, not error messages. The OData error code survives localization; “Feature … is already activated” does not.

Graph makes this pattern cleaner because it gives you a real status code. Copying folder structures into the new site:

catch (ODataError ex) when (ex.ResponseStatusCode == 409)
{
    // Item already exists in target - idempotent on retry
    logger.LogInformation("Item '{ItemName}' already exists in target, skipping copy.", itemName);
    return;
}

409 Conflict on a create is not a failure. It’s a receipt from your previous attempt.

Pattern 3: Upsert by Nature

The best idempotency is the kind you get for free by choosing the right API. The registration activity writes the new site’s URL and ID back to an inventory list, and it does so with SharePoint’s ValidateUpdateListItem against a known item ID. Setting the same fields on the same item to the same values twice is indistinguishable from doing it once. There’s nothing to guard because the operation has no failure mode for repetition.

When you’re designing an activity and you get to pick between “add a row” and “set fields on item X”, pick the second. Every Add needs a guard; a keyed Set guards itself.

Pattern 4: Verify After Write

The dark twin of idempotency: some SharePoint writes report success and then don’t stick. Property bag values on a freshly created site are notorious for this. The activity’s answer is to re-read everything it just wrote and throw if reality disagrees:

if (mismatches.Count > 0)
{
    throw new InvalidOperationException(
        $"Property bag verification failed on {siteUrl}: {string.Join("; ", mismatches)}. " +
        "Values did not persist to the server; retrying the activity.");
}

What’s happening here?

1. After writing the property bag values, the activity loads Web.AllProperties fresh from the server.
2. Each expected key/value is compared against what actually persisted.
3. Any mismatch throws a plain transient exception, which hands the problem to the orchestrator’s retry policy.
4. The rerun is safe precisely because setting a property bag key is an upsert - pattern 3 again. Verification and idempotency are two halves of the same loop: writes are repeatable, so failed verification can simply demand a repeat.

This inverts the usual relationship with retries. Instead of retries being something inflicted on the activity, the activity uses a throw to request one.

Pattern 5: Idempotent Deletes Too

Removal flows have the same problem in the other direction. Removing a user from a site’s members group when they’re already gone throws a ServerException, and treating that as failure would wedge every cleanup retry:

catch (ServerException ex)
{
    logger.LogWarning(ex, "Could not remove {LoginName} from {SiteUrl} ({ErrorType}); treating as already removed.",
        loginName, siteUrl, ex.ServerErrorTypeName);
    return;
}

Desired state: user is not in the group. User is not in the group. Done. Idempotency is about converging on a state, not performing an action.

Gotchas

• One non-idempotent activity poisons the whole pipeline. I had a security activity that created SharePoint groups without checking for existing ones. Every retry of that step risked duplicate groups, which meant retries weren’t safe, which meant the retry policy comment was a lie for the whole orchestration. It’s disabled until it earns its way back in. Idempotency is all-or-nothing per pipeline.
• The failure window is between success and checkpoint. The orchestrator records completed steps, but a crash can land after the activity finished and before the record was written. The completed-steps list narrows how often reruns happen; idempotency is what makes the remaining reruns harmless. You need both.
• catch the narrowest thing you can. The 409 handler above catches ODataError with status 409, nothing else. A broad catch { return; } also “makes retries pass”, by swallowing real failures. Idempotency guards should be precise enough that a genuinely broken call still throws.
• Test by running it twice, literally. My smoke test for a new activity is: run the provisioning, then immediately queue the exact same provisioning again. Zero errors and zero duplicates or it doesn’t ship.

Wrapping Up

A Durable Functions retry policy is a contract: it promises to rerun your activities, and your activities promise not to care. Check before create, read “already exists” as success, prefer upserts, verify writes that lie. Write every activity as if it will run twice, because on a long enough timeline, it will.

If retry policies and replay are new to you, Durable Functions: A Function That Sleeps for a Week covers what an orchestration actually is, and why “it will run twice” is a feature rather than a bug.

The first time step seven failed, I got to watch my engine try to create a site that already existed.

That run taught me the rule this post is about: in a long provisioning flow, you don’t roll back when something fails. You checkpoint, and you resume forward.

If Durable Functions are new to you, start with my intro to what they are and what they can do - this post builds on it.

Why Rollback Is the Wrong Instinct

When provisioning fails at step seven, you have a half-built site. The textbook answer is a compensation saga: undo steps six through one in reverse order. Delete the folders, detach the template, deactivate the features, delete the site, then recreate everything from scratch on the next attempt.

I started sketching that and stopped halfway through the list, because every line was either dangerous or absurd. Deleting a site collection to work around a failed navigation tweak is using a crane to hang a picture. And some steps don’t even have an undo - you can’t meaningfully “unapply” a provisioning template that merged fields into existing lists.

Here’s the thing rollback ignores: the six completed steps aren’t damage. They’re progress. The site is fine; what’s missing is the steps that haven’t run yet. So the only recovery that makes sense is forward: figure out where the run stopped, and continue from there.

That reframes the problem completely. I don’t need compensation logic. I need to know, reliably, which steps finished.

Carry the Progress in the State

Durable Functions makes this natural, because an orchestrator already passes state to each activity and gets state back. The trick is to make “what’s done” part of that state. My orchestration state looks something like this:

public sealed record SiteSetupState(
    string CustomerId,
    string SiteUrl,
    string[] CompletedSteps
);

And the orchestrator walks its steps like this:

var state = context.GetInput<SiteSetupState>() ?? await CreateSite(context, customerId);

string[] pipeline =
[
    nameof(ActivateFeatures),
    nameof(ApplyTemplate),
    nameof(SeedFolders),
    nameof(AddMemberGroups),
    nameof(RegisterSite),
];

foreach (var step in pipeline)
{
    if (state.CompletedSteps.Contains(step))
    {
        logger.LogInformation("{Step} already completed, skipping.", step);
        continue;
    }

    await context.CallActivityAsync(step, state, retryOptions);

    state = state with { CompletedSteps = [.. state.CompletedSteps, step] };
    context.SetCustomStatus(state);
}

What’s happening here?

1. The pipeline is just an ordered list of activity names. Nothing clever, and that’s the point - the interesting machinery is around the calls, not in them.
2. Before each step, the orchestrator checks whether this step already ran. On a fresh run the array is empty and nothing is skipped. On a resumed run, this check is what fast-forwards past the finished work.
3. After each successful step, the state is copied with the step name appended. An immutable with copy, so nothing mutates in place, which keeps replays honest.
4. SetCustomStatus publishes the updated state on the orchestration instance. This is the checkpoint. It costs one line.

That last line is doing more work than it looks like. Custom status is readable from outside the orchestration - through the management API, without touching the orchestration history. So the same call gives you two things: anyone polling the instance sees live progress (“three of six steps done”), and if the run fails, the last published state is sitting right there, telling you exactly where it stopped.

Resume Is Just Input

Because the checkpoint is a plain serializable record, resuming a failed run doesn’t need any special framework support. You read the failed instance’s custom status, and you start a new orchestration with that state as input:

var failed = await client.GetInstanceAsync(failedInstanceId, getInputsAndOutputs: true);
var checkpoint = failed.ReadCustomStatusAs<SiteSetupState>();

await client.ScheduleNewOrchestrationInstanceAsync(
    nameof(SiteSetupOrchestrator),
    checkpoint);

The new run enters the same loop, finds five steps in CompletedSteps, skips them in about a millisecond, and picks up at step six. No site deletion, no re-creation, no duplicate template apply. The half-built site becomes a five-sixths-built site, then a finished one.

I like how little there is to this. The “resume feature” is the skip-check in the loop plus the fact that the input type and the checkpoint type are the same type. That’s it.

The Gap That Idempotency Covers

One honest caveat. The checkpoint is written after the activity succeeds, so there’s a window: the activity finishes, the process dies before the checkpoint lands. On resume, that step’s name isn’t in CompletedSteps, and it runs again.

You can’t close that window - it’s inherent to doing the work and recording the work as two operations. What you do instead is make every activity safe to run twice: check before create, treat “already exists” as success, write with upserts. That’s a full topic on its own, but the division of labor is worth stating plainly: the checkpoint decides how often steps rerun, idempotency decides whether reruns hurt. You need both. The checkpoint alone has a crash window; idempotency alone means re-executing ten minutes of finished work on every hiccup.

And never checkpoint before the call to close the window from the other side - then a crashed step gets skipped on resume, which is far worse. A step that runs twice is a wasted minute; a step that runs zero times is a broken site that says “Completed”. Ask me how I know.

Gotchas

• Use the replay-safe logger. Orchestrator code replays from history every time the instance wakes up. context.CreateReplaySafeLogger(...) keeps your logs from repeating every completed step on each replay. A regular ILogger in an orchestrator will gaslight you.
• No clocks, no GUIDs in the orchestrator. DateTime.UtcNow, Guid.NewGuid(), and Random produce different values on replay and corrupt the history. Anything nondeterministic belongs inside an activity, including generated names and timestamps you want in the state.
• Custom status has a size limit (16 KB of JSON). A record with a customer ID, a URL, and an array of step names fits hundreds of times over, but don’t stuff a whole template or file manifest in there. Checkpoint the position, not the payload.
• Step names are a contract. The moment CompletedSteps is persisted anywhere - a failed instance you might resume next week - renaming an activity breaks the match and the step silently reruns (fine, if idempotent) or the resume misbehaves (not fine). Rename with the same care you’d give a database column.
• Report progress somewhere humans look. Custom status is great for machines; my engine also writes the current step name to a status column in the site inventory list, inside a try/catch that logs and swallows. A cosmetic status write must never kill a provisioning run.

Wrapping Up

Long provisioning flows fail in the middle, so design for resuming instead of undoing: carry a list of completed steps in the orchestration state, publish it as custom status after every step, skip completed steps on rerun, and feed the saved state back in to resume. Rollback is for databases. Provisioning goes forward.

My first instinct was a plain Azure Function. One trigger, one method, ten steps in a row.

That instinct survived about a day. A plain function has a timeout measured in minutes. It holds its state in local variables, so when the process restarts mid-run (deployments, scale-in, Azure having a Tuesday), everything it knew is gone. Step six fails and your options are “run all ten steps again” or “reconstruct what happened from logs like a crime scene”.

And it’s not like the step-sequencing part was new to me. I’d already built ProvisioningPipeline, an open-source .NET library for composing provisioning workflows from middleware-style steps, with hooks and parallel execution. It solves the “run these steps in order, cleanly” problem well. What no in-process pipeline can solve is the other half: a pipeline in memory still dies with the process, and all its progress dies with it.

What I actually needed was a function that can run for a long time, remember how far it got, and pick up where it left off. That’s not a pattern you bolt onto a normal function. It’s a different tool, and Azure ships it: Durable Functions.

Three Functions, One Flow

A durable workflow is made of three kinds of functions, and the separation is the whole idea.

The client is a normal function with a normal trigger - HTTP, queue, timer, whatever starts your process. Its only durable job is to schedule an orchestration and get out of the way:

[Function(nameof(StartProvisioning))]
public async Task Run(
    [ServiceBusTrigger("provisioning")] ProvisioningMessage message,
    [DurableClient] DurableTaskClient client)
{
    await client.ScheduleNewOrchestrationInstanceAsync(
        nameof(SiteSetupOrchestrator),
        message.ToSetupRequest());
}

The orchestrator is the recipe. It calls the steps in order and holds the workflow logic, but does no real work itself:

[Function(nameof(SiteSetupOrchestrator))]
public async Task RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var request = context.GetInput<SiteSetupRequest>();

    var site = await context.CallActivityAsync<SiteInfo>(nameof(CreateSite), request);
    await context.CallActivityAsync(nameof(ActivateFeatures), site);
    await context.CallActivityAsync(nameof(ApplyTemplate), site);
    await context.CallActivityAsync(nameof(SeedFolders), site);
    await context.CallActivityAsync(nameof(AddMemberGroups), site);
    await context.CallActivityAsync(nameof(RegisterSite), site);
}

The activities are where the actual SharePoint and Graph calls live. Each one is a small, self-contained function: CreateSite creates a site, ApplyTemplate applies a template, and so on.

What’s happening here?

1. The client schedules an orchestration instance and returns immediately. Nothing waits around; the run now has a life of its own, with an instance ID you can use to check on it later.
2. The orchestrator reads like a script: six awaited steps, top to bottom. This is the part that would be a state machine diagram in Logic Apps or a pile of queues in a hand-rolled solution. Here it’s just code, with loops and try/catch available when you need them.
3. Every CallActivityAsync is a checkpoint boundary. The runtime records that the step completed, and what it returned, before moving on.
4. Activities take input, do I/O, return output. They don’t know they’re part of a workflow, which keeps them easy to write and easy to test.

The “Durable” Part Is a History Table

Here’s what separates this from a normal async method: the orchestrator’s progress doesn’t live in memory. Every scheduled activity and every result gets written to a history table in storage as an event.

When the process dies mid-run - and over a multi-minute workflow across enough customers, it will - the runtime picks the orchestration up on another instance and replays it. The orchestrator code runs again from the top, but every activity that already completed doesn’t re-execute; its recorded result comes straight back from history. The replay fast-forwards through the finished steps in milliseconds and resumes real work at the first step without a recorded result.

It’s a save file in a video game. The console losing power doesn’t send you back to level one; you reload and continue from the last save point. The orchestrator saves after every activity, and it never plays a level twice.

That same mechanism is why an orchestration can wait for days without costing you anything. Awaiting a durable timer or an external event just means “write the wait to history and unload from memory”. No process is alive, no compute is billed, and when the timer fires next week, replay reconstructs the state and continues. A function that sleeps for a week, literally.

What Else Is in the Box

Sequential steps are the basics. A few more capabilities, through the provisioning lens:

Per-step retries. Every activity call can carry a retry policy, and this is the built-in retry logic worth knowing about. Mine looks like this:

private static readonly TaskOptions ActivityRetryOptions = new(
    new RetryPolicy(
        maxNumberOfAttempts: 10,
        firstRetryInterval: TimeSpan.FromSeconds(5),
        backoffCoefficient: 2.0,
        maxRetryInterval: TimeSpan.FromMinutes(5)));

await context.CallActivityAsync(nameof(ApplyTemplate), site, ActivityRetryOptions);

If ApplyTemplate throws because SharePoint returned a 429, the runtime waits 5 seconds and reruns the activity. Then 10, then 20, backing off exponentially up to 5 minutes, for up to 10 attempts. The orchestration just sits there durably waiting between attempts. For workflows built on APIs that throttle, this one feature pays for the whole ticket.

Fan-out/fan-in. Steps that don’t depend on each other can run as parallel activities and be awaited together with Task.WhenAll. Seeding twelve folders one by one is a for-loop; seeding them as twelve parallel activities is the same loop without the awaits until the end.

Durable timers. context.CreateTimer(...) gives you delays and polling loops that survive restarts. Waiting 30 seconds for SharePoint to finish creating a site is a timer, not a Task.Delay and a prayer.

External events. An orchestration can pause until something outside signals it - WaitForExternalEvent is how you build “wait for an admin to approve, then continue”, even if the admin takes four days.

Status from the outside. Every instance can be queried by ID: is it running, what step is it on, did it fail and why. My provisioning engine exposes this so the business system can show “site is being created” instead of a shrug.

What It Doesn’t Do for You

An honest intro should include the bill.

Retries are per step, not per run. The retry policy reruns a failing activity. If an activity exhausts all ten attempts, the orchestration fails, and a failed run stays failed - nothing in the box restarts it from where it stopped. Resuming a dead run is a pattern you build yourself on top of these primitives.

Retried activities rerun whole. When attempt two starts, it starts from the activity’s first line, not from where attempt one died. An activity that created a site and then failed will try to create the site again. Every activity has to be written to survive running twice, and that’s on you.

Orchestrator code lives under house rules. Because the orchestrator replays, it must be deterministic: same history in, same decisions out. No DateTime.UtcNow, no Guid.NewGuid(), no HTTP calls, no reading config that might change between replays. Anything nondeterministic belongs in an activity. The rules are simple but unforgiving, and the failure mode is a corrupted run, not a compile error.

None of these are dealbreakers. They’re the shape of the tool: the runtime guarantees your progress is never lost, and in exchange you write steps that tolerate being repeated.

Gotchas

• Use the replay-safe logger in orchestrators. context.CreateReplaySafeLogger(...) suppresses log lines during replay. A regular ILogger logs every completed step again on every replay, and the duplicated log stream will have you debugging problems you don’t have.
• Keep activities small and single-purpose. One activity per step means one checkpoint per step. A mega-activity that does five things gives you one checkpoint for five steps, and a failure at thing four reruns things one through three.
• The retry policy can’t tell a 429 from a typo. It retries every exception the same way, including the ones that will never succeed, like a broken config value. Distinguishing “try again” from “stop, a human needs to fix this” takes extra work.
• Don’t sneak I/O into the orchestrator. It compiles, it works on the happy path, and it breaks determinism the first time a replay gets a different answer. If it touches the network or the clock, it’s an activity.

Wrapping Up

A Durable Function is a workflow that remembers where it was: an orchestrator sequences the steps, activities do the work, and the runtime checkpoints every step so crashes, restarts, and week-long waits don’t lose progress. If your process has numbered steps, talks to flaky APIs, and takes longer than a request, it’s a fit - provisioning was mine.