// Outer safety net: retry any activity that throws a transient error (e.g. throttling
// that slips past HTTP-level retry). Exponential backoff: 5s, 10s, 20s, ... up to 5min,
// 10 attempts total. All activities are idempotent, so retrying the full activity is safe.
private static readonly TaskOptions ActivityRetryOptions = new(
    new RetryPolicy(
        maxNumberOfAttempts: 10,
        firstRetryInterval: TimeSpan.FromSeconds(5),
        backoffCoefficient: 2.0,
        maxRetryInterval: TimeSpan.FromMinutes(5)));

Read that comment again: “All activities are idempotent, so retrying the full activity is safe.”

That sentence is either the best thing about the whole engine or a lie that duplicates customer sites at 2am. There’s no middle ground. The retry policy reruns the entire activity, not the line that failed. If your activity created a site and then died resolving its ID, the retry creates the site again - unless the activity was written to survive being run twice.

So this post is a catalog of how I actually make that sentence true. Five patterns, all from real provisioning code.

Pattern 1: Check Before Create

The site creation activity is the highest-stakes one. If a previous attempt created the site but failed a moment later (say, while resolving the Graph site ID), the retry must not create it again. So it probes first:

// Idempotency: if the site already exists (e.g. previous attempt created it but
// failed during Graph ID resolution), skip creation and go straight to Graph lookup.
if (await SiteExistsAsync(newSiteUrl))
{
    logger.LogInformation("Site already exists at {SiteUrl}, skipping creation.", newSiteUrl);
}
else
{
    await CreateSiteAsync(parsedType, site, siteAlias, newSiteUrl);
}

Note what makes the probe possible in the first place: the site URL is deterministic, built from a prefix and alias in the schema. No timestamps, no random suffixes. If the URL contained Guid.NewGuid(), attempt two would probe a different URL, find nothing, and create a second site. Deterministic naming is the quiet prerequisite for check-before-create.

Pattern 2: “Already Exists” Is Success

Sometimes you can’t probe cheaply, so you attempt the operation and translate the failure. Activating a site feature that’s already active comes back from SharePoint as an error - but it means the activity already did its job on a previous run:

catch (InvalidOperationException ex) when (
    // "Feature already activated" comes back as HTTP 500 with odata.error.code
    // "System.Data.DuplicateNameException" - locale-independent, unlike the human message.
    ex.Message.Contains("System.Data.DuplicateNameException", StringComparison.OrdinalIgnoreCase))
{
    logger.LogInformation("{Feature} feature is already active, skipping.", displayName);
}

The comment is a scar. My first version matched on the English error text, which works great until the code runs against a tenant in another language. Match on error codes, not error messages. The OData error code survives localization; “Feature … is already activated” does not.

Graph makes this pattern cleaner because it gives you a real status code. Copying folder structures into the new site:

catch (ODataError ex) when (ex.ResponseStatusCode == 409)
{
    // Item already exists in target - idempotent on retry
    logger.LogInformation("Item '{ItemName}' already exists in target, skipping copy.", itemName);
    return;
}

409 Conflict on a create is not a failure. It’s a receipt from your previous attempt.

Pattern 3: Upsert by Nature

The best idempotency is the kind you get for free by choosing the right API. The registration activity writes the new site’s URL and ID back to an inventory list, and it does so with SharePoint’s ValidateUpdateListItem against a known item ID. Setting the same fields on the same item to the same values twice is indistinguishable from doing it once. There’s nothing to guard because the operation has no failure mode for repetition.

When you’re designing an activity and you get to pick between “add a row” and “set fields on item X”, pick the second. Every Add needs a guard; a keyed Set guards itself.

Pattern 4: Verify After Write

The dark twin of idempotency: some SharePoint writes report success and then don’t stick. Property bag values on a freshly created site are notorious for this. The activity’s answer is to re-read everything it just wrote and throw if reality disagrees:

if (mismatches.Count > 0)
{
    throw new InvalidOperationException(
        $"Property bag verification failed on {siteUrl}: {string.Join("; ", mismatches)}. " +
        "Values did not persist to the server; retrying the activity.");
}

What’s happening here?

1. After writing the property bag values, the activity loads Web.AllProperties fresh from the server.
2. Each expected key/value is compared against what actually persisted.
3. Any mismatch throws a plain transient exception, which hands the problem to the orchestrator’s retry policy.
4. The rerun is safe precisely because setting a property bag key is an upsert - pattern 3 again. Verification and idempotency are two halves of the same loop: writes are repeatable, so failed verification can simply demand a repeat.

This inverts the usual relationship with retries. Instead of retries being something inflicted on the activity, the activity uses a throw to request one.

Pattern 5: Idempotent Deletes Too

Removal flows have the same problem in the other direction. Removing a user from a site’s members group when they’re already gone throws a ServerException, and treating that as failure would wedge every cleanup retry:

catch (ServerException ex)
{
    logger.LogWarning(ex, "Could not remove {LoginName} from {SiteUrl} ({ErrorType}); treating as already removed.",
        loginName, siteUrl, ex.ServerErrorTypeName);
    return;
}

Desired state: user is not in the group. User is not in the group. Done. Idempotency is about converging on a state, not performing an action.

Gotchas

• One non-idempotent activity poisons the whole pipeline. I had a security activity that created SharePoint groups without checking for existing ones. Every retry of that step risked duplicate groups, which meant retries weren’t safe, which meant the retry policy comment was a lie for the whole orchestration. It’s disabled until it earns its way back in. Idempotency is all-or-nothing per pipeline.
• The failure window is between success and checkpoint. The orchestrator records completed steps, but a crash can land after the activity finished and before the record was written. The completed-steps list narrows how often reruns happen; idempotency is what makes the remaining reruns harmless. You need both.
• catch the narrowest thing you can. The 409 handler above catches ODataError with status 409, nothing else. A broad catch { return; } also “makes retries pass”, by swallowing real failures. Idempotency guards should be precise enough that a genuinely broken call still throws.
• Test by running it twice, literally. My smoke test for a new activity is: run the provisioning, then immediately queue the exact same provisioning again. Zero errors and zero duplicates or it doesn’t ship.

Wrapping Up

A Durable Functions retry policy is a contract: it promises to rerun your activities, and your activities promise not to care. Check before create, read “already exists” as success, prefer upserts, verify writes that lie. Write every activity as if it will run twice, because on a long enough timeline, it will.

If retry policies and replay are new to you, Durable Functions: A Function That Sleeps for a Week covers what an orchestration actually is, and why “it will run twice” is a feature rather than a bug.

First instinct: call the API straight from the web part. It works in twenty minutes. Then you press F12, open the network tab, click any request… and there it is. The company’s master API key, in plain text, readable by every single user who ever loads that intranet page.

And it’s not “they can see the data the web part shows anyway” - the web part shows a filtered slice. The key unlocks the whole API: every customer, every record, every write operation the key is licensed for. Anyone who copies it out of the inspector can call the API from Postman on their couch.

Why the Browser Can’t Keep a Secret

It’s worth being blunt about this, because the temptation to “just obfuscate it a bit” is real:

• Everything the browser sends is in the network tab.
• Everything the bundle contains is in the sources tab (source maps or not, strings are strings).
• Everything the app holds in memory is one breakpoint away.

There is no hiding place in the client. And “it’s only our internal SharePoint” doesn’t help - internal users are exactly the people who shouldn’t be walking around with the master key to a system they’re only supposed to see a corner of.

The Proxy Function

The fix is a small reverse proxy. One catch-all Azure Function fronts the entire third-party API:

private static readonly HashSet<string> _hopByHop = new(StringComparer.OrdinalIgnoreCase)
{
    "Connection", "Keep-Alive", "Proxy-Authenticate", "Proxy-Authorization",
    "TE", "Trailer", "Transfer-Encoding", "Upgrade"
};

[Function("ApiProxy")]
public async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post", "put", "delete",
                 Route = "api/{*path}")] HttpRequest request,
    string? path,
    CancellationToken ct)
{
    var targetUri = _settings.BaseUrl.TrimEnd('/') + "/" + (path ?? "")
                  + (request.QueryString.Value ?? "");

    using var upstream = new HttpRequestMessage(new HttpMethod(request.Method), targetUri);

    if (request.Method is "POST" or "PUT" || request.ContentLength is > 0)
    {
        upstream.Content = new StreamContent(request.Body);
        if (request.ContentType is not null)
            upstream.Content.Headers.TryAddWithoutValidation("Content-Type", request.ContentType);
    }

    foreach (var (key, values) in request.Headers)
    {
        // Never forward the client's Host, any API key they try to sneak in,
        // or hop-by-hop headers that belong to *this* connection only.
        if (key.Equals("Host", StringComparison.OrdinalIgnoreCase) ||
            key.Equals("X-Api-Key", StringComparison.OrdinalIgnoreCase) ||
            _hopByHop.Contains(key))
            continue;

        if (!upstream.Headers.TryAddWithoutValidation(key, (IEnumerable<string?>)values!))
            upstream.Content?.Headers.TryAddWithoutValidation(key, (IEnumerable<string?>)values!);
    }

    // The swap. The secret exists only here, on the outbound leg.
    upstream.Headers.Add("X-Api-Key", _settings.ApiKey);

    var client = _httpClientFactory.CreateClient("upstream-api");
    using var response = await client.SendAsync(upstream, HttpCompletionOption.ResponseHeadersRead, ct);

    _logger.LogInformation("Proxy: {Method} {Path} → {StatusCode}", request.Method, path, (int)response.StatusCode);

    var httpResponse = request.HttpContext.Response;
    httpResponse.StatusCode = (int)response.StatusCode;

    foreach (var (key, values) in response.Headers.Concat(response.Content.Headers))
    {
        if (_hopByHop.Contains(key)) continue;
        httpResponse.Headers.Append(key, values.ToArray());
    }

    await response.Content.CopyToAsync(httpResponse.Body, ct);
    return new EmptyResult();
}

What’s happening here?

1. Route = "api/{*path}" is a catch-all. GET /api/customers/123/tasks becomes GET https://the-api.example/customers/123/tasks, query string included. One function fronts the whole API surface - when the vendor adds endpoints, the proxy needs zero changes.
2. The header loop deliberately strips any inbound X-Api-Key. A caller can’t inject their own key or override yours - whatever they send in that header dies at the proxy.
3. The hop-by-hop set (Connection, Transfer-Encoding, …) is the detail naive proxies get wrong. Those headers describe one connection, not the request; forwarding them causes wonderfully confusing breakage.
4. The real key is added to the outbound request only. Response headers get copied back to the browser, but request headers are never echoed - so the key physically cannot appear in the inspector. It’s not hidden; it’s absent.
5. ResponseHeadersRead + CopyToAsync streams the upstream response straight through without buffering it in the function’s memory, and upstream status codes pass through untouched so the web part can react to a 404 or a 429 honestly.

Where the Key Lives

Server-side config, strongly typed, validated at boot:

services.AddOptions<ApiSettings>()
    .Bind(configuration.GetSection("UpstreamApi"))
    .Validate(s => IsConfigured(s.BaseUrl), "UpstreamApi:BaseUrl must be configured.")
    .Validate(s => IsConfigured(s.ApiKey), "UpstreamApi:ApiKey must be configured.")
    .ValidateOnStart();

private static bool IsConfigured(string value) =>
    !string.IsNullOrWhiteSpace(value) && !value.StartsWith('<');

The value itself comes from Function App settings - ideally as a Key Vault reference - never from a committed file. ValidateOnStart() plus the <placeholder> guard means a misconfigured secret fails the deployment at boot, not at the first user’s click three days later.

Hiding ≠ Authorizing

Here’s the part that’s easy to skip and shouldn’t be: as shown so far, the proxy hides the key from the browser, but anyone who discovers the function URL can call it - and burn your API quota with your key. We’ve moved the secret, not secured the door.

Lock the function to Entra ID. Create an app registration for the function API, expose a scope (the convention is access_as_user), and have the function validate incoming JWTs - signature, issuer, and audience, not just parsing the claims out. Parsing is reading the name tag; validating is checking the ID.

On the SPFx side, this is pleasantly little work. Declare the permission in package-solution.json:

"webApiPermissionRequests": [
  {
    "resource": "my-function-api",
    "scope": "access_as_user"
  }
]

And call the function with AadHttpClient - SPFx acquires and attaches the user’s token for you:

const client = await this.context.aadHttpClientFactory
  .getClient("api://<function-app-registration-client-id>");

const response = await client.get(
  `https://my-function.azurewebsites.net/api/customers/${customerCode}/tasks`,
  AadHttpClient.configurations.v1
);

Now open the inspector again. There is a token in the request - but it’s the user’s own token: short-lived, scoped to your function only, and useless against the third-party API. That’s the whole difference between a secret and an identity. A stolen API key is everyone’s master key forever; a stolen user token is one person’s function access for an hour.

Gotchas

• Watch what you log. Log method, path, and status - never headers. Otherwise the key you carefully kept out of the browser ends up in Application Insights, readable by everyone with portal access.
• You now front someone else’s quota. The vendor’s rate limits hit your key for all users combined. If the web part is chatty, add caching or throttling in the proxy before the vendor does it for you.
• CORS is on you. The browser is calling your function from a SharePoint origin - configure CORS on the Function App for your tenant’s SharePoint domain, not *.
• Consider narrowing the surface. A catch-all proxy forwards everything, including endpoints the web part never needed. If the API has destructive operations, whitelist the methods and paths you actually use.

Wrapping Up

A shared API key belongs on a server. Full stop. The moment it ships to a browser, every user has it, and you can’t take it back - most vendors will happily rotate the key for you, but you’ll be doing that dance on their schedule, not yours.

The proxy costs about fifty lines: catch-all route, header hygiene, one server-side header swap, and Entra ID on the front door. That turns “every intranet user carries the master key” into “every request is a named user, calling a locked endpoint, seeing exactly what the web part shows them”. 🔒

The first time step seven failed, I got to watch my engine try to create a site that already existed.

That run taught me the rule this post is about: in a long provisioning flow, you don’t roll back when something fails. You checkpoint, and you resume forward.

If Durable Functions are new to you, start with my intro to what they are and what they can do - this post builds on it.

Why Rollback Is the Wrong Instinct

When provisioning fails at step seven, you have a half-built site. The textbook answer is a compensation saga: undo steps six through one in reverse order. Delete the folders, detach the template, deactivate the features, delete the site, then recreate everything from scratch on the next attempt.

I started sketching that and stopped halfway through the list, because every line was either dangerous or absurd. Deleting a site collection to work around a failed navigation tweak is using a crane to hang a picture. And some steps don’t even have an undo - you can’t meaningfully “unapply” a provisioning template that merged fields into existing lists.

Here’s the thing rollback ignores: the six completed steps aren’t damage. They’re progress. The site is fine; what’s missing is the steps that haven’t run yet. So the only recovery that makes sense is forward: figure out where the run stopped, and continue from there.

That reframes the problem completely. I don’t need compensation logic. I need to know, reliably, which steps finished.

Carry the Progress in the State

Durable Functions makes this natural, because an orchestrator already passes state to each activity and gets state back. The trick is to make “what’s done” part of that state. My orchestration state looks something like this:

public sealed record SiteSetupState(
    string CustomerId,
    string SiteUrl,
    string[] CompletedSteps
);

And the orchestrator walks its steps like this:

var state = context.GetInput<SiteSetupState>() ?? await CreateSite(context, customerId);

string[] pipeline =
[
    nameof(ActivateFeatures),
    nameof(ApplyTemplate),
    nameof(SeedFolders),
    nameof(AddMemberGroups),
    nameof(RegisterSite),
];

foreach (var step in pipeline)
{
    if (state.CompletedSteps.Contains(step))
    {
        logger.LogInformation("{Step} already completed, skipping.", step);
        continue;
    }

    await context.CallActivityAsync(step, state, retryOptions);

    state = state with { CompletedSteps = [.. state.CompletedSteps, step] };
    context.SetCustomStatus(state);
}

What’s happening here?

1. The pipeline is just an ordered list of activity names. Nothing clever, and that’s the point - the interesting machinery is around the calls, not in them.
2. Before each step, the orchestrator checks whether this step already ran. On a fresh run the array is empty and nothing is skipped. On a resumed run, this check is what fast-forwards past the finished work.
3. After each successful step, the state is copied with the step name appended. An immutable with copy, so nothing mutates in place, which keeps replays honest.
4. SetCustomStatus publishes the updated state on the orchestration instance. This is the checkpoint. It costs one line.

That last line is doing more work than it looks like. Custom status is readable from outside the orchestration - through the management API, without touching the orchestration history. So the same call gives you two things: anyone polling the instance sees live progress (“three of six steps done”), and if the run fails, the last published state is sitting right there, telling you exactly where it stopped.

Resume Is Just Input

Because the checkpoint is a plain serializable record, resuming a failed run doesn’t need any special framework support. You read the failed instance’s custom status, and you start a new orchestration with that state as input:

var failed = await client.GetInstanceAsync(failedInstanceId, getInputsAndOutputs: true);
var checkpoint = failed.ReadCustomStatusAs<SiteSetupState>();

await client.ScheduleNewOrchestrationInstanceAsync(
    nameof(SiteSetupOrchestrator),
    checkpoint);

The new run enters the same loop, finds five steps in CompletedSteps, skips them in about a millisecond, and picks up at step six. No site deletion, no re-creation, no duplicate template apply. The half-built site becomes a five-sixths-built site, then a finished one.

I like how little there is to this. The “resume feature” is the skip-check in the loop plus the fact that the input type and the checkpoint type are the same type. That’s it.

The Gap That Idempotency Covers

One honest caveat. The checkpoint is written after the activity succeeds, so there’s a window: the activity finishes, the process dies before the checkpoint lands. On resume, that step’s name isn’t in CompletedSteps, and it runs again.

You can’t close that window - it’s inherent to doing the work and recording the work as two operations. What you do instead is make every activity safe to run twice: check before create, treat “already exists” as success, write with upserts. That’s a full topic on its own, but the division of labor is worth stating plainly: the checkpoint decides how often steps rerun, idempotency decides whether reruns hurt. You need both. The checkpoint alone has a crash window; idempotency alone means re-executing ten minutes of finished work on every hiccup.

And never checkpoint before the call to close the window from the other side - then a crashed step gets skipped on resume, which is far worse. A step that runs twice is a wasted minute; a step that runs zero times is a broken site that says “Completed”. Ask me how I know.

Gotchas

• Use the replay-safe logger. Orchestrator code replays from history every time the instance wakes up. context.CreateReplaySafeLogger(...) keeps your logs from repeating every completed step on each replay. A regular ILogger in an orchestrator will gaslight you.
• No clocks, no GUIDs in the orchestrator. DateTime.UtcNow, Guid.NewGuid(), and Random produce different values on replay and corrupt the history. Anything nondeterministic belongs inside an activity, including generated names and timestamps you want in the state.
• Custom status has a size limit (16 KB of JSON). A record with a customer ID, a URL, and an array of step names fits hundreds of times over, but don’t stuff a whole template or file manifest in there. Checkpoint the position, not the payload.
• Step names are a contract. The moment CompletedSteps is persisted anywhere - a failed instance you might resume next week - renaming an activity breaks the match and the step silently reruns (fine, if idempotent) or the resume misbehaves (not fine). Rename with the same care you’d give a database column.
• Report progress somewhere humans look. Custom status is great for machines; my engine also writes the current step name to a status column in the site inventory list, inside a try/catch that logs and swallows. A cosmetic status write must never kill a provisioning run.

Wrapping Up

Long provisioning flows fail in the middle, so design for resuming instead of undoing: carry a list of completed steps in the orchestration state, publish it as custom status after every step, skip completed steps on rerun, and feed the saved state back in to resume. Rollback is for databases. Provisioning goes forward.

My first instinct was a plain Azure Function. One trigger, one method, ten steps in a row.

That instinct survived about a day. A plain function has a timeout measured in minutes. It holds its state in local variables, so when the process restarts mid-run (deployments, scale-in, Azure having a Tuesday), everything it knew is gone. Step six fails and your options are “run all ten steps again” or “reconstruct what happened from logs like a crime scene”.

And it’s not like the step-sequencing part was new to me. I’d already built ProvisioningPipeline, an open-source .NET library for composing provisioning workflows from middleware-style steps, with hooks and parallel execution. It solves the “run these steps in order, cleanly” problem well. What no in-process pipeline can solve is the other half: a pipeline in memory still dies with the process, and all its progress dies with it.

What I actually needed was a function that can run for a long time, remember how far it got, and pick up where it left off. That’s not a pattern you bolt onto a normal function. It’s a different tool, and Azure ships it: Durable Functions.

Three Functions, One Flow

A durable workflow is made of three kinds of functions, and the separation is the whole idea.

The client is a normal function with a normal trigger - HTTP, queue, timer, whatever starts your process. Its only durable job is to schedule an orchestration and get out of the way:

[Function(nameof(StartProvisioning))]
public async Task Run(
    [ServiceBusTrigger("provisioning")] ProvisioningMessage message,
    [DurableClient] DurableTaskClient client)
{
    await client.ScheduleNewOrchestrationInstanceAsync(
        nameof(SiteSetupOrchestrator),
        message.ToSetupRequest());
}

The orchestrator is the recipe. It calls the steps in order and holds the workflow logic, but does no real work itself:

[Function(nameof(SiteSetupOrchestrator))]
public async Task RunOrchestrator(
    [OrchestrationTrigger] TaskOrchestrationContext context)
{
    var request = context.GetInput<SiteSetupRequest>();

    var site = await context.CallActivityAsync<SiteInfo>(nameof(CreateSite), request);
    await context.CallActivityAsync(nameof(ActivateFeatures), site);
    await context.CallActivityAsync(nameof(ApplyTemplate), site);
    await context.CallActivityAsync(nameof(SeedFolders), site);
    await context.CallActivityAsync(nameof(AddMemberGroups), site);
    await context.CallActivityAsync(nameof(RegisterSite), site);
}

The activities are where the actual SharePoint and Graph calls live. Each one is a small, self-contained function: CreateSite creates a site, ApplyTemplate applies a template, and so on.

What’s happening here?

1. The client schedules an orchestration instance and returns immediately. Nothing waits around; the run now has a life of its own, with an instance ID you can use to check on it later.
2. The orchestrator reads like a script: six awaited steps, top to bottom. This is the part that would be a state machine diagram in Logic Apps or a pile of queues in a hand-rolled solution. Here it’s just code, with loops and try/catch available when you need them.
3. Every CallActivityAsync is a checkpoint boundary. The runtime records that the step completed, and what it returned, before moving on.
4. Activities take input, do I/O, return output. They don’t know they’re part of a workflow, which keeps them easy to write and easy to test.

The “Durable” Part Is a History Table

Here’s what separates this from a normal async method: the orchestrator’s progress doesn’t live in memory. Every scheduled activity and every result gets written to a history table in storage as an event.

When the process dies mid-run - and over a multi-minute workflow across enough customers, it will - the runtime picks the orchestration up on another instance and replays it. The orchestrator code runs again from the top, but every activity that already completed doesn’t re-execute; its recorded result comes straight back from history. The replay fast-forwards through the finished steps in milliseconds and resumes real work at the first step without a recorded result.

It’s a save file in a video game. The console losing power doesn’t send you back to level one; you reload and continue from the last save point. The orchestrator saves after every activity, and it never plays a level twice.

That same mechanism is why an orchestration can wait for days without costing you anything. Awaiting a durable timer or an external event just means “write the wait to history and unload from memory”. No process is alive, no compute is billed, and when the timer fires next week, replay reconstructs the state and continues. A function that sleeps for a week, literally.

What Else Is in the Box

Sequential steps are the basics. A few more capabilities, through the provisioning lens:

Per-step retries. Every activity call can carry a retry policy, and this is the built-in retry logic worth knowing about. Mine looks like this:

private static readonly TaskOptions ActivityRetryOptions = new(
    new RetryPolicy(
        maxNumberOfAttempts: 10,
        firstRetryInterval: TimeSpan.FromSeconds(5),
        backoffCoefficient: 2.0,
        maxRetryInterval: TimeSpan.FromMinutes(5)));

await context.CallActivityAsync(nameof(ApplyTemplate), site, ActivityRetryOptions);

If ApplyTemplate throws because SharePoint returned a 429, the runtime waits 5 seconds and reruns the activity. Then 10, then 20, backing off exponentially up to 5 minutes, for up to 10 attempts. The orchestration just sits there durably waiting between attempts. For workflows built on APIs that throttle, this one feature pays for the whole ticket.

Fan-out/fan-in. Steps that don’t depend on each other can run as parallel activities and be awaited together with Task.WhenAll. Seeding twelve folders one by one is a for-loop; seeding them as twelve parallel activities is the same loop without the awaits until the end.

Durable timers. context.CreateTimer(...) gives you delays and polling loops that survive restarts. Waiting 30 seconds for SharePoint to finish creating a site is a timer, not a Task.Delay and a prayer.

External events. An orchestration can pause until something outside signals it - WaitForExternalEvent is how you build “wait for an admin to approve, then continue”, even if the admin takes four days.

Status from the outside. Every instance can be queried by ID: is it running, what step is it on, did it fail and why. My provisioning engine exposes this so the business system can show “site is being created” instead of a shrug.

What It Doesn’t Do for You

An honest intro should include the bill.

Retries are per step, not per run. The retry policy reruns a failing activity. If an activity exhausts all ten attempts, the orchestration fails, and a failed run stays failed - nothing in the box restarts it from where it stopped. Resuming a dead run is a pattern you build yourself on top of these primitives.

Retried activities rerun whole. When attempt two starts, it starts from the activity’s first line, not from where attempt one died. An activity that created a site and then failed will try to create the site again. Every activity has to be written to survive running twice, and that’s on you.

Orchestrator code lives under house rules. Because the orchestrator replays, it must be deterministic: same history in, same decisions out. No DateTime.UtcNow, no Guid.NewGuid(), no HTTP calls, no reading config that might change between replays. Anything nondeterministic belongs in an activity. The rules are simple but unforgiving, and the failure mode is a corrupted run, not a compile error.

None of these are dealbreakers. They’re the shape of the tool: the runtime guarantees your progress is never lost, and in exchange you write steps that tolerate being repeated.

Gotchas

• Use the replay-safe logger in orchestrators. context.CreateReplaySafeLogger(...) suppresses log lines during replay. A regular ILogger logs every completed step again on every replay, and the duplicated log stream will have you debugging problems you don’t have.
• Keep activities small and single-purpose. One activity per step means one checkpoint per step. A mega-activity that does five things gives you one checkpoint for five steps, and a failure at thing four reruns things one through three.
• The retry policy can’t tell a 429 from a typo. It retries every exception the same way, including the ones that will never succeed, like a broken config value. Distinguishing “try again” from “stop, a human needs to fix this” takes extra work.
• Don’t sneak I/O into the orchestrator. It compiles, it works on the happy path, and it breaks determinism the first time a replay gets a different answer. If it touches the network or the clock, it’s an activity.

Wrapping Up

A Durable Function is a workflow that remembers where it was: an orchestrator sequences the steps, activities do the work, and the runtime checkpoints every step so crashes, restarts, and week-long waits don’t lose progress. If your process has numbered steps, talks to flaky APIs, and takes longer than a request, it’s a fit - provisioning was mine.

And for a while, every single request did the same dumb thing: download the blob from storage, push the bytes to the browser. The file hadn’t changed since five seconds ago. The user’s browser literally had an identical copy already. Didn’t matter - full download from storage, full response to the client, every time. I was paying latency and moving bytes just to deliver files nobody actually needed re-delivered.

The fix was already sitting in every storage response I’d been ignoring: the ETag.

Thirty Seconds on ETags

Every blob response (and every properties call) includes an ETag header, something like "0x8DC5F3A2B1E4D70". Storage changes it whenever the blob’s content or metadata changes. It costs nothing, it’s always there.

On the HTTP side, the handshake is old and boring and great: the server sends ETag with a response; the browser saves it; next time the browser asks for the same URL it includes If-None-Match: <that etag>; if the server still has the same version, it answers 304 Not Modified with no body, and the browser uses its cached copy.

Boring. Reliable. Built into every browser since forever.

The Naive Version

My first pass was the obvious one:

var response = await blob.DownloadStreamingAsync(cancellationToken: ct);
return new FileStreamResult(response.Value.Content, contentType);

Works fine. But look at what the function actually is in this setup: a photocopier standing between storage and the browser, dutifully copying files that both sides already agree on. The browser has the file. Storage knows the file hasn’t changed. And my function in the middle is the only one who never asked.

The Trick: Be a Pipe, Not a Cache

The Azure SDK supports conditional requests natively. So instead of checking anything myself, I just hand the browser’s ETag straight to storage:

public async Task<AssetResponse?> GetAsync(string path, string? ifNoneMatch, CancellationToken ct)
{
    var blob = _container.GetBlobClient(path);
    var options = new BlobDownloadOptions();

    // Pass the client's ETag to the storage service as a conditional request.
    // Azure Blob Storage will short-circuit at the network level and return a 304,
    // meaning we never transfer the file body when the client is up to date.
    if (!string.IsNullOrEmpty(ifNoneMatch))
        options.Conditions = new BlobRequestConditions { IfNoneMatch = new ETag(ifNoneMatch) };

    try
    {
        var response = await blob.DownloadStreamingAsync(options, ct);

        if (response.GetRawResponse().Status == 304)
        {
            // Client's copy is current - signal it without a body.
            return new AssetResponse { ETag = ifNoneMatch!, IsNotModified = true };
        }

        var details = response.Value.Details;
        return new AssetResponse
        {
            ETag = details.ETag.ToString(),
            Content = response.Value.Content,
            ContentType = details.ContentType,
        };
    }
    catch (RequestFailedException ex) when (ex.Status == 404)
    {
        return null;
    }
}

What’s happening here?

1. The browser’s If-None-Match value goes into BlobRequestConditions.IfNoneMatch. That turns the download into a conditional request, the same ETag handshake, just one hop deeper.
2. The comparison happens inside Azure Storage, not in my code. On a match, storage answers 304 and the response has no body. The file bytes never even reach my function.
3. On a miss (file changed, or first visit), it’s a normal download, and DownloadStreamingAsync gives me a live stream I pass straight through without buffering the whole file in memory.
4. Missing blob → the RequestFailedException filter turns a 404 into a null, which the endpoint maps to a proper NotFound.

There is no dictionary of ETags, no memory cache, no invalidation logic. I didn’t build a cache, I connected two caches that already existed: the browser’s and storage’s own knowledge of its blobs.

Finishing the Loop Toward the Browser

The function endpoint relays the result:

req.HttpContext.Response.Headers[HeaderNames.ETag] = asset.ETag;

if (asset.IsNotModified)
    return new StatusCodeResult(StatusCodes.Status304NotModified);

req.HttpContext.Response.Headers[HeaderNames.CacheControl] = asset.IsImmutable
    ? "public, max-age=31536000, immutable"
    : "no-cache";

return new FileStreamResult(asset.Content!, asset.ContentType!);

Two details worth pausing on:

• no-cache doesn’t mean “don’t cache”. It means “cache it, but revalidate before using it”. That revalidation is exactly the If-None-Match round trip, which the pass-through just made nearly free - a header-only 304 instead of a file download. This is what index.html gets.
• Content-hashed files skip the conversation entirely. My build tool outputs filenames like app.ByJ3R0Az.js - the hash is the version. Those get max-age=31536000, immutable, so the browser never revalidates them at all. A new deploy produces a new filename, which is simply a different URL. The ETag dance is only for files whose names stay stable while their content changes.

Why I Like This Better Than a Memory Cache

My first instinct was an IMemoryCache of blob contents in the function. I’m glad I resisted:

• Nothing to size. No “how many MB of blobs do I keep in memory” question.
• Nothing to invalidate. The ETag comparison is against live storage, so a deploy is visible on the very next request. Stale-cache bugs can’t exist because there’s no cache to go stale.
• Scale-out safe. Ten function instances behave identically because none of them hold state. A per-instance memory cache would give ten different answers for 60 seconds after every deploy.

The thing that owns the data does the validation. Everyone else just forwards headers. 📮

Gotchas

• Round-trip the ETag string untouched. The quotes are part of the value. Trim them, “clean them up”, or re-wrap them and the comparison silently never matches again - everything still works, you just download every file every time and never notice.
• No ETag out, no If-None-Match back. Browsers only revalidate if your response included the ETag header in the first place. Forget it on one branch (error paths are a classic) and that file is a full download forever.
• no-cache still costs one round trip per file per load. That’s the deal: a header-only 304 instead of a body. For files that never change, don’t negotiate - content-hash the filename and go immutable.

Wrapping Up

The cheapest download is the one that never happens. Blob Storage already fingerprints every blob and already speaks conditional requests - most backends just never pass the browser’s question along. Forward If-None-Match, relay the 304, and let the two parties who actually know the answer talk to each other.

And that requirement quietly kills all the easy hosting options. Blob Storage’s static website feature? Public. A plain CDN? Public. The moment “who is asking” matters for every single file - not just the pages, the images and search index too - the files have to live somewhere private, and something with auth has to sit in front and hand them out.

My something is one Azure Function. One endpoint. It serves the entire site.

The Blob Container Is the Filesystem

There’s no clever mapping layer. I upload the build output into the container exactly as the generator produced it:

index.html
guides/onboarding/index.html
guides/expenses/index.html
assets/main.css
assets/search.js
images/office-map.png

The container layout is the URL space. Deploying a new version of the site is one CLI command:

az storage blob upload-batch --source ./public --destination site-content --overwrite

The Catch-All Function

Here’s the whole thing, trimmed to its skeleton:

[Function("Site")]
public async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "{*path}")] HttpRequest req,
    string? path,
    CancellationToken ct)
{
    // Every file goes through this gate - pages, scripts, images, all of it.
    if (!await _auth.IsSignedInAsync(req))
        return new UnauthorizedResult();

    var blobPath = MapToBlobPath(path);
    var blob = _container.GetBlobClient(blobPath);

    try
    {
        var response = await blob.DownloadStreamingAsync(cancellationToken: ct);
        var contentType = response.Value.Details.ContentType ?? ContentTypeFor(blobPath);
        return new FileStreamResult(response.Value.Content, contentType);
    }
    catch (RequestFailedException ex) when (ex.Status == 404)
    {
        return new NotFoundResult();
    }
}

private static string MapToBlobPath(string? path)
{
    if (string.IsNullOrEmpty(path))
        return "index.html";                    // "/" → the front page

    if (path.EndsWith('/') || !Path.HasExtension(path))
        return $"{path.TrimEnd('/')}/index.html"; // "/guides/onboarding/" → its index

    return path;                                 // "/assets/main.css" → as-is
}

What’s happening here?

1. Route = "{*path}" is the whole trick. The * makes it a catch-all: one route binding matches /, /guides/onboarding/, /assets/main.css, anything. One function, every file on the site.
2. The auth guard runs before anything touches storage. That’s the entire reason this setup exists - a static host can protect a site, this protects every byte. (How IsSignedInAsync works - cookies, Entra ID, whatever fits - is its own topic; the point is it’s one if at the top.)
3. MapToBlobPath does the job a web server normally does silently: default documents. / becomes index.html, extension-less routes like /guides/onboarding become guides/onboarding/index.html. Forget this and your front page is a 404.
4. DownloadStreamingAsync returns a live stream, and FileStreamResult pipes it straight to the response. The function never buffers a whole file in memory - a 4 MB image flows through, it doesn’t land here.
5. A missing blob throws RequestFailedException with status 404; the exception filter turns that into a clean NotFoundResult instead of a pre-flight existence check (which would just be a second storage call).

Content Types: The Unglamorous Part That Breaks Everything

If you serve HTML with the wrong Content-Type, the browser doesn’t render your page, it downloads it. Ask me how I know.

Best option: set correct content types on the blobs at upload time (upload-batch infers most of them). But blobs uploaded by hand or by older scripts often end up as application/octet-stream, so I keep a fallback map:

private static string ContentTypeFor(string path) => Path.GetExtension(path).ToLowerInvariant() switch
{
    ".html" => "text/html",
    ".css"  => "text/css",
    ".js"   => "text/javascript",
    ".json" => "application/json",
    ".svg"  => "image/svg+xml",
    ".png"  => "image/png",
    ".woff2" => "font/woff2",
    _ => "application/octet-stream",
};

What You Get for Free

• Auth on every file. Not just page-level protection - the org chart PNG and the search index JSON are exactly as protected as the pages.
• One deploy target. Build the site, upload-batch the folder, done. No web server to configure, nothing to restart.
• Generator-agnostic. The function doesn’t know or care if the folder came from Hugo, Astro, Docusaurus, or hand-written HTML.
• Cheap environments. Staging is just a second container and one config value. Rollback is re-uploading yesterday’s build folder.

One More Trick: Stop Re-Downloading Unchanged Files

There’s an elephant in the version above: every request downloads the full file from storage and pushes it to the browser - even when the file hasn’t changed in weeks and the browser has a perfect copy from five minutes ago.

The fix is already sitting in Blob Storage: every blob has an ETag, a version fingerprint that changes on every write. Browsers already know the game - once they’ve seen an ETag header, they send it back as If-None-Match on the next request. All the function has to do is forward that header into the SDK call:

var options = new BlobDownloadOptions();

var ifNoneMatch = req.Headers.IfNoneMatch.ToString();
if (!string.IsNullOrEmpty(ifNoneMatch))
    options.Conditions = new BlobRequestConditions { IfNoneMatch = new ETag(ifNoneMatch) };

var response = await blob.DownloadStreamingAsync(options, ct);

if (response.GetRawResponse().Status == 304)
{
    // Browser's copy is current - Azure never sent us the body at all.
    req.HttpContext.Response.Headers[HeaderNames.ETag] = ifNoneMatch;
    return new StatusCodeResult(StatusCodes.Status304NotModified);
}

req.HttpContext.Response.Headers[HeaderNames.ETag] = response.Value.Details.ETag.ToString();
var contentType = response.Value.Details.ContentType ?? ContentTypeFor(blobPath);
return new FileStreamResult(response.Value.Content, contentType);

The beautiful part: the ETag comparison happens inside Azure Storage, not in your code. On a match, storage answers 304 with no body - the file bytes never reach the function, and the function relays a bare 304 to the browser. No server-side cache, nothing to invalidate, and it works identically across scaled-out instances because nobody holds any state. The browser is the cache, Azure is the validator, and the function stays what it was: a pipe.

Two rules to make it stick: always set the ETag response header (no ETag out means the browser never asks conditionally again), and pass the value through untouched - the quotes are part of it, and “cleaning them up” silently breaks the match forever.

Gotchas

• You are the web server now. Default documents, trailing slashes, 404 pages - all the invisible things a real web server does are your job. MapToBlobPath above is the minimum, not the maximum.
• SPA? Then unknown routes need a fallback. For a client-side-routed app, a route with no matching blob should serve index.html (200, not 404) and let the router sort it out. For a docs site like mine, a real 404 is correct.
• Cold starts sit in front of your CSS. The function is in the path of every byte, so a consumption-plan cold start delays the whole page, not just an API call. For an internal tool I can live with it; know your tolerance.

Wrapping Up

One catch-all route, a private container, and a twenty-line function: a whole authenticated website with no web server to run. Add the If-None-Match pass-through and unchanged files stop moving entirely. The container is the filesystem, the function is the doorman. 🚪

Both approaches work. Both have real trade-offs. And for a long time I did not have a strong opinion either way. Then I started testing with Dev Proxy and suddenly the answer became very clear to me.

What Is a PnP Provisioning Template?

If you have not worked with PnP provisioning before: a PnP Provisioning Template is an XML (or JSON) file that describes the structure of a SharePoint site. It can contain:

• Lists and libraries — custom columns, views, and settings
• Content types — site or list content types with their fields
• Pages and web parts — modern pages, hero sections, text blocks
• Navigation and branding — site navigation structure, themes, logos

You apply a template to a site using the PnP Framework library for .NET, which reads the file and provisions everything described in it. The library handles the order of operations, dependencies between objects, and a lot of edge cases you would rather not think about.

The typical entry point looks like this:

web.ApplyProvisioningTemplate(template, applyInfo);

One call. One template. PnP handles the rest — until throttling enters the picture.

The Monolith Approach

The simplest way to structure your provisioning is one template that contains everything. All lists, all content types, all pages, all navigation — in a single file.

Pros:

• ✅ Simple to manage — one file, one version, one deployment artifact
• ✅ Easy to apply — a single ApplyProvisioningTemplate call
• ✅ PnP handles dependencies between objects internally

Cons:

• ❌ All-or-nothing retry — if anything fails, you restart from the beginning
• ❌ Long templates take a long time to apply, which means long retries too
• ❌ Hard to reason about what failed and where

The all-or-nothing behavior is the killer. SharePoint throttles aggressively under load, and ApplyProvisioningTemplate does not pick up where it left off. Under a sustained throttle — the kind where even 15 retries at the CSOM call level are not enough to get through — the template application eventually throws a MaximumRetryAttemptedException — a PnP-specific exception signalling that all retry attempts were used up. At that point your outer retry logic has no choice but to start the whole template over from the very first list.

The Modular Approach

The alternative is splitting your template into multiple smaller files, each representing a logical feature or concern. For example:

• 01-lists-and-libraries.xml
• 02-content-types.xml
• 03-pages.xml
• 04-navigation.xml

You apply them in sequence, and after each one completes successfully you record that fact in memory. If a template fails, you retry only from the failed template — not from the beginning.

Pros:

• ✅ Granular retry — only the failed feature is re-applied
• ✅ Faster recovery from throttling
• ✅ Easier to reason about failures (“pages failed, lists are fine”)
• ✅ Easier to test individual features in isolation

Cons:

• ❌ More files to manage and version
• ❌ Ordering matters — you need to think about dependencies
• ❌ Slightly more orchestration code on your end

The trade-off is real. Modular means more moving parts. But for resilience, the payoff is significant.

Testing With DevProxy — Where It Got Interesting

I wanted to put both approaches through their paces under realistic throttling conditions. For that I used Dev Proxy with two plugins:

• GenericRandomErrorPlugin — injects random 429 responses to simulate SharePoint throttling unpredictably
• RateLimitingPlugin — caps requests per second to trigger organic throttling at volume

With these configured, I ran my provisioning engine against both template structures.

The result with the monolithic template: PnPClientContext did its job — it absorbed a lot of 429s silently and kept going. But under sustained throttling, the retries ran out, PnP threw a MaximumRetryAttemptedException, and my outer retry had to restart it from scratch. From list number one. Even though we were deep into page provisioning when it failed.

The result with modular templates: the same CSOM-level retry absorbed the same throttling. But when a sustained burst finally exhausted PnP’s retries on the pages template, my retry loop skipped straight to 03-pages.xml — lists and content types had already been marked done in memory and were left alone.

This is exactly the kind of thing that looks fine in a happy-path test, but silently destroys your provisioning SLA in production.

PnPClientContext: Bump the Retry Count for Provisioning

The PnP Framework’s internal CSOM calls already use ExecuteQueryRetry() — an extension method that handles 429 throttling with exponential backoff, respecting the Retry-After header SharePoint returns. You get this for free with a plain ClientContext. The default is 10 retries.

If you expect heavier throttling during provisioning — and a large site with lots of lists, content types, and pages is a reasonable place to expect it — you can easily raise that limit by converting to a PnPClientContext:

using var siteCtx = CSOMClientFactory.Create(provCtx.NewSiteUrl, credential);
var pnpCtx = PnPClientContext.ConvertFrom(siteCtx, retryCount: 15);

That is the entire change. The provisioning code picks up the higher retry count automatically. Your outer retry logic only comes into play if throttling is so sustained that even 15 attempts on a single CSOM call are not enough — at which point PnP throws a MaximumRetryAttemptedException.

Idempotency Strategy

The key property that makes modular templates safe to retry is idempotency — applying a template that has already been applied should not cause errors or duplicate data. PnP handles most of this for you: before creating a list, content type, or page, it checks whether it already exists.

A very simple example of how this looks in practice:

foreach (var source in config.Templates)
{
    var template = LoadTemplate(source);
    pnpCtx.Web.ApplyProvisioningTemplate(template, applyInfo);
}

If this loop is re-invoked after a failure (for example by an Azure Function retry policy), templates that already completed will be re-applied — but because each template is idempotent, that is safe. The already-provisioned lists and pages are left untouched. The only cost is the time it takes to re-apply them.

That wasted time is the whole point. With a monolithic template, a retry means re-applying everything from scratch. With modular templates, it means re-applying a small number of already-done features before reaching the one that actually failed.

Comparison

Verdict

Neither approach is universally correct. If you are provisioning simple sites with low template complexity and you are not worried about throttling, a monolithic template is perfectly fine and a lot less work to maintain.

But if you are building a production provisioning engine that needs to be resilient — where throttling is a real risk, where retries need to be efficient, and where you care about how long a failure takes to recover from — modular templates win clearly.

The combination that works best for me is: modular templates + PnPClientContext with conservative retry settings. PnPClientContext handles the low-level CSOM throttling so most 429s never surface at all. When they do surface (and eventually they will), modular templates ensure your retry loop does the minimum amount of work to get back on track.

Conclusion

The biggest takeaway for me is not even the monolith-vs-modular debate. It is that I never would have found this out without DevProxy. Happy-path testing showed no difference between the two approaches. It was only when I introduced realistic throttling that the behavior diverged in a way that actually matters.

If you are building any kind of SharePoint automation that makes CSOM or Graph calls under load, DevProxy should be a standard part of your testing setup. It is free, it integrates with your local development environment, and it reveals exactly the kind of subtle failure modes that only appear in production.

References

• PnP Framework on GitHub
• Dev Proxy overview
• SharePoint throttling and retry guidance
• PnPClientContext — ExecuteQueryRetry docs