So there I was, building what I thought was a pretty straightforward API endpoint. Users fill out a form, click save, and boom—SharePoint list item gets updated. Simple, right?

Wrong.

It was during one of those routine maintenance windows when I decided to review some API logs that I noticed something odd. The numbers just didn’t add up. We were making way more SharePoint API calls than seemed necessary for the amount of actual data changes happening in the system.

Every operation was hitting SharePoint. Even when nothing had actually changed.

That’s when it hit me. Our application was treating every potential update as an actual update, regardless of whether the data was different from what SharePoint already had. Users could open a form, change nothing, click save, and we’d still fire off an API call to “update” the item with identical values.

But SharePoint doesn’t care if you’re setting a field to the exact same value it already has. It still counts that as an API call. It still hits your throttling limits. And if you’re paying for “Service Prioritization in SharePoint”? It still costs you money.

The “Why Did I Even Build This?” Moment

I spent some time digging deeper into the patterns, and the numbers were… enlightening.

This wasn’t just a user behavior issue. It was happening everywhere:

• Users frequently save forms without making actual changes
• Automated processes periodically “sync” data that’s already current
• Applications send complete object states rather than just changed fields
• Integration systems perform routine updates as part of larger workflows

When I looked at a typical day’s worth of operations, I found that roughly 70% of our SharePoint update calls weren’t actually changing anything. We were essentially paying SharePoint to accept identical data and politely say “thanks for the update!”

The more I thought about it, the more I realized this pattern was probably happening in systems everywhere. How many developers have built the same “just update everything” approach without stopping to ask whether anything actually changed?

SharePoint’s “Sure, I’ll Take Your Money” Attitude

Here’s the thing about SharePoint’s CSOM that nobody really talks about: it doesn’t care if you’re being wasteful. You want to set a field to the exact same value it already has? “No problem!” says SharePoint. “That’ll be one API call, please.”

And if you’re using Service Prioritization in SharePoint? That’s real money walking out the door.

Let me show you what I mean with some concrete numbers from a client I worked with recently:

The Head-Scratching Cases

// These should be considered equal, but .NET says "nope"
currentValue: "John Doe"
newValue: "John Doe"  // Easy case, no problem here

currentValue: null
newValue: ""  // Is empty string the same as null? Your call!

currentValue: 42
newValue: "42"  // Different types, same value. Fun times.

The SharePoint Special Cases

// User fields (oh boy...)
currentValue: FieldUserValue { LookupId = 15, LookupValue = "John Doe" }
newValue: FieldUserValue { LookupId = 15, LookupValue = "John Doe" }

// Multi-choice fields that hate you
currentValue: ["Option A", "Option C", "Option B"]
newValue: ["Option B", "Option A", "Option C"]  // Same options, different order

// Taxonomy fields (because why not make it complicated?)
currentValue: TaxonomyFieldValue { TermGuid = "abc-123", Label = "Technology" }
newValue: TaxonomyFieldValue { TermGuid = "abc-123", Label = "Technology" }

After an hour of testing different scenarios, I realized I needed something more sophisticated than oldValue == newValue.

Building the “Actually Smart” Comparison Engine

Alright, time to get serious. I needed a comparison function that could handle all of SharePoint’s quirky field types and still be fast enough to not slow down my API.

Here’s what I ended up building (and yes, it’s a bit of a beast):

internal static List<(string InternalName, object? NewValue)> GetDifferences(
    ListItem item,
    IDictionary<string, object?> newValues,
    bool treatEmptyStringAsNull)
{
    var diffs = new List<(string, object?)>();

    foreach (var kvp in newValues)
    {
        var name = kvp.Key;
        var newVal = kvp.Value;

        object? currentVal = null;
        if (item.FieldValues != null && item.FieldValues.TryGetValue(name, out var cv))
        {
            currentVal = cv;
        }

        if (!ValuesEqual(currentVal, newVal, treatEmptyStringAsNull))
        {
            diffs.Add((name, newVal));
        }
    }

    return diffs;
}

private static bool ValuesEqual(object? a, object? b, bool treatEmptyStringAsNull)
{
    if (ReferenceEquals(a, b))
    {
        return true;
    }

    if (a is null || b is null)
    {
        if (treatEmptyStringAsNull)
        {
            if ((a is null && IsEmptyStringLike(b)) || (b is null && IsEmptyStringLike(a)))
            {
                return true;
            }
        }
        return false;
    }

    a = Normalize(a, treatEmptyStringAsNull);
    b = Normalize(b, treatEmptyStringAsNull);

    if (a is IStructuralEquatable seA && b is IStructuralEquatable seB)
    {
        return StructuralComparisons.StructuralEqualityComparer.Equals(seA, seB);
    }

    return Equals(a, b);
}

private static bool IsEmptyStringLike(object? x)
{
    return x is string s && string.IsNullOrWhiteSpace(s);
}

private static object? Normalize(object? v, bool treatEmptyStringAsNull)
{
    if (v == null)
    {
        return null;
    }

    switch (v)
    {
        case string s:
            return (treatEmptyStringAsNull && string.IsNullOrWhiteSpace(s)) ? null : s;

        case bool b:
            return b;

        case byte or sbyte or short or ushort or int or uint or long or ulong or float or double or decimal:
            return Convert.ToDecimal(v, CultureInfo.InvariantCulture);

        case DateTime dt:
            var utc = (dt.Kind == DateTimeKind.Utc ? dt : DateTime.SpecifyKind(dt, DateTimeKind.Unspecified)).ToUniversalTime();
            return new DateTime(utc.Year, utc.Month, utc.Day, utc.Hour, utc.Minute, utc.Second, DateTimeKind.Utc);

        case FieldUserValue u:
            return u.LookupId;

        case FieldLookupValue l:
            return l.LookupId;

        case IEnumerable<FieldLookupValue> multiLookup:
            return multiLookup.Select(x => x?.LookupId ?? 0).OrderBy(x => x).ToArray();

        case TaxonomyFieldValue tx:
            return tx.TermGuid?.Trim().ToLowerInvariant();

        case TaxonomyFieldValueCollection txc:
            return txc
                .Where(x => x != null && !string.IsNullOrEmpty(x.TermGuid))
                .Select(x => x.TermGuid.Trim().ToLowerInvariant())
                .OrderBy(g => g)
                .ToArray();

        case FieldGeolocationValue geo:
            return new ValueTuple<double, double, double, double>(
                Math.Round(geo.Latitude, 6),
                Math.Round(geo.Longitude, 6),
                Math.Round(geo.Altitude, 2),
                Math.Round(geo.Measure, 2));

        case string[] ss:
            return ss
                .Select(x => treatEmptyStringAsNull && string.IsNullOrWhiteSpace(x) ? null : x)
                .OrderBy(x => x, StringComparer.Ordinal)
                .ToArray();

        case IEnumerable enumerable when v is not string:
        {
            var list = new List<object?>();
            foreach (var e in enumerable)
            {
                list.Add(Normalize(e, treatEmptyStringAsNull));
            }
            return list.ToArray();
        }

        default:
            return v;
    }
}

The Magic Behind the Scenes

1. The Normalization Dance

The Normalize method is where the real magic happens. It takes SharePoint’s various field types and converts them into something we can actually compare:

• Numbers: Everything becomes a decimal because SharePoint loves to mix integers and strings
• DateTime: Converted to UTC and truncated to seconds (because who needs millisecond precision for list items?)
• User/Lookup Fields: We only care about the LookupId, not the display text that might change
• Collections: Sorted alphabetically because SharePoint doesn’t guarantee order
• Strings: Can optionally treat empty/whitespace as null (trust me, you want this)

2. The “It’s Not You, It’s SharePoint” Handling

For arrays and complex objects, I use IStructuralEquatable to do deep comparisons. Because yes, SharePoint will absolutely give you arrays that contain the same items but in different orders.

3. The Business Logic Escape Hatch

The treatEmptyStringAsNull parameter saved my sanity. Different parts of your application might handle empty values differently, and this lets you define your own rules for what counts as “unchanged.”

But Wait, Why Not Just Use JSON Comparison?

Before you ask (because I know you’re thinking it): “Why build this elaborate comparison engine when you could just serialize both objects to JSON and compare the strings?”

Great question. I actually thought the same thing initially. How hard could it be, right? Just JsonSerializer.Serialize() both the old and new values, compare the resulting strings, and call it a day.

Here’s what the “simple” JSON approach looks like:

internal static List<(string InternalName, object? NewValue)> GetDifferencesJson(
    IDictionary<string, object?> oldValues,
    IDictionary<string, object?> newValues,
    bool treatEmptyStringAsNull)
{
    var diffs = new List<(string, object?)>();

    foreach (var kvp in newValues)
    {
        var name = kvp.Key;
        var newVal = kvp.Value;

        object? currentVal = null;
        oldValues?.TryGetValue(name, out currentVal);

        var newJson = JsonSerializer.Serialize(newVal);
        var oldJson = JsonSerializer.Serialize(currentVal);

        if (newJson != oldJson)
        {
            diffs.Add((name, newVal));
        }
    }

    return diffs;
}

Looks clean and simple, right? That’s what I thought too. So I built both approaches and put them head-to-head with BenchmarkDotNet. The results were… eye-opening.

The Performance Reality Check

Here’s what I found when comparing the performance of my custom approach versus JSON string comparison:

Single List Item Comparison:

• Custom Dictionary Approach: 749,625 operations/second (1.334 μs per operation)
• JSON String Approach: 201,109 operations/second (4.972 μs per operation)
• Performance difference: 3.7x faster with the custom approach

Batch Operations (300 items):

• Custom Dictionary Approach: 3,705 operations/second (269.91 μs per batch)
• JSON String Approach: 712 operations/second (1,404.06 μs per batch)
• Performance difference: 5.2x faster with the custom approach

Why JSON Comparison Falls Short

The numbers tell the story, but here’s what’s actually happening under the hood:

1. Serialization Overhead: Converting SharePoint field values to JSON strings requires multiple allocations and string operations
2. Memory Pressure: JSON approach allocated nearly 50% more memory (4.67KB vs 3.17KB per operation)
3. Type Conversion Issues: JSON serialization doesn’t handle SharePoint’s quirky field types the way we need
4. Garbage Collection: More allocations = more GC pressure = slower overall performance

When you’re processing hundreds or thousands of list items in batch operations, that 3-5x performance difference really adds up. Plus, the custom approach gives us complete control over how different field types are compared, which JSON comparison simply can’t provide.

Putting It All Together: The Real-World Implementation

Here’s how I actually use this in my APIs:

public async Task<bool> UpdateListItemAsync(int itemId, Dictionary<string, object?> newValues)
{
    var clientContext = GetClientContext();
    var list = clientContext.Web.Lists.GetByTitle("YourList");
    var listItem = list.GetItemById(itemId);
    
    clientContext.Load(listItem);
    clientContext.ExecuteQuery();

    // The moment of truth - what actually changed?
    var differences = GetDifferences(listItem, newValues, treatEmptyStringAsNull: true);
    
    if (!differences.Any())
    {
        // Nothing changed! Skip the update entirely
        return false; // "Thanks for playing, but you didn't actually change anything"
    }

    // Only update the fields that actually changed
    foreach (var (fieldName, newValue) in differences)
    {
        listItem[fieldName] = newValue;
    }

    listItem.Update();
    clientContext.ExecuteQuery();
    
    return true; // "Something actually happened!"
}

The Numbers Don’t Lie (And They’re Pretty Great)

After implementing this change detection across our applications, here’s what we typically see:

Before implementing change detection:

• Applications making thousands of update requests per day
• Every single one triggered a SharePoint ExecuteQuery()
• High API call volumes and occasional throttling issues

After implementing change detection:

• Same number of update requests from users/systems
• 60-80% contained no actual changes and were skipped
• Dramatically reduced SharePoint API consumption
• Much lower throttling risk

The typical impact:

• Significant reduction in monthly API calls
• Cost savings with Service Prioritization in SharePoint (varies by usage)
• Reduced throttling risk during busy periods
• Faster API responses because skipped operations are basically instant

But here’s what really matters: the performance improvement isn’t just about the numbers. Users notice when applications feel more responsive, and developers notice when they stop getting throttling alerts.

The Gotchas (Because There Are Always Gotchas)

When This Approach Might Not Be For You

1. Audit Requirements: If you need to log every single “save” action regardless of whether anything changed, this might not work for your use case.

2. Always Update Timestamps: Some business requirements dictate that “LastModified” should always be updated when a user clicks save, even if the content is identical.

3. Super Simple Scenarios: If you’re only updating one field and it’s a simple string comparison, the overhead of this elaborate comparison might not be worth it.

The SharePoint Field Types That Made Me Question My Life Choices

Some field types need special handling:

// Calculated fields - don't even try to update these
if (field.ReadOnlyField || field.FieldTypeKind == FieldType.Calculated)
{
    continue; // SharePoint will just ignore you anyway
}

// Attachment fields - these need completely different logic
if (field.FieldTypeKind == FieldType.Attachments)
{
    // Handle separately - attachments are their own special nightmare
    continue;
}

Wrapping Up (And Why This Matters More Than You Think)

Look, I know this might seem like over-engineering for a simple “update SharePoint item” operation. But here’s the thing: those small inefficiencies add up fast.

In my case, this one change:

• Improved user experience with faster response times
• Reduced throttling headaches during busy periods
• Made me feel like a responsible developer (this one’s important too!)

The pattern I’ve shown you isn’t just about SharePoint or just about API optimization. It’s about asking the fundamental question: “Is this operation actually necessary?”

That question has made me a better developer across all the platforms I work with, not just SharePoint.

So next time you’re building any kind of update operation—whether it’s SharePoint, a database, or that JSON file you’re definitely not using as a database—pause for a moment and ask: “Did anything actually change?”

Your future self (and your API bills) will thank you.

Pro tip: Start implementing this pattern on your highest-traffic endpoints first. That’s where you’ll see the biggest impact, and it’ll give you the confidence to roll it out everywhere else.

Now go forth and eliminate those unnecessary API calls. Your SharePoint environment will purr like a happy cat.

Change detection is one of five techniques in The SharePoint CSOM Performance Playbook. If you want the whole picture of where CSOM calls leak, start there.

Picture this: It’s 2 AM, you’re on your third cup of coffee, and you’re watching your perfectly crafted Microsoft Graph batch request fail spectacularly. Again.

You’ve got 25 files to download from SharePoint. Your batch processes 24 of them perfectly, then one lonely file decides to throw a throttling tantrum. What does your retry logic do? It throws away all 24 successful downloads and starts over. From scratch. Like a digital Groundhog Day, but less amusing and more soul-crushing.

Sound familiar? Welcome to the “retry everything” club – where perfectly good API calls go to die unnecessarily. 😅

The Grocery Cart Problem (Or: Why We’re Doing This Wrong)

Let me paint you a picture. You’re at the grocery store with a cart full of 20 items. You get to checkout, and the cashier says, “Sorry, we’re out of milk.”

What would you do?

• Option A: Put back everything, go home, and come back later to shop for all 20 items again
• Option B: Buy the 19 items you can get, then come back just for the milk

If you picked Option A, congratulations – you think like most API retry logic! If you picked Option B, you’re ready to learn about smart retries.

The “retry everything” approach is like Option A, and here’s why it’s bonkers:

• 🔄 Wasted effort: You’re re-requesting stuff that already worked perfectly
• 🐌 Slower performance: Users wait longer while you redo successful work
• 📈 Throttling amplification: You’re actually making the problem worse by hitting successful endpoints again
• 🔍 Poor debugging: Can’t easily identify which specific requests are the real troublemakers

I learned this the hard way when I watched a simple file sync turn into an API call avalanche. 20 requests became 40, then 80, then… well, let’s just say Microsoft’s throttling system got very acquainted with my application.

The “Aha!” Moment (It’s Simpler Than You Think)

The solution hit me during one of those 2 AM debugging sessions: What if we only retry the stuff that actually failed?

Revolutionary, right? 😏

Here’s the beautiful thing – this isn’t some PhD-level computer science. It’s just common sense applied to code. Keep the winners, retry the losers. Simple.

But (there’s always a “but”), there’s one sneaky technical challenge that makes this trickier than it sounds. Microsoft’s Graph SDK has a helpful method called NewBatchWithFailedRequests(), but it has a quirk: it generates brand new request IDs. This breaks your ability to map responses back to your original data.

Think of it like this: You order pizza for table 5, but when they bring the replacement slice, they call it table 23. Good luck figuring out who ordered what!

If you’re new to Graph batching or request mapping, I’d recommend checking out my post on Graph Batching for File Content: Mapping Requests to Responses first. It’s like the prequel to this story – explains how to keep track of what’s what when dealing with batch responses.

Quick Win Summary (For the Impatient Developers)

The Problem: Your retry logic is like that friend who starts the entire conversation over when they missed one word. Inefficient and annoying.

The Solution: A drop-in extension method that only retries the actual failures while keeping successful responses safe and sound.

The Payoff:

• ⚡ Faster operations (no more re-downloading working files)
• 📉 Fewer API calls (your rate limits will thank you)
• 🎯 Less throttling (stop beating dead endpoints)
• 😌 Happier users (and happier you at 2 AM)

The Catch: You need to understand request-to-response mapping. Don’t worry, it’s not rocket science, and I’ve got a whole post about it.

Time Investment: About 5 minutes to implement, countless hours of frustration saved.

Ready for the nitty-gritty? Let’s dive in! 👇

The Hero of Our Story: The Smart Retry Extension

Okay, here’s where we get our hands dirty. The main challenge isn’t just filtering out successful requests – it’s that pesky NewBatchWithFailedRequests method that scrambles your request IDs like eggs at Sunday brunch.

Here’s the extension method that saves the day:

internal static class GraphServiceClientExtensions 
{
    internal static async Task<(IReadOnlyDictionary<string, HttpStatusCode> Statuses, Dictionary<string, HttpResponseMessage> BatchResponse)> 
        PostBatchWithFailedDependencyRetriesAsync(this GraphServiceClient graphClient, BatchRequestContentCollection originalBatch) 
    {
        const int maxRetries = 5;
        TimeSpan delay = TimeSpan.FromSeconds(1);

        Dictionary<string, HttpResponseMessage> allResponses = new Dictionary<string, HttpResponseMessage>();
        Dictionary<string, HttpStatusCode> allStatuses = new Dictionary<string, HttpStatusCode>();

        BatchRequestContentCollection batchToSend = originalBatch;

        for (int attempt = 1; attempt <= maxRetries; attempt++) 
        {
            BatchResponseContentCollection batchResponse = await graphClient.Batch.PostAsync(batchToSend);
            Dictionary<string, HttpStatusCode> responses = await batchResponse.GetResponsesStatusCodesAsync();

            // Filter out failures (excluding redirects which are normal for file content)
            Dictionary<string, HttpStatusCode> failedRequests = responses
                .Where(kvp => !BatchResponseContent.IsSuccessStatusCode(kvp.Value) && kvp.Value != HttpStatusCode.Found)
                .ToDictionary(kvp => kvp.Key, kvp => kvp.Value);

            // Collect all responses from this attempt
            foreach (var kvp in responses) 
            {
                var response = await batchResponse.GetResponseByIdAsync(kvp.Key);
                allResponses[kvp.Key] = response;
                allStatuses[kvp.Key] = kvp.Value;
            }

            if (failedRequests.Count == 0 || attempt == maxRetries) 
            {
                break;
            }

            await Task.Delay(delay);
            delay = TimeSpan.FromSeconds(delay.TotalSeconds * 2);

            // The key problem: NewBatchWithFailedRequests creates new request IDs!
            batchToSend = batchToSend.NewBatchWithFailedRequests(responses);
            
            // This is why we need this method - restore the original request IDs
            await RestoreOriginalRequestIdsAsync(batchToSend, originalBatch);
        }

        return (allStatuses, allResponses);
    }

    private static async Task RestoreOriginalRequestIdsAsync(
        BatchRequestContentCollection newBatch, 
        BatchRequestContentCollection originalBatch) 
    {
        var stepsSnapshot = newBatch.BatchRequestSteps.ToArray();

        foreach (var kvp in stepsSnapshot) 
        {
            var oldStepId = kvp.Key;
            var step = kvp.Value;
            var requestPath = step.Request.RequestUri!.AbsolutePath;

            // Find the original request ID by matching the request path
            var matchingOriginal = originalBatch.BatchRequestSteps
                .First(x => x.Value.Request.RequestUri!.AbsolutePath == requestPath);

            var originalStepId = matchingOriginal.Key;

            if (oldStepId != originalStepId) 
            {
                newBatch.RemoveBatchRequestStepWithId(oldStepId);

                var newStep = new BatchRequestStep(
                    requestId: originalStepId,
                    httpRequestMessage: step.Request,
                    dependsOn: step.DependsOn);

                newBatch.AddBatchRequestStep(newStep);
            }
        }
    }
}

What’s happening here? Think of it as a diplomatic negotiator for your API calls:

1. The ID Shuffle Problem: NewBatchWithFailedRequests gives failed requests shiny new IDs, like witness protection for HTTP requests
2. The Detective Work: RestoreOriginalRequestIdsAsync plays detective, matching requests by their paths to find their original identities
3. The Happy Reunion: Failed requests get their original IDs back, so your mapping dictionary doesn’t break down in tears

It’s like having a really good wedding planner who makes sure everyone sits at the right table, even after the venue changes.

Showtime: Watching Smart Retries in Action

Now let’s see our smart retry logic work its magic in a real-world scenario. Imagine you’re building a document sync tool and need to download 25 files from SharePoint. Some will work perfectly, others might throw tantrums due to throttling or network hiccups.

Here’s how the new approach handles it like a champ:

// Let's say you need to download content from 25 SharePoint files
var batch = new BatchRequestContentCollection(graphClient);
var fileMapping = new Dictionary<string, FileContentRequest>();

// Build the batch for file content downloads
var filesToDownload = await GetFilesToProcess(); // Your method to get file list

foreach (var file in filesToDownload) 
{
    var requestInfo = graphClient.Sites[siteId]
                                .Drives[driveId]
                                .Items[file.DriveItemId]
                                .Content
                                .ToGetRequestInformation();
    
    var requestId = await batch.AddBatchRequestStepAsync(requestInfo);
    
    // Map the request ID to your file info (same pattern as previous post)
    fileMapping[requestId] = new FileContentRequest 
    { 
        DriveItemId = file.DriveItemId,
        FileName = file.Name,
        ExpectedSize = file.Size,
        DownloadStartTime = DateTime.Now
    };
}

// 🎯 Here's where the magic happens - just one line change!
var (statuses, responses) = await graphClient.PostBatchWithFailedDependencyRetriesAsync(batch);

// Process results - this is where the retry really shines
var successfulDownloads = new List<FileDownloadResult>();
var failedDownloads = new List<string>();

foreach (var kvp in fileMapping) 
{
    var requestId = kvp.Key;
    var fileRequest = kvp.Value;
    
    if (responses.TryGetValue(requestId, out var response)) 
    {
        var statusCode = statuses[requestId];
        
        if (BatchResponseContent.IsSuccessStatusCode(statusCode)) 
        {
            // Success! Handle the file content
            var contentBytes = await response.Content.ReadAsByteArrayAsync();
            
            // Save to your desired location
            var localPath = Path.Combine(downloadFolder, fileRequest.FileName);
            await File.WriteAllBytesAsync(localPath, contentBytes);
            
            successfulDownloads.Add(new FileDownloadResult 
            { 
                FileName = fileRequest.FileName,
                LocalPath = localPath,
                ActualSize = contentBytes.Length,
                ExpectedSize = fileRequest.ExpectedSize,
                DownloadTime = DateTime.Now - fileRequest.DownloadStartTime
            });
        } 
        else 
        {
            // Even after smart retries, this file failed
            failedDownloads.Add($"{fileRequest.FileName} ({statusCode})");
            
            // Log the specific failure for debugging
            Console.WriteLine($"Failed to download {fileRequest.FileName}: {statusCode}");
        }
        
        // Always clean up the response
        response.Dispose();
    }
}

Console.WriteLine($"Successfully downloaded: {successfulDownloads.Count} files");
if (failedDownloads.Any())
{
    Console.WriteLine($"Failed downloads: {string.Join(", ", failedDownloads)}");
}

The beautiful part? Look at that line with the magic emoji 🎯. That’s literally the only change you need to make to your existing batch processing code. Everything else stays exactly the same.

Here’s what’s happening behind the scenes:

1. First batch attempt: Say 20 files succeed, 5 fail due to throttling
2. Smart filtering: Keep those 20 successful responses safe
3. Targeted retry: Build a new batch with just the 5 failures
4. ID preservation: Make sure those 5 retries still map to your original file info
5. Rinse and repeat: Maybe 4 of the 5 succeed on retry, leaving just 1 persistent troublemaker

The result? Instead of making 250 API calls (25 files × 5 retry attempts for the unlucky ones), you might only make 35 total calls. Your throttling problems become manageable, and files download way faster.

// Supporting classes for the example above
public class FileContentRequest 
{
    public string DriveItemId { get; set; } = string.Empty;
    public string FileName { get; set; } = string.Empty;
    public long ExpectedSize { get; set; }
    public DateTime DownloadStartTime { get; set; }
}

public class FileDownloadResult 
{
    public string FileName { get; set; } = string.Empty;
    public string LocalPath { get; set; } = string.Empty;
    public long ActualSize { get; set; }
    public long ExpectedSize { get; set; }
    public TimeSpan DownloadTime { get; set; }
}

The Method to the Madness (What’s Really Happening)

I know that extension method looks intimidating – like trying to read assembly instructions in a foreign language. But once you break it down, it’s actually pretty logical. Let me walk you through the step-by-step dance:

Step 1: The First Attempt “Let’s try everything once and see what happens”

Sends your original batch of 25 files and carefully captures every single response and status code. No throwing anything away yet.

Step 2: The Great Sorting “Okay, who succeeded and who’s being difficult?”

Separates the winners from the losers, but (and this is important) ignores redirect responses. Why? Because when you’re downloading large files, redirects are totally normal – SharePoint often redirects you to the actual storage location.

Step 3: The Preservation Society “Keep the good stuff safe while we deal with the troublemakers”

All successful responses get stored in a safe place while we build a new, smaller batch containing only the failed requests. It’s like having a really good filing system for your API responses.

Step 4: The Identity Crisis Resolution “Wait, who are you again? Let me check your original ID…”

This is the tricky bit! The NewBatchWithFailedRequests method gives everyone new IDs, like a witness protection program for HTTP requests. Our RestoreOriginalRequestIdsAsync method plays detective, matching requests by their URL paths to restore their original identities.

Step 5: The Polite Wait “Let’s not be pushy – maybe try again in a second?”

Implements exponential backoff – starts with a 1-second wait, then 2 seconds, then 4 seconds, etc. This prevents your app from being that annoying person who keeps knocking on the door every second.

Step 6: The Safety Net “Okay, we tried 5 times. Some files just aren’t meant to be downloaded today.”

Gives up after 5 attempts to prevent infinite retry loops. Because sometimes you need to know when to walk away from the poker table.

Why This Actually Works (The Science Behind the Magic)

Here’s what makes this approach so much better than the “retry everything” strategy:

🎯 Surgical Precision Only retry what actually failed – it’s like having a really good therapist who focuses on the actual problems instead of rehashing everything from childhood.

No wasted API calls on requests that already succeeded. If 24 out of 25 files downloaded perfectly, why punish them with another round trip?

⚡ Speed Demon Successful requests don’t get repeated, so everything finishes faster – sometimes dramatically faster.

Users see their successful downloads immediately while you quietly retry the problematic ones in the background.

🤝 Throttling-Friendly Fewer total requests means you’re less likely to hit Microsoft’s rate limits, and when you do, recovery is faster.

Instead of amplifying throttling issues, you’re actually helping to resolve them by reducing load on the endpoints that are already struggling.

🔄 Drop-in Simplicity Change literally one line of code and you’re done. No architectural rewrites, no complex state management – just swap out the method call.

Your existing error handling, logging, and business logic all stay exactly the same.

🔍 Debug Paradise Easy to see exactly which requests are consistently failing, making troubleshooting a breeze instead of a nightmare.

When file “ImportantDocument.pdf” fails on every retry attempt, you know there’s something specific about that file, not your entire batch logic.

The “Before and After” Moment

Let me paint you a picture of how this changes your life:

Before Smart Retries:

• 25 file batch fails on 3 files due to throttling
• Retry all 25 files → now 5 files fail due to increased throttling
• Retry all 25 files again → now 8 files fail
• You’re now in the throttling spiral of doom
• Users are staring at loading spinners
• You’re questioning your career choices

After Smart Retries:

• 25 file batch fails on 3 files due to throttling
• Keep the 22 successful files, retry only the 3 failures
• Maybe 2 of the 3 succeed on retry, leaving 1 stubborn file
• Final retry gets the last file, or you log it as a persistent issue
• Users get 24/25 files quickly, you sleep better at night

It’s the difference between being stuck in traffic because one lane is blocked (and everyone keeps switching to that lane), versus just using the open lanes and going around the problem.

The Bottom Line (And Why Your Future Self Will Thank You)

I’ll be real with you – when I first started working with Microsoft Graph batching, I thought the built-in retry policies were enough. “How hard could it be?” I thought. “APIs fail sometimes, just retry them!”

Then I built my first real-world document sync application. Suddenly, I was dealing with users uploading hundreds of files, enterprise throttling limits, and the occasional network hiccup that would bring the whole operation to a screeching halt.

That’s when I learned the hard way that “retry everything” is like using a sledgehammer to hang a picture frame. Sure, it might work, but you’re probably going to break some stuff in the process.

This selective retry approach has been a game-changer. Not just for performance (though users definitely notice when their bulk operations actually complete), but for debugging too. When you can see that ImportantReport_v23_FINAL_REALLY_FINAL.docx is the file that keeps failing, you can actually do something about it.

The best part? Once you have this extension method in your toolkit, it becomes muscle memory. You’re not adding complexity to your day-to-day development – you’re just swapping out one method call for a smarter one. It’s like upgrading from a flip phone to a smartphone – you wonder how you ever lived without it.

Pro tip: After implementing this, keep an eye on your application logs. You’ll start to notice patterns in failures that you never saw before. Maybe certain file types are more prone to issues, or maybe there’s a specific time of day when throttling gets worse. This kind of insight is pure gold for optimization.

The moral of the story? Sometimes the biggest performance improvements come not from doing things faster, but from doing fewer unnecessary things. And sometimes, the best debugging tool is just… not breaking the working stuff while you fix the broken stuff.

Your 2 AM debugging sessions will never be the same. 😌

Want to Learn More? (The Reading List)

• Microsoft Graph JSON batching - The official documentation (surprisingly readable!)
• Graph Batching for File Content: Mapping Requests to Responses - My previous post that sets up the foundation for this one
• Microsoft Graph throttling guidance - Understanding what makes Microsoft’s APIs cranky
• Graph Explorer - Test your batch requests interactively (great for experimenting)
• Exponential Backoff Pattern - The polite way to retry things

Now go forth and batch smarter, not harder! 🚀

If you’ve ever optimized your C# API calls with parallel programming, you might know this story.

Your client wants faster performance. You run multiple API requests in parallel. Locally, everything flies — especially late at night when traffic is low. But the next morning, on the final pre-deployment test, disaster strikes.

Random errors. Requests delayed for 45 seconds.
Microsoft Graph or SharePoint has slammed you with throttling and rate limits. Your blazing-fast local solution crumbles under production-like conditions.

How CRUD Operations Can Secretly Burn Through Your Limits — and How to Catch Them with DevProxy

Here’s the thing:
SharePoint Online charges “Resource Units” (RUs) for every request you make. Think of RUs as an invisible currency — every API call you send deducts from your allowance. Run out too fast, and throttling kicks in.

And those “simple” operations? They’re not as cheap as you think.

From Microsoft’s RU table:

Source: Microsoft’s official SharePoint throttling documentation

Permission expansions are just one type of “expensive” call. Another common scenario that can impact RU consumption is making multiple separate queries instead of using CAML joins — something I explored in my CAML join post.
While joining multiple lists in a single query is actually more efficient than separate calls, poorly structured queries or retrieving unnecessarily large datasets can still consume RUs quickly if not optimized properly.

Why This Sneaks Up on You

When coding locally, it’s easy to run a handful of queries without noticing.
But in production, with real concurrency and volume, these calls pile up in seconds.

Imagine:

• 10 parallel updates (2 RUs each) = 20 RUs in one burst
• Add a few joins/expansions or large list queries, and you’re suddenly burning through limits 5x faster.

How DevProxy Can Show You the Pain Before Production

Here’s where DevProxy shines.
If you configure it to simulate throttling based on these RU-heavy calls, you’ll see the impact locally:

1. Enable throttling plugins in your devproxy.json (as shown earlier).
2. Point urlsToWatch at your SharePoint endpoints.
3. Run your CRUD-heavy code.

DevProxy will start throwing 429s (Too Many Requests) once your “fake” RU budget is exhausted — just like SharePoint would in production.

The beautiful part?
You can crank the limits down during testing to make expensive patterns obvious. Even a single large query or unbatched Update will light up your logs.

Example DevProxy log when hitting a throttling simulation:

[Warning] Throttling triggered: 2 parallel Create calls exceeded RU limit (RateLimitingPlugin)
Retry after: 30 seconds

Pro tip:
Use DevProxy as a budget meter for your API calls. Treat every 2-RU and 5-RU operation as “spending big” — and redesign those spots before they become a production outage.

What is DevProxy?

DevProxy is an open-source HTTP/HTTPS proxy server that can simulate real-world network issues, including:

• Rate limiting
• Throttling
• Network delays
• Intermittent errors

It’s free, open source, and designed for developers who want to catch performance bottlenecks before they reach production.

Installing DevProxy

Follow the official setup guide here:
DevProxy Installation Documentation

Basic Usage

Run DevProxy with the default configuration:

devproxy

It will begin intercepting all HTTP/HTTPS requests.

Tip: Start DevProxy before running your own code — otherwise, it won’t capture the traffic.

Simulating Microsoft Graph Throttling

Here’s how to configure DevProxy to reproduce Microsoft Graph and SharePoint throttling issues locally.

1. Create a devproxy.json configuration file

This is the exact configuration I used when testing my problematic parallel code:

{
  "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v0.27.0/rc.schema.json",
  "rate": 25,
  "plugins": [
    {
      "name": "RetryAfterPlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll"
    },
    {
      "name": "RateLimitingPlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
      "configSection": "rateLimitingPlugin"
    },
    {
      "name": "GraphRandomErrorPlugin",
      "enabled": true,
      "pluginPath": "~appFolder/plugins/dev-proxy-plugins.dll",
      "configSection": "graphRandomErrorPlugin"
    }
  ],
  "urlsToWatch": [
    "https://graph.microsoft.com/v1.0/*",
    "https://graph.microsoft.com/beta/*",
    "https://graph.microsoft.us/v1.0/*",
    "https://graph.microsoft.us/beta/*",
    "https://dod-graph.microsoft.us/v1.0/*",
    "https://dod-graph.microsoft.us/beta/*",
    "https://microsoftgraph.chinacloudapi.cn/v1.0/*",
    "https://microsoftgraph.chinacloudapi.cn/beta/*",
    "https://*.sharepoint.*/*_api/web/GetClientSideComponents",
    "https://*.sharepoint.*/*_api/*",
    "https://*.sharepoint.*/*_vti_bin/*",
    "https://*.sharepoint-df.*/*_api/*",
    "https://*.sharepoint-df.*/*_vti_bin/*"
  ],
  "graphRandomErrorPlugin": {
    "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v0.27.0/graphrandomerrorplugin.schema.json",
    "allowedErrors": [
      429,
      503,
      504
    ]
  },
  "rateLimitingPlugin": {
    "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v0.27.0/ratelimitingplugin.schema.json",
    "costPerRequest": 2,
    "rateLimit": 120
  },
  "logLevel": "information",
  "newVersionNotification": "stable",
  "showSkipMessages": true,
  "showTimestamps": true
}

2. Start DevProxy with your configuration

devproxy --config-file devproxy.json

Example of Problematic Code

Here’s the snippet that caused my throttling nightmare. It uses Parallel.ForEachAsync to query SharePoint in bulk:

// ❌ This will likely trigger throttling in production
var customers = new ConcurrentBag<CustomerDTO>();

await Parallel.ForEachAsync(departmentIds, async (departmentId, token) => 
{
    var clonedContext = _clientContext.Clone(_clientContext.Url);

    var query = Camlex.Query()
        .ViewFields(new CustomerDTO().ViewFields().ToArray().Append("DepartmentId"))
        .LeftJoin(x => x["DepartmentLookup"].ForeignList(DEPARTMENT_LIST_GUID))
        .ProjectedField(x => x["DepartmentId"].List(DEPARTMENT_LIST_GUID).ShowField("ID"))
        .Where(x => x["DepartmentId"] == departmentId);

    var departmentCustomers = await SharePointService.GetItemsFromListByQuery<CustomerDTO>(
        CUSTOMER_LIST_GUID,
        clonedContext,
        query);

    if (departmentCustomers?.Any() == true)
    {
        foreach (var customer in departmentCustomers)
        {
            customers.Add(customer);
        }
    }
});

Why This Code Fails in Production

1. No throttling safeguards — Multiple parallel requests overwhelm SharePoint.
2. No retry logic — Requests fail instead of recovering.
3. No request limiting — All departments are processed simultaneously.
4. Silent failures — Errors are ignored without logging or fallback.

The Better Way

Instead of hard-coding fixes here, I recommend Bert Jansen’s detailed guide on throttling and rate limit handling:
Throttling & Rate Limit Handling Patterns

Key principles:

• Limit concurrency with SemaphoreSlim.
• Use exponential backoff for retries.
• Monitor rate limit headers to adjust requests dynamically.
• Handle errors explicitly to prevent silent failures.

Conclusion

DevProxy has become an essential tool in my Microsoft 365 development workflow. It helps me:

• Catch throttling issues before production.
• Test error handling and retry logic locally.
• Deliver applications that can survive real-world API limits.

If I’d used DevProxy from the start, I could have avoided the last-minute throttling meltdown entirely.

💡 Pro tip: Make DevProxy part of your early development process, not your emergency toolkit.

Additional Resources

• DevProxy GitHub Repository
• Microsoft Graph Throttling Guidelines
• DevProxy Documentation