Exactly Once Webhook Delivery at Scale

Introduction: The Twenty-Line Feature That Never Stays Twenty Lines

Every webhook feature starts in the same planning meeting. Someone says the integration is trivial: when an order is paid, POST a JSON body to whatever URL the customer configured. Twenty lines of code, maybe thirty once you have wired up the HTTP client. The ticket gets a two-point estimate and the team moves on to the interesting work. Eighteen months later that same feature owns a chunk of the on-call rotation, has its own Grafana dashboard, and produces the support tickets nobody wants to pick up. Customers report events they never received. Other customers report the same event four times, and every one of those duplicates did real work on the other side. One customer’s endpoint has been returning 502 for three days and the retry backlog is now large enough to be its own capacity problem.

This post makes one argument: the hard part of webhooks was never the HTTP request. The hard part is that a webhook is a distributed transaction between two systems that do not share a database, do not share a clock, cannot see each other’s failures, and are operated by two organisations with different definitions of “up”. Everything that follows is a consequence of that. We will look at why exactly once webhook delivery is genuinely hard, what the traditional stack costs you, and why the same system becomes almost boring to build on Cloudflare.

Table of contents

Jump to a section

• Exactly Once Delivery Does Not Exist
• Why Webhook Delivery Is Surprisingly Hard
  • The Receiver Is Not Your Code
  • Ordering Is a Promise You Did Not Mean to Make
  • One Slow Endpoint Poisons Everyone Else’s
  • Retries Multiply Load Exactly When You Can Least Afford It
• The Naive Implementation and the Five Bugs Hiding In It
• Making the Intent Durable: The Outbox Pattern
• Deduplication Is a Consensus Problem in Disguise
• What the Traditional Stack Actually Costs
• The Cloudflare Version in One Picture
• Step One: Accept the Event at the Edge
• Step Two: Let Cloudflare Queues Own the Retry Ladder
• Step Three: One Durable Object per Subscriber
• Step Four: Sign the Request So the Receiver Can Trust It
• Long Retry Ladders with Cloudflare Workflows
• The Other Half of the Problem: Being a Good Receiver
• What It Costs and Where the Limits Are
• Failure Modes That Do Not Go Away
• Five Things Worth Remembering
• Final Thoughts

Exactly Once Delivery Does Not Exist

Let us clear this up before writing any code, because it reframes everything that follows. Exactly once delivery over an unreliable network is impossible. This is not a limitation of your framework or your message broker, it is the Two Generals Problem wearing a modern hat. When your sender fires a POST and the connection dies before the response comes back, the sender cannot distinguish between “the receiver never got it” and “the receiver processed it and the acknowledgement was lost”. Those two states look identical from the outside, and no amount of additional messages resolves the ambiguity, because every additional message has the same problem.

So the sender has exactly two options. It can retry, and risk a duplicate. Or it can not retry, and risk a loss. Retrying gives you at-least-once, not retrying gives you at-most-once, and there is no third door. Every serious webhook provider, Stripe and GitHub and Shopify included, chose at-least-once and said so in their documentation, usually in a sentence most integrators skim past.

What is achievable, and what people actually mean when they say “exactly once”, is exactly once processing: at-least-once delivery combined with a receiver that recognises a repeat and refuses to do the work twice. The guarantee moves from the wire into the application. Your job as a webhook provider is then to deliver every event at least once, in a sane order, without falling over, and to hand the receiver everything it needs to deduplicate cheaply. That is a far more tractable problem than the impossible one, and it is still harder than most teams expect.

Why Webhook Delivery Is Surprisingly Hard

The difficulty does not live in any single place. It is spread thinly across a dozen small decisions that each look reasonable in isolation and combine into a system nobody wants to own.

The Receiver Is Not Your Code

Everything you know about writing resilient services assumes you control both ends. With webhooks you control one end. The other end is a customer’s endpoint, and it will do things your integration tests never modelled. It will accept the connection and then never respond. It will return 200 with an HTML error page in the body. It will return 301 to a different host. It will have a TLS certificate that expired at 02:00 on a Sunday. It will be a laptop behind a tunnel that closes when someone shuts the lid. It will be an internal address like 169.254.169.254 that turns your sender into a server-side request forgery machine if you are not filtering egress. You cannot fix any of this, you can only survive it.

Ordering Is a Promise You Did Not Mean to Make

Nobody writes “events arrive in order” in the API docs, and every customer assumes it anyway. If subscription.created arrives after subscription.cancelled, the customer’s database now says a cancelled subscription is active, and the resulting support ticket will be about your bug, not theirs. The moment you add concurrency to your sender, which you must do to get throughput, you have introduced reordering. Two workers pull two events for the same customer, the first one hits a slow endpoint and the second one wins the race. Preserving order per subscriber while still processing thousands of subscribers in parallel is the central tension in the whole design.

One Slow Endpoint Poisons Everyone Else’s

This is the failure mode that turns a small incident into a large one. You have a worker pool draining a shared queue. One customer’s endpoint starts taking thirty seconds to respond instead of eighty milliseconds. Your workers block on that customer, the queue backs up, and every other customer’s events are late too. Head-of-line blocking has turned a single tenant’s outage into everyone’s outage. Fixing it properly means per-subscriber isolation, per-subscriber concurrency limits, and a circuit breaker that stops hammering an endpoint that is clearly down: three more subsystems nobody estimated.

Retries Multiply Load Exactly When You Can Least Afford It

Retries are the obvious answer to transient failure and a beautiful way to build a self-inflicted denial of service. A receiver that is struggling under load returns 503. Your sender retries. Now the struggling receiver gets more traffic than before it started struggling. If your retry schedule has no jitter, every one of your workers retries at the same instant and arrives as a synchronised thundering herd. Exponential backoff with full jitter is not a nice-to-have here, it is the difference between helping the receiver recover and holding it under water.

The Naive Implementation and the Five Bugs Hiding In It

Here is the twenty-line version from the planning meeting, roughly as it always gets written.

// The version that ships in sprint one.
async function onOrderPaid(order: Order) {
  await db.orders.markPaid(order.id);

  const subscription = await db.subscriptions.findByTenant(order.tenantId);
  await fetch(subscription.url, {
    method: "POST",
    headers: { "content-type": "application/json" },
    body: JSON.stringify({ type: "order.paid", order }),
  });
}

There are at least five defects in those lines. The first is a lost update: if the process dies between markPaid and fetch, the order is paid and the webhook will never be sent, because nothing durable ever recorded the intent to send it. The second is the ambiguity we discussed above: if fetch times out, this code has no idea whether the receiver saw the event, and no identifier that would let the receiver work it out either. The third is that the delivery happens on the request path, so a slow customer endpoint becomes a slow checkout for your own users. The fourth is ordering: two concurrent onOrderPaid calls for the same tenant can land in either order. The fifth is trust: the receiver has no way to verify this request came from you, which means anyone who learns the URL can forge paid orders.

Every one of those five defects is fixed by infrastructure, not by cleverness in the handler. That is the real lesson.

Making the Intent Durable: The Outbox Pattern

The first fix is the oldest one. You cannot atomically write to your database and publish to a broker, because they are two systems and there is no shared transaction. So you stop trying. You write the event into an outbox table inside the same transaction that changes your business state, and a separate process reads that table and publishes. If the transaction commits, the intent to send exists. If it rolls back, it does not. There is no window where the two disagree.

BEGIN;
  UPDATE orders SET status = 'paid' WHERE id = $1;
  INSERT INTO webhook_outbox (event_id, tenant_id, topic, payload, created_at)
  VALUES ($2, $3, 'order.paid', $4, now());
COMMIT;

Note the event_id. It is generated once, here, at the moment of truth, and it never changes for the lifetime of that event no matter how many delivery attempts follow. That single identifier is what makes downstream deduplication possible at all. Generate it in the retry loop instead and you have built a duplicate factory. I used the same pattern for image events in my Dapr and Lightstreamer write-up, and the reasoning is identical: the durable record comes first, the network call comes second.

Deduplication Is a Consensus Problem in Disguise

With a stable event ID, deduplication looks easy. Keep a set of IDs you have already delivered, check the set before sending. Then you scale to more than one worker and discover the problem. Two workers pick up the same retried event at the same moment. Both check the set, both see nothing, both send. Your “deduplicated” pipeline just delivered twice.

The reflex is to reach for Redis and SET NX. That helps, but it moves the race rather than removing it, because the check and the delivery are still not atomic. If you claim the lock, send, and then crash before recording success, the lock either expires and you send again, or it does not expire and the event is silently dropped forever. You are choosing between at-least-once and at-most-once all over again, one layer down.

What you actually need is a serialisation point: exactly one place in the system where, for a given subscriber, only one thing happens at a time, and where the record of what happened lives in the same failure domain as the decision to act. In a traditional stack you approximate this with partitioned log consumers, where a partition key pins all events for a subscriber to a single consumer. That works, and it is how I have built it on Azure Event Hubs for a stateful FIX engine, but it brings partition rebalancing, consumer group lag, hot partitions and a rigid ceiling on parallelism.

What the Traditional Stack Actually Costs

Sketch the full architecture on a whiteboard and the shape of the bill becomes obvious. Postgres holds the outbox. Kafka or Event Hubs carries the events, partitioned by subscriber so ordering survives. A worker deployment on Kubernetes drains the partitions, with an autoscaler that reacts to consumer lag and a disruption budget so deploys do not drop in-flight deliveries. Redis holds the idempotency ledger and the circuit breaker state, with a replica for failover. A scheduler handles delayed retries, because a broker that redelivers immediately is useless for a four-hour backoff. A dead letter topic collects the failures, plus an internal UI so support can replay them. Then per-tenant rate limiting, egress filtering to stop SSRF, secret storage for signing keys, and a metrics pipeline to prove your delivery SLA.

That is six stateful systems running around the clock for a workload that is bursty by nature, and it is why “we will just POST some JSON” turns into a quarter of platform work. What changed in the last few years is that most of those boxes now exist as managed primitives with the right semantics baked in, and the assembly job has become genuinely small.

The Cloudflare Version in One Picture

The design collapses to four moving parts, none of which you operate.

  producer            transport              serialisation point       receiver
 +----------+      +---------------+      +--------------------+      +----------+
 |  Worker  | ---> |    Queue      | ---> |  Durable Object    | ---> | customer |
 | (ingest) |      | retries, DLQ, |      | one per subscriber |      | endpoint |
 |          |      | delays, batch |      | dedup + order +    |      |          |
 +----------+      +---------------+      | circuit breaker    |      +----------+
                                          +--------------------+
                                                    |
                                          +--------------------+
                                          |     Workflow       |
                                          | multi-day retries  |
                                          +--------------------+

A Worker accepts the event and puts it on a queue. Cloudflare Queues owns durability, batching, retries with delays, and the dead letter queue. A Durable Object, addressed by subscriber ID, is the serialisation point: it is the single instance in the world responsible for that subscriber, it has transactional SQLite storage sitting right next to the code, and it can schedule its own future wake-ups. A Workflow handles retry ladders that stretch across days. There is no cluster, no broker to size, no Redis, and nothing to pay for while it sits idle.

Step One: Accept the Event at the Edge

The ingest Worker does almost nothing, which is the point. Its only job is to make the event durable and get out of the way.

export interface Env {
  EVENTS: Queue<WebhookEvent>;
  SUBSCRIBER: DurableObjectNamespace<Subscriber>;
}

export type WebhookEvent = {
  id: string;             // generated once, at the source of truth
  subscriptionId: string;
  topic: string;
  occurredAt: string;
  payload: unknown;
};

export default {
  async fetch(request: Request, env: Env): Promise<Response> {
    const event = await request.json<WebhookEvent>();
    await env.EVENTS.send(event, { contentType: "json" });
    return new Response(null, { status: 202 });
  },
};

Once send resolves, the event is durably stored and Cloudflare owns the problem of getting it delivered. Your business transaction can commit and your user gets their response in single-digit milliseconds regardless of how slow the customer’s endpoint is today.

Step Two: Let Cloudflare Queues Own the Retry Ladder

Queues gives you at-least-once delivery, which is the correct and honest guarantee, and it exposes the knobs that usually cost you a scheduler and a dead letter dashboard. The consumer configuration is a few lines of wrangler.jsonc.

{
  "queues": {
    "producers": [{ "queue": "webhook-events", "binding": "EVENTS" }],
    "consumers": [
      {
        "queue": "webhook-events",
        "max_batch_size": 25,        // default 10, up to 100
        "max_batch_timeout": 5,      // seconds, up to 60
        "max_retries": 8,            // default 3, up to 100
        "retry_delay": 30,           // baseline; we override per message
        "dead_letter_queue": "webhook-events-dlq"
      }
    ]
  }
}

The numbers worth knowing: a message can be up to 128 KB, a queue sustains around 5,000 messages per second, messages are retained for four days by default and up to fourteen if you ask, a single message can be retried up to 100 times, and a retry can be delayed by up to 24 hours. Push consumers scale out to 250 concurrent invocations. That covers the entire retry ladder that most teams build by hand.

In the consumer you acknowledge each message individually, so one bad delivery in a batch of twenty-five does not drag the other twenty-four back through the ladder with it.

export default {
  async queue(batch: MessageBatch<WebhookEvent>, env: Env): Promise<void> {
    await Promise.all(batch.messages.map(async (message) => {
      const event = message.body;
      const id = env.SUBSCRIBER.idFromName(event.subscriptionId);
      const subscriber = env.SUBSCRIBER.get(id);

      try {
        const result = await subscriber.deliver(event);
        if (result.status === "delivered" || result.status === "duplicate") {
          message.ack();
        } else {
          message.retry({ delaySeconds: backoff(message.attempts) });
        }
      } catch {
        message.retry({ delaySeconds: backoff(message.attempts) });
      }
    }));
  },
};

// Full jitter, capped at one hour. `attempts` starts at 1.
function backoff(attempts: number): number {
  const ceiling = Math.min(2 ** attempts * 5, 3600);
  return Math.floor(Math.random() * ceiling);
}

message.attempts is what makes the backoff possible without any state of your own, and the jitter is what stops all your retries arriving at a limping receiver in the same millisecond. After the eighth attempt the message lands in the dead letter queue, which is just another queue with another consumer, so alerting on it or replaying from it is ordinary code rather than a bespoke admin tool.

Step Three: One Durable Object per Subscriber

This is where the design earns its keep. idFromName(subscriptionId) gives you a globally unique, consistently routed instance for that subscriber. Every event for that subscriber, from any Worker in any data centre, reaches the same object. That object has SQLite storage attached, colocated with the code, so a read is a local function call rather than a network round trip.

There is one subtlety that decides whether this actually works, and it is the single most important detail in the whole design. A Durable Object is single-threaded, and its input gates stop other events interleaving while a storage operation is in flight. But awaiting an outbound fetch() opens the gate. While you are waiting on a customer’s endpoint, a second delivery for the same event can begin executing. So the dedup record has to be written before the request goes out, not after.

import { DurableObject } from "cloudflare:workers";

export class Subscriber extends DurableObject<Env> {
  constructor(ctx: DurableObjectState, env: Env) {
    super(ctx, env);
    this.ctx.storage.sql.exec(`
      CREATE TABLE IF NOT EXISTS deliveries (
        event_id  TEXT PRIMARY KEY,
        state     TEXT NOT NULL,          -- 'in_flight' | 'delivered'
        attempted INTEGER NOT NULL
      )`);
  }

  async deliver(event: WebhookEvent) {
    const seen = this.ctx.storage.sql
      .exec("SELECT state FROM deliveries WHERE event_id = ?", event.id)
      .toArray();

    // Already done, or already being sent by a concurrent attempt.
    if (seen.length > 0) return { status: "duplicate" as const };

    if (this.breakerIsOpen()) return { status: "deferred" as const };

    // Claim the event *before* yielding to the network.
    this.ctx.storage.sql.exec(
      "INSERT INTO deliveries (event_id, state, attempted) VALUES (?, 'in_flight', ?)",
      event.id, Date.now(),
    );

    const response = await this.post(event);

    if (response.ok) {
      this.ctx.storage.sql.exec(
        "UPDATE deliveries SET state = 'delivered' WHERE event_id = ?", event.id);
      await this.ctx.storage.put("consecutiveFailures", 0);
      return { status: "delivered" as const };
    }

    // Release the claim so the queue's retry can try again later.
    this.ctx.storage.sql.exec("DELETE FROM deliveries WHERE event_id = ?", event.id);
    await this.tripBreakerIfNeeded();
    return { status: "failed" as const };
  }
}

Order falls out of the same property almost for free. Because one object handles one subscriber and processes one delivery at a time, events for that subscriber cannot overtake each other. You get per-subscriber ordering without partition keys, without rebalancing, and without capping your total concurrency, since a million subscribers means a million independent objects.

The circuit breaker is the third thing you get in the same place. Count consecutive failures in storage, and once the count crosses a threshold, stop dialling and return deferred so the queue holds the backlog instead of your object. Then call setAlarm to wake up in a few minutes and probe the endpoint once. Alarms have guaranteed at-least-once execution and are retried with exponential backoff if the handler throws, so a half-open probe is a few lines rather than a cron service. Crucially, that breaker is scoped to one subscriber. A customer whose endpoint has been down since Friday cannot slow down anyone else, because nothing is shared.

Step Four: Sign the Request So the Receiver Can Trust It

Signing belongs to the sender and is cheap to get right with WebCrypto. Include a timestamp inside the signed payload so the receiver can reject replays, and use the stable event ID as the deduplication header.

async function signedHeaders(secret: string, eventId: string, body: string) {
  const timestamp = Math.floor(Date.now() / 1000);
  const key = await crypto.subtle.importKey(
    "raw", new TextEncoder().encode(secret),
    { name: "HMAC", hash: "SHA-256" }, false, ["sign"],
  );
  const mac = await crypto.subtle.sign(
    "HMAC", key, new TextEncoder().encode(`${timestamp}.${body}`),
  );
  const signature = [...new Uint8Array(mac)]
    .map((b) => b.toString(16).padStart(2, "0")).join("");

  return {
    "content-type": "application/json",
    "webhook-id": eventId,
    "webhook-timestamp": String(timestamp),
    "webhook-signature": `v1,${signature}`,
  };
}

Signing the timestamp together with the body is what makes replay protection possible: a receiver rejects anything outside a five-minute window, and an attacker who captures a valid request cannot re-use it tomorrow. Send the ID in a header as well as the body so a receiver can deduplicate before parsing anything.

Long Retry Ladders with Cloudflare Workflows

Queues handles retries over hours. If your product promises a three-day retry schedule, the way Stripe does, reach for Workflows instead of building a scheduler. A Workflow is durable execution: each step.do is retried and memoised independently, and step.sleep survives restarts, deploys and machine failures without holding anything open.

export class WebhookDelivery extends WorkflowEntrypoint<Env, WebhookEvent> {
  async run(event: WorkflowEvent<WebhookEvent>, step: WorkflowStep) {
    const schedule = ["10 seconds", "5 minutes", "1 hour", "6 hours", "1 day"];

    for (const [attempt, wait] of schedule.entries()) {
      const ok = await step.do(`attempt-${attempt}`, () => post(event.payload));
      if (ok) return;
      await step.sleep(`backoff-${attempt}`, wait);
    }

    await step.do("give-up", () => markUndeliverable(event.payload));
  }
}

A step that has already succeeded is never re-run, so a crash halfway through the ladder resumes at the right rung rather than starting over. That is precisely the property that makes long retry schedules safe.

The Other Half of the Problem: Being a Good Receiver

Most teams are on the receiving end at least as often as the sending end, and the receiver’s contract is short. Verify the signature. Reject anything with a stale timestamp. Deduplicate on the event ID with a uniqueness constraint rather than a read-then-write. Return 200 as fast as you can, and do the real work asynchronously, because your response time is the sender’s timeout budget.

app.MapPost("/webhooks/orders", async (HttpRequest req, AppDb db, IQueue queue) =>
{
    var body = await new StreamReader(req.Body).ReadToEndAsync();
    if (!SignatureVerifier.IsValid(req.Headers, body, TimeSpan.FromMinutes(5)))
        return Results.Unauthorized();

    var eventId = req.Headers["webhook-id"].ToString();

    // The unique index does the deduplication; no read-then-write race.
    try
    {
        db.ProcessedEvents.Add(new ProcessedEvent(eventId, DateTime.UtcNow));
        await db.SaveChangesAsync();
    }
    catch (DbUpdateException) when (db.IsUniqueViolation())
    {
        return Results.Ok();   // already handled, and that is a success
    }

    await queue.EnqueueAsync(body);
    return Results.Ok();
});

The catch block is the entire exactly once story on the receiving side. A duplicate is not an error, it is the expected consequence of a correct at-least-once sender, and answering 200 is the right thing to do. Let the database enforce uniqueness; it is better at concurrency than your if statement is.

What It Costs and Where the Limits Are

Queues bills per operation, where an operation is 64 KB written, read or deleted. A normal delivery is three operations, and each retry adds a read. At $0.40 per million operations with a million operations included on the paid plan, a service pushing a million webhooks a day sits in the tens of dollars per month, and that number scales down to nothing on a quiet weekend. Compare that to the always-on cost of a broker, a Redis pair and a worker pool sized for peak.

The limits worth designing around: 128 KB per message, so send a reference rather than a 2 MB payload. Around 5,000 messages per second per queue, so shard queues by topic if you are genuinely above that. A soft limit of roughly 1,000 requests per second to any single Durable Object, which is a real constraint only if one subscriber alone is that busy. And 10 GB of SQLite storage per object, which means the delivery ledger needs a retention policy: keep event IDs for long enough to cover your maximum retry window plus a comfortable margin, then prune them in the alarm handler. Nothing here is exotic, but all of it is worth knowing before the traffic arrives rather than after.

Failure Modes That Do Not Go Away

Good infrastructure removes whole classes of bugs, and it is worth being honest about the ones that survive. A receiver that returns 200 and then loses the event is indistinguishable from a receiver that succeeded, so publish a replay API and let customers pull what they missed. Clock skew breaks signature windows, so make the tolerance generous enough to survive a badly synchronised server. Customers change their endpoint URL while events are in flight, so decide deliberately whether in-flight deliveries follow the change. Payloads outlive their contents, which matters when someone exercises a deletion request and a copy is sitting in a dead letter queue. And a dead letter queue nobody looks at is just a slower way to lose data, so alert on its depth on day one, not after the first incident.

Five Things Worth Remembering

If you take nothing else from this, take these. Exactly once delivery is impossible, so stop designing for it and design for at-least-once delivery with idempotent processing instead. Generate the event ID once, in the same transaction that changes your business state, and never regenerate it. Give every subscriber its own serialisation point, because that single decision buys you ordering, deduplication and blast-radius isolation at the same time. Make retries exponential and jittered, and give them a dead end that a human actually watches. And do not build the six stateful systems that used to be required, because Queues, Durable Objects and Workflows now cover the boring parts with the right semantics.

Final Thoughts

Webhooks are a good example of a problem that looks like application code and is really a distributed systems problem in disguise. The naive version is genuinely twenty lines, and it is genuinely wrong, and no amount of care inside those twenty lines fixes it, because the defects live in the gaps between the systems rather than inside any one of them. What changed is not the theory, which has not moved since the Two Generals Problem was described. What changed is that the primitives you need to respond to that theory, durable queues with real retry semantics, single-threaded stateful objects with transactional storage, and durable execution for long-running schedules, are now things you configure instead of things you operate. That is what makes the modern version almost boring to build, and boring is exactly what you want from the system your customers rely on to tell them what just happened.