Webhook capacity planning

Webhook throughput is a two-sided problem. Fairmeter dispatches audit events to your destination, signs each one, retries the ones that fail, and dead-letters the ones that keep failing. Your receiver has to accept them fast enough that the queue never grows. This page covers what the gateway guarantees on its side, so you can size your side against real numbers instead of guesses.

What the gateway does for you

Delivery concurrency, connection reuse, and queue draining are managed by Fairmeter. There is nothing to tune in your dashboard and no knob to set: deliveries are dispatched from a bounded worker pool on our side, and they never run on the request hot path, so a slow or dead receiver cannot slow down the LLM calls that produced the events.

What you should plan against is the delivery envelope. There is no per-destination or per-org cap on delivery volume today, so your arrival rate is whatever your event volume produces and the whole burst reaches your receiver. The gateway waits up to 10 seconds for your receiver to respond before treating the attempt as failed, and retries on the backoff schedule documented in Webhooks, which is short: the whole ladder finishes inside about three minutes. Payloads are metadata-sized, a few hundred bytes per event, so bandwidth is never the constraint. Your handler’s latency is.

Sizing your receiver

Start from your own event volume rather than the caps. Read the deliveries view under Settings, Webhooks, “View deliveries” during a busy window, or count the matching rows in /audit over a known interval, and divide by the interval to get a deliveries-per-second figure. Multiply by your peak-to-average ratio, where 2x is a reasonable default, to get a design rate.

The number that actually matters next is your receiver’s p95 handling latency, because concurrency at steady state is throughput multiplied by latency:

in_flight_at_steady_state = deliveries_per_second * p95_seconds

Use p95 rather than the mean, because you are sizing for worst-case occupancy, not average occupancy. A receiver taking 2 seconds at p95 under a design rate of 50 deliveries per second needs to handle roughly 100 concurrent requests. The same design rate against a receiver that answers in 100 ms only needs 5. If your handler writes to a database or calls a downstream API synchronously, that concurrency figure is also the connection-pool size you need on your side, and it is usually where capacity planning actually goes wrong.

The cheapest fix, when the arithmetic gets uncomfortable, is to stop doing work inline. Acknowledge the delivery with a 2xx as soon as you have verified the signature and durably written the event, then process it asynchronously. That collapses p95 to a few milliseconds and makes the concurrency question disappear.

Worked example

Take 1000 events per minute, one receiver, p95 of 500 ms. The arrival rate is about 17 deliveries per second sustained, and a 2x peak allowance puts the design point at 34 per second. Required in-flight capacity is 34 * 0.5, so 17 concurrent requests, which any normal web service handles without tuning. Nothing on the gateway side smooths that arrival rate for you, so 34 per second is what your receiver has to be able to absorb.

If the same workload had a p95 of 3 seconds instead, required in-flight capacity rises to 102 concurrent requests. At that point the answer is not more sockets, it is moving the work off the request path as described above.

Failure modes

The signal that you are falling behind is a growing pending count in the deliveries view while the success count stays flat. Arrival rate has passed your dispatch rate. In practice this is nearly always a slow receiver rather than a volume problem, so measure your handler’s p95 before anything else.

A rising failed count means something different. Every 3xx and 4xx is terminal on the first attempt, because a receiver misconfiguration cannot resolve itself by retrying, so a burst of failures usually means a bad path, an expired certificate, or a signature-verification bug on your side. Failed and dead-lettered deliveries carry a “Retry” button in the deliveries view once you have fixed the cause.

Repeated 429 responses from your own receiver are the last common case, and they are the one worth watching hardest, because a 429 is treated as terminal rather than retried. If your service rate-limits below the arrival rate, those deliveries are lost until you replay them by hand. Raise your receiver’s own limit for the Fairmeter source, or absorb the burst by acknowledging fast and queueing internally, rather than rate-limiting the deliverer.

Verification

Two surfaces tell you where you stand. The deliveries view under Settings, Webhooks shows the last 100 deliveries per destination, with per-delivery status, response code, attempt count, and timing, which is enough to spot both a latency problem and a signature problem. The audit log at /audit carries the same events with a deliveries-attempted column, which is the right place to reconcile “the event happened” against “the event arrived”.

Use a test fire (Settings, Webhooks, row, “Test fire”) after any change on your side. It exercises the full signing and verification path and writes its result to the delivery log immediately, so you find a broken verifier on your own schedule rather than during an incident.