To deduplicate SMS webhooks, pick a stable key for each event, try to record it before you act, and skip the work if the key already exists. A Postgres unique constraint written in the same transaction as your side effects is the safest way to do it. Redis SET key value NX EX ttl is faster and good enough when the side effect is cheap to repeat and a rare double is acceptable.
Providers retry a webhook when your endpoint is slow, returns an error, or the response gets lost on the way back. That's why delivery is at least once, not exactly once. The same report can reach you twice, and a naive handler will send two follow-up texts, credit a customer twice or count a reply double. This post covers the dedupe step only. Ordering is a separate problem, covered in out-of-order SMS webhooks, and events that fail for good are handled in dead-letter queues and replay.
Why duplicates happen
You can't prevent them on the sender's side, so plan for all of these:
- Your endpoint took longer than the provider's timeout, so it retried while your first run was still going.
- Your endpoint returned a 5xx during a deploy, then recovered.
- Your 2xx response was lost in transit, so the provider thinks it failed.
- Someone replayed events from a dead-letter queue or a provider dashboard on purpose.
The second and fourth cases mean duplicates can arrive minutes or days after the original. Your dedupe window has to cover that.
Choose a dedupe key
The key decides what counts as the same event. Use the first of these your provider gives you:
| Option | Strength | Weakness |
|---|---|---|
| A unique event or delivery ID from the provider | Exact, one key per event | Only exists if the provider sends one |
| Message ID plus status plus provider timestamp | Works without an event ID | Two different events with identical fields collapse into one |
| Hash of the raw request body | Needs no field knowledge | A retry with a changed body, such as a new attempt counter, looks new |
Read your provider's webhook documentation for what it guarantees. For Ready, the API quickstart is the source for the payload fields, and we don't guess them here. The code below uses generic names, eventId, messageId, status and occurredAt, as stand-ins for whatever your provider documents.
Scope the key by tenant if you receive webhooks for several sub-accounts on one endpoint. A key like acct_42:evt_9f3 stops one customer's event from ever colliding with another's, and it keeps the cross-tenant boundary obvious in your data.
Option 1: a Postgres unique constraint
The strongest approach puts the dedupe record and your side effects in one database transaction. Either both commit or neither does, so a crash can't leave you with a claimed key and no work done.
CREATE TABLE webhook_receipts (
dedupe_key text PRIMARY KEY,
received_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX webhook_receipts_received_at ON webhook_receipts (received_at); import { Pool } from "pg";
const pool = new Pool();
interface Event {
eventId: string; // placeholder: use your provider's documented field
messageId: string; // placeholder
status: string; // placeholder
occurredAt: string; // placeholder
}
export async function handleOnce(accountId: string, ev: Event): Promise<boolean> {
const key = `${accountId}:${ev.eventId}`;
const client = await pool.connect();
try {
await client.query("BEGIN");
const claim = await client.query(
"INSERT INTO webhook_receipts (dedupe_key) VALUES ($1) ON CONFLICT DO NOTHING",
[key],
);
if (claim.rowCount === 0) {
await client.query("ROLLBACK");
return false; // duplicate: acknowledge and do nothing
}
await applyStatusChange(client, accountId, ev); // your real work, same transaction
await client.query("COMMIT");
return true;
} catch (err) {
await client.query("ROLLBACK");
throw err; // return 5xx so the provider retries
} finally {
client.release();
}
} ON CONFLICT DO NOTHING is what makes this safe under concurrency. If two workers get the same event at the same moment, Postgres lets exactly one insert succeed. The second blocks until the first commits, then sees the conflict and returns zero rows.
The costs are modest. You pay one extra indexed insert per event, and you need a cleanup job. Delete rows older than your window with a batched DELETE ... WHERE received_at < now() - interval '14 days', or partition the table by day and drop old partitions, which is cheaper at scale.
Option 2: Redis SETNX
Redis gives you an atomic claim with an expiry in one command. It's quick, it keeps load off your main database, and it expires keys for you.
import Redis from "ioredis";
const redis = new Redis(process.env.REDIS_URL!);
const DEDUPE_TTL_SECONDS = 7 * 24 * 60 * 60; // assumption: one week
export async function claim(accountId: string, eventId: string): Promise<boolean> {
const key = `whk:${accountId}:${eventId}`;
const ok = await redis.set(key, "1", "EX", DEDUPE_TTL_SECONDS, "NX");
return ok === "OK"; // null means the key already existed
}
export async function release(accountId: string, eventId: string) {
await redis.del(`whk:${accountId}:${eventId}`);
} Use the single-command form with NX and EX together. Writing SETNX and then EXPIRE as two calls leaves a gap: if the process dies between them, the key lives forever and the event is blocked for good.
The weak point is the crash window. If you claim the key, then crash before the work finishes, the retry arrives, sees the key, and skips an event you never handled. There are two ways to handle that:
- Claim, do the work, and call
releasein acatchblock so a failed attempt can be retried. This covers exceptions but not a killed process. - Store a state in the value, such as
processingwith a short expiry, then overwrite it withdoneand the long expiry after the work succeeds. A retry that seesprocessingthat has expired can run again.
Either way, the work itself should be safe to repeat, so a rare double run does no harm.
Which should you use
| Question | Postgres unique constraint | Redis SET NX EX |
|---|---|---|
| Can it share a transaction with your side effects? | Yes | No |
| Survives a cache flush or failover? | Yes | Not guaranteed |
| Speed at high event volume | Fine for most; one indexed insert | Faster |
| Expiry | You build a cleanup job | Built in |
| Best when | Duplicate work costs money or sends a text | Duplicate work is cheap and idempotent |
For a platform that sends texts or bills customers in response to events, start with Postgres. Add Redis in front only if the insert rate becomes a measured problem, and keep the Postgres constraint as the final guard. A Redis check in front catches most repeats cheaply, and the database handles the ones that slip through.
Pick a TTL on purpose
A dedupe record has to outlive the longest plausible gap between a delivery and its retry or replay. These are assumptions to adjust:
- Provider automatic retries usually span minutes to hours, so a few hours is the floor.
- Manual replays from a dead-letter queue can come days later, so 7 to 14 days is a sensible default.
- Past the TTL, a replay gets treated as new. If you replay older events, your handler's own transition rules, such as those in a status state machine, should make that harmless.
Longer windows cost storage. A table with one short text key per event is small, but at millions of events a day it adds up, so measure it before choosing a window.
Make the side effects safe too
Dedupe reduces repeats. It doesn't remove the need for idempotent work. If your handler sends a follow-up text, send it with an Idempotency-Key header derived from the event key, so even a double run can't produce two messages. Our idempotency guide explains the header, and it's a good habit to combine both layers.
Also respond fast. Store the raw payload, return a 2xx, and process in the background. A short response time is the best way to avoid retries at all, and it's why signature checks should happen before anything else in the handler.
The short version
Choose a key from the provider's event ID if there is one, scoped by account. Claim it before you act, using a unique constraint in the same transaction as your work, or Redis SET NX EX when repeats are cheap. Keep records for days, not minutes, and make the follow-up actions idempotent anyway. Duplicates stop being an incident and become a log line you can count.