Prevent Duplicate Webhook Processing with Redis and Express

Your webhook endpoint just got hit three times in the same second with the exact same Stripe charge ID, and now your database has three identical orders. Today we are building an Express middleware backed by Redis to catch and drop these duplicate webhooks before they touch your database.

The Problem with Double-Firing Webhooks

Most webhook providers guarantee at least-once delivery, not exactly-once delivery. If a network hiccup occurs or your server takes longer than two seconds to respond, the sender assumes the message was lost and fires it again. This is especially common with platforms like Stripe, Shopify, or GoHighLevel.

If you are running a heavy asynchronous process inside your webhook handler, the second request will often arrive before the first one finishes. This creates a race condition. If you do not have a way to identify and drop these duplicate webhooks, you end up with double-billed customers, duplicate database rows, and messy support tickets.

Before writing the deduplication layer, make sure you are already handling basic security. You should verify webhook signatures with HMAC in Node.js and validate webhook payloads with Zod in Express to ensure you are processing clean, authentic data.

Setting Up the Sandbox

Let’s set up a clean project to test this. You will need Node.js installed and a running Redis instance. If you do not have Redis running locally, you can spin one up instantly using Docker.

Run this command in your terminal to start a local Redis container:

docker run -d -p 6379:6379 --name webhook-redis redis:alpine

Next, initialize a new Node.js project and install the required dependencies. We will use Express for our server and the official Node Redis client to talk to our database.

mkdir webhook-dedup
cd webhook-dedup
npm init -y
npm install express redis dotenv crypto

Designing the Idempotency Key Strategy

To prevent duplicate webhooks, we need a unique identifier for each incoming event. This is called an idempotency key. We have two main ways to get this key:

• Provider Headers: Many platforms send a unique event ID in the request headers (for example, Stripe-Signature or X-GoHighLevel-Event-Id).
• Payload Hashing: If the provider does not send a unique ID, we can generate a SHA-256 hash of the incoming JSON body. If two payloads are identical, their hashes will match perfectly.

Our middleware will look for a specific header first. If that header is missing, it will fall back to hashing the entire request body. This makes the system work with any webhook sender without changing the core logic.

Building the Redis Middleware

Let’s write the Express middleware. We will use the Redis SET command with the NX (Not Exists) option. This is an atomic operation: Redis will only set the key if it does not already exist in the database.

Create a file named middleware.js and add the following code:

const crypto = require('crypto');
const { createClient } = require('redis'); const redisClient = createClient({ url: process.env.REDIS_URL || 'redis://localhost:6379'
}); redisClient.on('error', (err) => console.error('Redis Client Error', err)); (async () => { await redisClient.connect();
})(); function webhookDeduplicator(options = {}) { const ttl = options.ttl || 300; // Default to 5 minutes const headerName = options.headerName || 'x-idempotency-key'; return async (req, res, next) => { let lockKey = ''; if (req.headers[headerName]) { lockKey = `webhook:key:${req.headers[headerName]}`; } else { const payloadString = JSON.stringify(req.body); if (!payloadString || payloadString === '{}') { return res.status(400).json({ error: 'Empty payload cannot be deduplicated' }); } const hash = crypto.createHash('sha256').update(payloadString).digest('hex'); lockKey = `webhook:hash:${hash}`; } try { const acquired = await redisClient.set(lockKey, 'processing', { NX: true, EX: ttl }); if (!acquired) { console.warn(`Duplicate webhook blocked: ${lockKey}`); return res.status(409).json({ error: 'Conflict: This webhook is already being processed or has been processed recently.' }); } req.webhookLockKey = lockKey; res.on('finish', async () => { if (res.statusCode >= 400) { await redisClient.del(lockKey); } else { await redisClient.set(lockKey, 'completed', { EX: ttl }); } }); next(); } catch (error) { console.error('Deduplication middleware error:', error); next(); } };
} module.exports = { webhookDeduplicator, redisClient };

This middleware does a few important things. It attempts to set the key in Redis with a status of “processing” and an expiration time (TTL). If the key already exists, Redis returns null, and we immediately send a 409 Conflict response back to the sender.

We also listen to the response finish event. If your server crashes or returns a 4xx/5xx error, we delete the key so the sender can safely retry the webhook. If the request succeeds, we update the status to “completed” and keep the key locked for the duration of the TTL to prevent late-arriving duplicates.

Handling Concurrent Race Conditions

Using a simple SET NX works for most basic workloads. However, if you are building high-volume event architectures, you might need a more advanced lock. If your webhook handler takes 10 seconds to run and the lock expires after 5 seconds, a second request could slip through while the first is still running.

To solve this for complex setups, you can implement a full distributed lock pattern. For details on managing complex locks in Node, check out our guide on using a Redis distributed lock in Node.js to prevent race conditions across multiple server instances.

Setting Up the Express Server

Now let’s build a simple Express server to use our new middleware. Create a file named server.js and add the following code:

const express = require('express');
const { webhookDeduplicator } = require('./middleware'); const app = express();
app.use(express.json()); app.post('/webhook', webhookDeduplicator({ ttl: 60 }), async (req, res) => { console.log('Processing webhook payload:', req.body); await new Promise((resolve) => setTimeout(resolve, 3000)); console.log('Webhook processing finished successfully.'); res.status(200).json({ status: 'success' });
}); const PORT = process.env.PORT || 3000;
app.listen(PORT, () => { console.log(`Server running on port ${PORT}`);
});

We added a artificial delay of 3 seconds using setTimeout inside our route handler. This simulates a slow operation like updating a database or calling a third-party API, making it easy to test our deduplication layer in action.

Testing the Setup with Curl

Start your server by running this command in your terminal:

node server.js

Now open a second terminal window and send two identical requests immediately after each other. We will use curl to send a JSON payload with an explicit idempotency key header.

curl -X POST http://localhost:3000/webhook  -H "Content-Type: application/json"  -H "x-idempotency-key: unique-event-123"  -d '{"order_id": 999, "amount": 49.99}' & curl -X POST http://localhost:3000/webhook  -H "Content-Type: application/json"  -H "x-idempotency-key: unique-event-123"  -d '{"order_id": 999, "amount": 49.99}' &

The & at the end of the curl commands tells your terminal to run both requests in the background simultaneously. In your server logs, you will see the first request begin processing, and the second request get blocked instantly:

Server running on port 3000
Processing webhook payload: { order_id: 999, amount: 49.99 }
Duplicate webhook blocked: webhook:key:unique-event-123
Webhook processing finished successfully.

The first request returned a 200 OK after 3 seconds, while the second request returned a 409 Conflict immediately. This saved your server from running the heavy database logic twice.

Common Edge Cases and Gotchas

While this system works well, there are a few edge cases you should keep in mind when deploying this to production:

• Choosing the Right TTL: If your TTL is too short (e.g., 5 seconds), a delayed retry from a provider might hit your server after the key has expired, causing a duplicate run. A TTL of 1 to 24 hours is standard for payment events.
• Handling Server Crashes: If your Node process crashes mid-execution, the key will remain set to “processing” until the TTL expires. This prevents retries from going through during that window. If you want to allow instant retries on crash, you must catch process termination signals and clear active keys.
• Payload Ordering: If your webhook provider does not guarantee order, a “user.updated” event might get processed before a “user.created” event. Consider pairing this deduplication layer with an event queue.

For high-volume webhooks, you might want to offload the processing entirely to a background worker system. You can read about how to queue GoHighLevel webhooks with Node.js and BullMQ to handle heavy workloads asynchronously without blocking your Express server.

Frequently Asked Questions

What HTTP status code should I return for a duplicate webhook?

We recommend returning a 409 Conflict. This clearly indicates to the sending service that the request was received but rejected as a duplicate. Some developers prefer returning a 200 OK to prevent the sending service from retrying again, but a 409 is semantically correct and helps with debugging logs.

Can I use this middleware with multiple Express instances?

Yes, because Redis is a centralized, in-memory database. As long as all your Express instances connect to the same Redis cluster, the deduplication will work perfectly across your entire server pool.

What happens if Redis goes down?

In our middleware code, the try/catch block ensures that if Redis fails, we log the error and call next(). This allows the webhook to process even if Redis is offline. It is better to risk processing a duplicate than to drop legitimate webhooks completely.

Next Steps

Now that you have built a reliable deduplication layer, you should look into building a robust retry handler for outbound webhooks. Learn how to build a webhook retry system with exponential backoff in Node.js to ensure your outgoing events always reach their destination safely.