How to Handle Webhook Spikes with a Redis Buffer Queue in Node.js

Your Express server will crash when a third-party API dumps 10,000 webhook events on you in five seconds. Today we will build a Redis-backed buffer queue in Node.js to ingest webhooks instantly and process them at a safe, controlled rate.

I hit this exact wall last year when a partner platform sent us a massive wave of subscription sync events. Our server CPU spiked to 100%, database connections maxed out, and we started dropping requests. The solution was to decouple ingestion from processing using a queue.

The Webhook Meltdown Problem

When you process webhooks directly inside your HTTP route handler, you are begging for a system failure. Each incoming request forces your server to perform database queries, call external APIs, or run heavy business logic while keeping the HTTP connection open.

If Stripe, Shopify, or HubSpot decides to batch-retry failed events or sync thousands of records, your server gets hammered. You run out of database connections, your event loop blocks, and you start returning 504 Gateway Timeouts. The webhook provider sees these errors, thinks your server is down, and retries again—creating a self-inflicted DDoS attack.

A webhook buffer queue fixes this. Instead of processing the event immediately, your HTTP server does two things: validates the payload and pushes it into a Redis queue. Then, it immediately returns a 202 Accepted status code. A separate worker process pulls jobs from the queue and processes them at a speed your database can actually handle.

Setting Up the Redis Buffer Architecture

We need a fast, reliable queue system. For Node.js, BullMQ is the gold standard because it is built on top of Redis, supports automatic retries, handles concurrency limits, and is incredibly fast.

Make sure you have Redis running locally. If you do not have it installed, you can spin it up quickly using Docker:

docker run -d --name redis-buffer -p 6379:6379 redis:alpine

Next, initialize your Node.js project and install the required dependencies. We will use express for our HTTP server, bullmq for queue management, ioredis for connecting to Redis, and dotenv for managing environment variables.

npm init -y
npm install express bullmq ioredis dotenv
npm install --save-dev nodemon

Building the Webhook Receiver

First, we need to define our Redis connection and create our queue instance. Create a file named queue.js. This file will export our queue configuration so both our Express server and our background worker can use it.

// queue.js
const { Queue } = require('bullmq');
const IORedis = require('ioredis');
require('dotenv').config(); const redisConnection = new IORedis(process.env.REDIS_URL || 'redis://127.0.0.1:6379', { maxRetriesPerRequest: null,
}); const webhookQueue = new Queue('webhook-buffer', { connection: redisConnection, defaultJobOptions: { attempts: 3, backoff: { type: 'exponential', delay: 2000, }, removeOnComplete: true, removeOnFail: 1000, },
}); module.exports = { webhookQueue, redisConnection };

Now, let’s build the Express server in server.js. The server’s only job is to receive the webhook, optionally verify it, push the raw payload to our Redis queue, and respond to the sender as fast as possible.

Before putting this in production, you should always verify webhook signatures with HMAC to make sure malicious actors aren’t flooding your queue with garbage data.

// server.js
const express = require('express');
const { webhookQueue } = require('./queue');
require('dotenv').config(); const app = express();
app.use(express.json()); app.post('/api/webhook', async (req, res) => { const payload = req.body; const eventId = req.headers['x-event-id'] || `evt_${Date.now()}`; if (!payload || !payload.event) { return res.status(400).json({ error: 'Invalid payload' }); } try { // Push the webhook payload to our Redis buffer queue await webhookQueue.add(payload.event, { id: eventId, payload: payload, receivedAt: new Date().toISOString() }); // Respond immediately with 202 Accepted return res.status(202).json({ accepted: true, message: 'Webhook queued' }); } catch (error) { console.error('Failed to queue webhook:', error); return res.status(500).json({ error: 'Internal queue failure' }); }
}); const PORT = process.env.PORT || 3000;
app.listen(PORT, () => { console.log(`Webhook receiver running on port ${PORT}`);
});

Building the Worker to Process Webhooks

With our server queueing up jobs, we need a worker to drain the queue. The worker runs as a separate process. This separation is key: if the worker crashes under heavy database load, your Express server stays up and continues to safely buffer incoming webhooks.

Create a file named worker.js. We will simulate a slow database write or an API call inside the worker using a helper timeout function.

// worker.js
const { Worker } = require('bullmq');
const { redisConnection } = require('./queue'); const delay = (ms) => new Promise((resolve) => setTimeout(resolve, ms)); const worker = new Worker( 'webhook-buffer', async (job) => { console.log(`[Worker] Started processing job ${job.id} (Type: ${job.name})`); // Simulate slow database processing or external API integration await delay(1500); console.log(`[Worker] Successfully processed job ${job.id}`); return { success: true }; }, { connection: redisConnection, concurrency: 5, // Limit how many webhooks we process at the exact same time }
); worker.on('active', (job) => { console.log(`Job ${job.id} is now active`);
}); worker.on('failed', (job, err) => { console.error(`Job ${job.id} failed with error: ${err.message}`);
}); console.log('Worker listening for queued webhooks...');

Notice the concurrency: 5 setting in the worker options. This is where you control your resource usage. If your database can only handle 10 concurrent writes before slowing down, you set your worker concurrency to 5 or 8. Redis holds the rest of the webhooks in memory until a worker slot opens up.

Simulating a Traffic Spike

Let’s run a load test to see how this setup performs under stress. First, start your Express server and your worker in separate terminal windows.

Terminal 1 (Server):

node server.js

Terminal 2 (Worker):

node worker.js

To simulate a massive traffic surge, we will write a quick Node.js script that fires 100 webhook requests almost instantly. Create a file named load-test.js.

// load-test.js
const http = require('http'); const sendWebhook = (id) => { return new Promise((resolve) => { const data = JSON.stringify({ event: 'user.updated', data: { userId: `user_${id}`, email: `user_${id}@example.com`, updatedAt: new Date().toISOString() } }); const options = { hostname: 'localhost', port: 3000, path: '/api/webhook', method: 'POST', headers: { 'Content-Type': 'application/json', 'Content-Length': data.length, 'X-Event-Id': `evt_test_${id}` } }; const req = http.request(options, (res) => { resolve(res.statusCode); }); req.on('error', (e) => { console.error(`Request error: ${e.message}`); resolve(500); }); req.write(data); req.end(); });
}; const runTest = async () => { console.log('Starting load test: sending 100 webhooks as fast as possible...'); const start = Date.now(); const promises = []; for (let i = 1; i <= 100; i++) { promises.push(sendWebhook(i)); } const statuses = await Promise.all(promises); const duration = Date.now() - start; const successCount = statuses.filter(s => s === 202).length; console.log(`
--- Test Completed ---`); console.log(`Total Requests Sent: 100`); console.log(`Successful Ingestions (202): ${successCount}`); console.log(`Time taken to ingest: ${duration}ms`);
}; runTest();

Run the load test script in a third terminal window:

node load-test.js

When you run this script, you will see that all 100 requests complete in less than 150 milliseconds. Your Express server barely breaks a sweat because it is not processing the heavy logic—it is only appending jobs to Redis.

Meanwhile, if you look at your worker terminal, you will see it processing exactly 5 jobs at a time. The database is never overwhelmed, and your external APIs are not rate-limiting you.

If you want to test your local setup with real external webhooks, you can use Cloudflare Tunnels to expose your local port safely to the internet.

Handling Failures and Retries

What happens when the processing fails? If your database goes offline for 30 seconds, your worker will fail to execute the jobs. Without a queue, those webhook events are lost forever.

With our BullMQ configuration, we set up automatic exponential backoff retries. If a job fails, BullMQ will wait, then retry it up to 3 times before moving it to a failed state. You can read more about setting up a custom webhook retry system with exponential backoff to handle flaky third-party integrations.

To prevent processing the same webhook event twice during retries or network failures, make sure you implement a deduplication layer. You can use our guide on how to prevent duplicate webhook processing with Redis to keep your workers idempotent.

Monitoring Queue Health in Production

You cannot fly blind when running a queue in production. If your workers stop running, your Redis memory will slowly fill up until it crashes. You need to keep an eye on your queue size.

You can write a simple endpoint in your Express server to monitor your queue health:

// Add this to server.js
app.get('/api/queue-health', async (req, res) => { try { const [waiting, active, failed, completed] = await Promise.all([ webhookQueue.getWaitingCount(), webhookQueue.getActiveCount(), webhookQueue.getFailedCount(), webhookQueue.getCompletedCount(), ]); return res.json({ waiting, active, failed, completed, status: waiting > 1000 ? 'CONGESTED' : 'HEALTHY' }); } catch (error) { return res.status(500).json({ error: error.message }); }
});

For a visual interface, you can integrate Bull Board, which is an open-source dashboard that hooks directly into BullMQ and lets you inspect, retry, or delete jobs manually from a clean UI.

Frequently Asked Questions

Can I use raw Redis Lists instead of BullMQ?

Yes, you can use RPUSH and BLPOP commands in Redis to build a simple queue. However, you will have to write your own logic for retries, exponential backoff, concurrency limiting, and failure states. BullMQ handles all of these edge cases out of the box, saving you hundreds of lines of fragile code.

What happens if Redis crashes? Will I lose my webhooks?

If Redis is configured with default settings and crashes, you could lose in-memory data. To prevent this, enable AOF (Append Only File) persistence in your Redis configuration. This ensures every write operation is logged to disk, allowing Redis to recover your queue state after a reboot.

How do I handle webhooks that must be processed in order?

If order matters (e.g., you must process user.created before user.updated), standard concurrent queues can cause race conditions. You can handle this by setting your worker concurrency to 1, or by using Redis Streams to group events by a specific partition key like a user ID.

Next Steps for Your Webhook Pipeline

Now that you have built a resilient ingestion layer, you should secure your endpoints and add validation. Take a look at our tutorial on how to validate webhook payloads with Zod in Express to make sure you only queue well-formed data.

Official resources

• Node.js documentation