How to Build a Delayed Job Queue with Redis and Node.js

I needed a way to trigger webhook retries on a delay without spinning up a massive, heavy queue framework. Polling a PostgreSQL database every few seconds with a cron job felt dirty, and it quickly dragged down database performance when traffic spiked. I wanted something fast, in-memory, and simple enough to write in a couple of files.

Redis is perfect for this. By using a specific data structure called a Sorted Set, we can schedule jobs to run at an exact millisecond in the future. Here is how I built a working delayed job queue from scratch using Node.js, the official ioredis client, and an atomic Lua script to prevent duplicate processing when scaling to multiple worker processes.

The Strategy: Redis Sorted Sets

To build a delayed queue, we need a way to store tasks sorted by their execution time. Standard Redis lists (LPUSH/RPOP) don’t work here because they only let us push and pop from the ends. Instead, we use Redis Sorted Sets documentation (ZSETs).

In a Sorted Set, every element is associated with a numeric score. Redis automatically keeps the elements ordered by this score. For our delayed queue:

• The value is the job payload (serialized as a JSON string).
• The score is the absolute Unix timestamp (in milliseconds) when the job should execute.

To find jobs that are ready to run, we query the set for any items with a score between 0 and the current timestamp. If a job’s score is less than or equal to Date.now(), its delay has expired, and it is ready for processing.

Setting Up the Project

Let’s get the boring setup out of the way. We only need two dependencies: ioredis to talk to our Redis instance, and uuid to give each job a unique identifier so Redis doesn’t overwrite duplicate payloads.

Initialize your project and install the packages:

mkdir redis-delayed-queue
cd redis-delayed-queue
npm init -y
npm install ioredis uuid

Make sure you have a Redis instance running locally. If you use Docker, you can spin one up with a single command:

docker run -d -p 6379:6379 --name local-redis redis:7-alpine

Building the Queue Producer

The producer’s job is simple: take a payload, calculate when it should run, and push it into the Sorted Set. Create a file named queue.js to hold our core queue logic.

const Redis = require('ioredis');
const { v4: uuidv4 } = require('uuid'); const redis = new Redis({ host: '127.0.0.1', port: 6379
}); const QUEUE_KEY = 'delayed_jobs'; async function addJob(payload, delayMs) { const jobId = uuidv4(); const job = { id: jobId, payload, createdAt: Date.now(), runAt: Date.now() + delayMs }; const score = job.runAt; const member = JSON.stringify(job); await redis.zadd(QUEUE_KEY, score, member); console.log(`[Producer] Scheduled job ${jobId} to run in ${delayMs}ms`); return jobId;
} module.exports = { addJob, redis, QUEUE_KEY };

Notice how we stringify the payload and inject a unique ID. Redis Sorted Sets require unique members. If you push two identical payloads, Redis will just update the timestamp of the first one instead of creating a second job. The UUID prevents this.

The Naive Consumer (And Why It Breaks in Production)

Now we need a worker to poll Redis. Let’s start with the obvious, naive approach so we can see exactly why it breaks under real-world pressure.

A simple worker might fetch expired jobs with zrangebyscore, loop through them, and then delete them using zrem.

// Do not use this in production!
async function pollJobsNaive() { const now = Date.now(); // Get jobs where score is between 0 and now const jobs = await redis.zrangebyscore(QUEUE_KEY, 0, now, 'LIMIT', 0, 1); if (jobs.length === 0) { return; } const job = JSON.parse(jobs[0]); // Delete the job first so no other worker grabs it const deleted = await redis.zrem(QUEUE_KEY, jobs[0]); if (deleted > 0) { console.log(`[Worker] Processing job: ${job.id}`); // Process job logic here }
}

This looks fine on your local machine, but it is a disaster in production. If you scale up to three worker processes, they will all call zrangebyscore at the exact same millisecond. They will all fetch the same job, and all three will process it. Only one will successfully delete it via zrem, but the damage is done—you just ran the same job three times.

We hit a similar concurrency bottleneck when writing about how to debounce webhook events with Redis. The fix is the same: we must make the fetch-and-delete operation atomic.

Fixing the Race Condition with Lua

To make this atomic, we use a Lua script. Redis runs Lua scripts in a single thread, meaning no other command can run while our script is executing. This guarantees that if a worker grabs a job, it deletes it from the queue before any other worker can even look at it.

Let’s write the script and register it with our ioredis client on GitHub.

const fetchJobLua = ` local queue = KEYS[1] local maxScore = ARGV[1] -- Get the first job that is ready local jobs = redis.call('zrangebyscore', queue, 0, maxScore, 'LIMIT', 0, 1) if #jobs == 0 then return nil end local job = jobs[1] -- Remove it immediately so no other worker can grab it redis.call('zrem', queue, job) return job
`; // Register the custom command with ioredis
redis.defineCommand('fetchDelayedJob', { numberOfKeys: 1, lua: fetchJobLua
}); async function pollJobAtomic() { const now = Date.now(); // Execute our atomic Lua script const jobString = await redis.fetchDelayedJob(QUEUE_KEY, now); if (!jobString) { return null; } return JSON.parse(jobString);
}

By wrapping zrangebyscore and zrem in Lua, they run as a single transaction. If a worker gets a job back, it has already been safely deleted from the queue. No duplicate processing can occur, no matter how many workers you spin up.

Building the Worker Loop

Next, we need a solid polling loop. We don’t want to spam Redis with thousands of queries a second when the queue is empty. Instead, we will use a dynamic sleep: if we find a job, we process it and immediately check for the next one. If the queue is empty, we back off and sleep for a bit.

Create worker.js and add the following code:

const { redis, QUEUE_KEY } = require('./queue'); const fetchJobLua = ` local queue = KEYS[1] local maxScore = ARGV[1] local jobs = redis.call('zrangebyscore', queue, 0, maxScore, 'LIMIT', 0, 1) if #jobs == 0 then return nil end local job = jobs[1] redis.call('zrem', queue, job) return job
`; redis.defineCommand('fetchDelayedJob', { numberOfKeys: 1, lua: fetchJobLua
}); function sleep(ms) { return new Promise(resolve => setTimeout(resolve, ms));
} async function startWorker() { console.log('[Worker] Started polling for delayed jobs...'); while (true) { try { const now = Date.now(); const job = await redis.fetchDelayedJob(QUEUE_KEY, now); if (job) { console.log(`[Worker] [${new Date().toISOString()}] Processing job ${job.id} (Scheduled for: ${new Date(job.runAt).toISOString()})`); // Simulate work await sleep(500); console.log(`[Worker] Finished job ${job.id}`); // Check for next job immediately continue; } } catch (error) { console.error('[Worker] Error processing job:', error); } // No jobs found, sleep for 1 second before checking again await sleep(1000); }
} startWorker();

This loop keeps CPU and Redis usage incredibly low when idle, but reacts instantly when there is work to do. If you need more throughput, just run multiple instances of this worker.js script using PM2.

Testing It Out

Let’s write a quick test script to schedule several jobs with different delays to verify everything is processed in the correct order. Create test.js:

const { addJob, redis } = require('./queue'); async function runTest() { console.log('Scheduling test jobs...'); // Schedule jobs with varying delays await addJob({ email: 'user1@example.com', type: 'welcome' }, 5000); // 5 seconds await addJob({ email: 'user2@example.com', type: 'feedback' }, 2000); // 2 seconds await addJob({ email: 'user3@example.com', type: 'promo' }, 10000); // 10 seconds console.log('All test jobs scheduled. You can close this script.'); redis.disconnect();
} runTest();

To run the test, open two terminal windows. In the first, start your worker:

node worker.js

In the second, run the test script to populate the queue:

node test.js

Keep an eye on the worker terminal. Even though we queued the 5-second job first, the worker processes the 2-second job first, then the 5-second one, and finally the 10-second one. The sorting works exactly as expected.

What Happens When a Worker Crashes?

This setup is incredibly lightweight, but there is a catch: if a worker grabs a job and crashes halfway through processing it, that job is gone forever. Our Lua script deletes it from the queue immediately upon retrieval.

If you are handling critical data where jobs absolutely cannot be lost, you have two options:

1. Use a “Processing” Set: Instead of deleting the job, your Lua script atomically moves it from the main queue to an “active” Sorted Set with a visibility timeout. Once the worker finishes, it deletes the job from the active set. If a job hangs in the active set for too long, a separate cleanup script pushes it back to the main queue.
2. Use a Distributed Lock: Grab a temporary lock on the job ID using Redis distributed locks, run your job, and delete it only when finished. If the worker dies, the lock expires and another worker picks it up.

For mission-critical stuff like billing, building these safety nets yourself gets messy fast. If you find yourself writing hundreds of lines of recovery code, save yourself the headache and use a battle-tested library like BullMQ, which handles all of this out of the box.

Frequently Asked Questions

Can I change a job’s delay after scheduling it?

Yep. Since Sorted Sets update existing members when you add them with a new score, rescheduling is easy. Just make sure the job payload (including the UUID) is identical, and call zadd again with the new timestamp.

How many jobs can this handle?

Since Redis keeps everything in memory, your only real limit is RAM. A typical 200-byte job payload means you can easily store millions of jobs in a cheap 1GB Redis instance. Your bottleneck will almost always be your worker’s execution speed, not Redis.

What happens if Redis restarts?

If you have persistence (AOF or RDB) enabled, your jobs will survive a restart. Note that default Docker setups usually don’t persist data to disk. Make sure your production Redis instance has AOF (Append Only File) turned on so you don’t lose your queue during maintenance.

Is this better than using setInterval in Node.js?

Absolutely. setInterval or setTimeout keeps everything in your Node process’s memory. If your app crashes, restarts, or redeploys, your entire queue vanishes. Offloading the schedule to Redis keeps your jobs safe and decoupled from your app’s lifecycle.

Next Steps

You now have a lightweight, atomic delayed queue running on zero heavy dependencies. If you are handling failed API calls, you can easily extend this into a full webhook retry system with exponential backoff by rescheduling failed jobs with longer delays.

If you eventually need advanced features like UI dashboards, parent-child dependencies, or complex retry logic, check out our guide on how to manage complex queues with BullMQ.

Official resources

• Node.js documentation