Running BullMQ Background Workers on Persistent Containers
BullMQ is the leading Node.js job queue library, used by thousands of apps for background work: sending emails, processing payments, generating reports, resizing images, and any task that should not block the request-response cycle. It is fast, reliable, and built on Redis, which makes it a natural choice for AI-generated apps that need background processing. But BullMQ workers are long-lived, stateful processes that do not fit the serverless model, because they need to maintain a persistent connection to Redis and hold worker state in memory. Deployxa's persistent containers are the natural home for BullMQ workers, and the zero-config engine handles the Redis deployment and worker configuration automatically. Here is how to deploy BullMQ workers on Deployxa.
The direct answer is that BullMQ workers are long-lived processes that maintain a persistent connection to Redis, poll for jobs, execute them, and report the results. This architecture requires a persistent container (for the worker process) and a Redis instance (for the queue). Serverless platforms, which spin processes down when idle and impose timeout limits, cannot host BullMQ workers, because the workers would be killed mid-job and the Redis connection would be dropped. Deployxa's persistent containers keep the worker alive indefinitely, which means jobs run to completion, the Redis connection stays open, and the worker state (concurrency, rate limiting) persists across jobs.
Why BullMQ Workers Need Persistent Containers
Three properties of BullMQ workers make persistent containers essential:
1. Long-lived worker processes
BullMQ workers are designed to run as long-lived processes. The worker starts, connects to Redis, and polls for jobs in a loop. When a job arrives, the worker executes it and reports the result. This loop continues indefinitely, which means the worker process must stay alive. Serverless platforms, which spin processes down when idle, cannot host BullMQ workers, because the worker would be killed when idle and would miss jobs that arrive during the downtime.
2. Persistent Redis connection
BullMQ workers maintain a persistent connection to Redis, which is used to poll for jobs, report results, and handle retries. This connection is expensive to establish (TCP handshake, authentication, Redis client initialization), so BullMQ reuses it across jobs. Serverless platforms, which spin processes down and create new ones for each invocation, cannot maintain a persistent Redis connection, which means each job requires a new connection, which is slow and resource-intensive.
3. Worker state
BullMQ workers maintain state in memory: the concurrency level (how many jobs to process simultaneously), the rate limit (how many jobs to process per second), and the active jobs (which jobs are currently being processed). This state is assumed to persist across jobs. Serverless platforms, which create a new process for each invocation, cannot maintain this state, which means concurrency and rate limiting do not work correctly.
The Architecture: Worker + Redis + Producer
Here is the architecture of a BullMQ deployment on Deployxa.
The Redis container
BullMQ requires Redis as the queue backend. Deployxa can deploy Redis as a sibling container, which means your worker and Redis run on the same infrastructure with low-latency networking. The Redis container is configured automatically, and you just need to set the REDIS_URL environment variable on your worker container.
The worker container
The worker container runs your BullMQ worker code. It connects to Redis, polls for jobs, and executes them. The worker is a long-lived process that stays alive indefinitely. You can scale the worker to multiple containers for higher throughput, and Deployxa's load balancer distributes jobs across them (via Redis's built-in job distribution).
The producer
The producer is the part of your app that enqueues jobs. It can be your web server (e.g., an Express route that enqueues an email job when a user signs up) or a separate service. The producer connects to the same Redis instance and enqueues jobs, which the worker then picks up and processes. For more on this pattern, see our article on why AI-generated cron jobs don't run on serverless, which covers a similar architecture for scheduled tasks.
Step-by-Step: Deploying a BullMQ Worker on Deployxa
Here is the exact workflow for a typical BullMQ worker.
Step 1: Install BullMQ
npm install bullmq ioredisStep 2: Create the queue and worker
// worker.js
const { Worker, Queue } = require('bullmq');
const IORedis = require('ioredis');
const connection = new IORedis(process.env.REDIS_URL, {
maxRetriesPerRequest: null,
});
// Define the queue
const emailQueue = new Queue('email', { connection });
// Define the worker
const worker = new Worker('email', async (job) => {
const { to, subject, body } = job.data;
console.log(`Sending email to ${to}: ${subject}`);
// Send the email (e.g., via SendGrid)
await sendEmail(to, subject, body);
console.log(`Email sent to ${to}`);
}, {
connection,
concurrency: 5, // process up to 5 jobs simultaneously
});
worker.on('completed', (job) => {
console.log(`Job ${job.id} completed`);
});
worker.on('failed', (job, err) => {
console.error(`Job ${job.id} failed:`, err);
});
console.log('Worker started, waiting for jobs...');Step 3: Create the producer (in your web server)
// server.js
const express = require('express');
const { Queue } = require('bullmq');
const IORedis = require('ioredis');
const app = express();
app.use(express.json());
const connection = new IORedis(process.env.REDIS_URL, {
maxRetriesPerRequest: null,
});
const emailQueue = new Queue('email', { connection });
app.post('/signup', async (req, res) => {
const { email, name } = req.body;
// Create the user...
// Enqueue the welcome email
await emailQueue.add('welcome', {
to: email,
subject: 'Welcome to our app!',
body: `Hi ${name}, welcome aboard!`,
});
res.json({ status: 'ok' });
});
app.listen(process.env.PORT || 3000, () => {
console.log(`Server running on port ${process.env.PORT || 3000}`);
});Step 4: Push to GitHub
git init
git add .
git commit -m "bullmq worker and producer"
git remote add origin https://github.com/yourname/my-app.git
git push -u origin mainStep 5: Connect to Deployxa
In the Deployxa dashboard, connect your repository. Deployxa detects the Node.js project and configures the build and start commands. You will need to deploy three services: the web server (producer), the worker, and Redis. Deployxa's polyglot engine can handle all three as sibling containers.
Step 6: Configure environment variables
For both the web server and the worker, set REDIS_URL to the Redis container's connection string (e.g., redis://my-app-redis:6379). For the web server, also set PORT and any other required variables.
Step 7: Deploy
Deploy all three services. The web server starts, the worker starts and connects to Redis, and Redis is ready to accept connections. Jobs enqueued by the web server are picked up by the worker and processed.
Step 8: Monitor the worker
Use deployxa get logs to monitor the worker's output. The logs show when jobs are picked up, when they complete, and when they fail. For more sophisticated monitoring, you can use BullMQ's built-in metrics (via bull-board or bull-exporter for Prometheus).
Common Pitfalls and Troubleshooting
The first pitfall is Redis connection management. BullMQ requires maxRetriesPerRequest: null on the Redis connection, because it uses blocking commands (like BRPOP) that do not have a retry semantic. If you forget this setting, BullMQ will throw an error. The second pitfall is job idempotency. If a job fails and is retried, it might be executed multiple times, which can cause issues for non-idempotent jobs (e.g., sending an email twice). The fix is to make jobs idempotent (e.g., check if the email was already sent before sending it again) or to use BullMQ's job ID feature to deduplicate. The third pitfall is worker concurrency. Setting concurrency too high can overwhelm the worker's resources (CPU, memory, database connections). The fix is to start with a low concurrency (e.g., 5) and increase it based on monitoring. The fourth pitfall is dead letter queues. BullMQ's default behavior is to retry failed jobs up to a limit (default 20), after which the job is moved to a "failed" set. The fix is to configure a dead letter queue (a separate queue for jobs that have exhausted their retries) and to monitor it for jobs that need manual intervention. The fifth pitfall is Redis persistence. By default, Redis persists data to disk, which means jobs survive Redis restarts. If you configure Redis without persistence (for performance), jobs will be lost on Redis restart. The fix is to enable Redis persistence (AOF or RDB) for production deployments.
Scaling BullMQ Workers
As your app grows, you may need to scale your BullMQ workers to handle higher job volume. The first scaling approach is to increase the worker's concurrency (e.g., from 5 to 10), which lets a single worker process more jobs simultaneously. This is the simplest approach, but it is limited by the worker's resources (CPU, memory, database connections). The second scaling approach is to deploy multiple worker containers, which distributes jobs across them via Redis's built-in job distribution. Deployxa's persistent containers support this natively: just scale the worker service to multiple containers, and BullMQ handles the distribution. The third scaling approach is to use separate queues for different job types (e.g., email queue, image-processing queue), which lets you scale each queue independently. This is useful when some job types are fast (emails) and others are slow (image processing), because you can scale the slow queue to more workers without over-provisioning the fast queue. For more on scaling, see our article on SPA vs SSR hardware sizing, which covers Deployxa's automatic sizing and manual override features.
Advanced BullMQ Patterns
Beyond the basics, BullMQ benefits from several advanced patterns. The first is job priorities. BullMQ supports job priorities, which let you process high-priority jobs before low-priority ones. For example, a password reset email (high priority) should be sent before a weekly digest (low priority). The second is job scheduling. BullMQ supports delayed jobs, which run at a specific time in the future. This is useful for reminders (e.g., "send a reminder email 3 days after signup"). The third is job repetition. BullMQ supports repeatable jobs, which run on a schedule (e.g., daily, weekly). This is an alternative to cron jobs for scheduled tasks. The fourth is job flows. BullMQ supports job flows (parent-child relationships), where a parent job waits for its children to complete. This is useful for multi-step workflows (e.g., "process image" parent job with "resize", "optimize", and "upload" child jobs). The fifth is queue events. BullMQ emits events (e.g., completed, failed, progress) that you can listen to for real-time monitoring. For example, you can emit a WebSocket event when a job completes, to update a progress bar in the UI. The sixth is queue pausing. BullMQ supports pausing a queue, which stops job processing without canceling queued jobs. This is useful for maintenance (e.g., pause the queue while you run a database migration). For more on BullMQ and background workers, see our articles on why AI-generated cron jobs don't run on serverless and long-lived WebSockets.
Conclusion: Give Your Workers a Persistent Home
BullMQ is the leading Node.js job queue library, but its workers are long-lived, stateful processes that do not fit the serverless model. Deployxa's persistent containers give BullMQ workers the home they need: no timeouts, no cold starts, no connection drops. Combined with the zero-config engine that handles Redis deployment automatically, Deployxa makes BullMQ deployment as simple as pushing to Git.
Ready to deploy your BullMQ workers? Drag your project to Deployxa Drop for an instant live preview, or install the CLI with npm i -g @deployxa/cli and deploy from your terminal. For more on background workloads, see our articles on why AI-generated cron jobs don't run on serverless and long-lived WebSockets. Learn about deploying Django + React and deploying Go Fiber APIs in our companion articles. Explore our free developer tools to speed up your workflow.