Learn how to create and manage repeatable jobs using Job Schedulers in BullMQ
A Job Scheduler acts as a factory, producing jobs based on specified “repeat” settings. The Job Scheduler is highly flexible, accommodating various scenarios, including jobs produced at fixed intervals, according to cron expressions, or based on custom requirements.
Job Schedulers replace “repeatable jobs” and are available in v5.16.0 and onwards. They are the recommended approach for creating recurring jobs in BullMQ.
To create a scheduler, use the upsertJobScheduler method:
import { Queue } from 'bullmq';const queue = new Queue('myQueue', { connection: { host: 'localhost', port: 6379 },});// Creates a new Job Scheduler that generates a job every 1000 milliseconds (1 second)const firstJob = await queue.upsertJobScheduler('my-scheduler-id', { every: 1000,});
This example creates a new Job Scheduler that produces a new job every second. It also returns the first job created for this Job Scheduler, which will be in “delayed” status waiting to be processed after 1 second.
The upsert operation (update or insert) is used instead of add to simplify management of recurring jobs, especially in production deployments. It ensures the scheduler is updated or created without duplications.
// First call creates the schedulerawait queue.upsertJobScheduler('my-scheduler-id', { every: 1000,});// Subsequent calls update the existing schedulerawait queue.upsertJobScheduler('my-scheduler-id', { every: 2000, // Updated interval});
The scheduler will only generate new jobs when the last job begins processing. Therefore:
If your queue is very busy
If you don’t have enough workers or concurrency
Jobs may be produced less frequently than the specified repetition interval
The actual job execution frequency depends on worker availability and processing speed. The scheduler guarantees jobs won’t be created more often than specified, but they may be created less frequently if the queue is congested.
As long as a Job Scheduler is producing jobs, there will always be one job associated with the scheduler in the “Delayed” status, waiting to be processed.
You can define a template with standard names, data, and options for jobs added to a queue. This ensures that all jobs produced by the Job Scheduler inherit these settings:
// Create jobs every day at 3:15 AMconst firstJob = await queue.upsertJobScheduler( 'my-scheduler-id', { pattern: '0 15 3 * * *' }, { name: 'my-job-name', data: { foo: 'bar' }, opts: { backoff: 3, attempts: 5, removeOnFail: 1000, }, },);
All jobs produced by this scheduler will use the given settings. You can call upsertJobScheduler again with the same scheduler ID to update any settings of this particular job scheduler, such as the repeat options or job template settings.
Since jobs produced by the Job Scheduler get a special job ID to guarantee that jobs will never be created more often than the given repeat settings, you cannot choose a custom job ID. However, you can use the job’s name to discriminate these jobs from other jobs.
The job ID format is: repeat:{jobSchedulerId}:{nextMillis}