Worker queues

What is a Worker Queue?

A worker queue is a system for handling background tasks asynchronously by distributing jobs to queue workers (processes). In a serverless environment, worker queues enable persistent and scalable job execution, allowing you to decouple long-running or compute-heavy tasks from user-facing code.

With worker queues, you can process jobs like sending emails, generating PDFs, or running workflows reliably — even under load.

How Worker Queues Work

Objects (tasks) are added to worker queues using the built-in JavaScript Queue API. Serverless worker queue functions process each object in a queue asynchronously. The illustration below shows how three independent worker queues (named topic-1, topic-2, topic-3) with their payload objects (x) are being processed by workers. Note how topic-1 has multiple parallel workers dequeuing items from the same queue.

Worker Queue Example (Serverless Function)

The following example worker queue function prints to the log each time a request.body is added to the queue as a POST to the example route /addQueue.

index.js

import { app, Datastore } from 'codehooks-js';

// set the number of parallel workers
const workerOptions = {workers: 1}

// Serverless worker queue function for topic 'myqueue'
app.worker('myqueue', (req, res) => {
  console.log('The queue item is', req.body.payload);
  res.end(); // done with one item
});

// API route that adds to the worker queue
app.post('/addQueue', async (req, res) => {
  const conn = await Datastore.open();
  const jobId = await conn.enqueue('myqueue', { data: req.body });
  res.status(201).json({ jobId });
});

export default app.init();

Worker queue configuration

• Topic names are unique strings. E.g. "sendEmails"
• Queue data are accessed through the request.body.payload object
• Each topic is processed as a separate queue
• workerOptions controls the number of parallel workers. For example: { workers: 5 }
• Each queue is stored in separate system collections prefixed with _queue_{TOPIC}. E.g. if the topic is sendEmail the queue would be named _queue_sendEmails
• Read more about the JavaScript Queue API

tip

If you need more complex functionality, which uses worker queues, make sure to check out the Workflow API.