Skip to main content

Worker queues

What is a Worker Queue?​

A worker queue is a system for handling background tasks asynchronously by distributing jobs to queue workers (processes). In a serverless environment, worker queues enable persistent and scalable job execution, allowing you to decouple long-running or compute-heavy tasks from user-facing code.

With worker queues, you can process jobs like sending emails, generating PDFs, or running workflows reliably — even under load.

How Worker Queues Work​

Objects (tasks) are added to worker queues using the built-in JavaScript Queue API. Serverless worker queue functions process each object in a queue asynchronously. The illustration below shows how three independent worker queues (named topic-1, topic-2, topic-3) with their payload objects (x) are being processed by independent workers that eventually execute your function.

Worker Queue Example (Serverless Function)​

The following example worker queue function prints to the log each time a request.body is added to the queue as a POST to the example route /addQueue.

index.js
import { app, Datastore } from 'codehooks-js';

// Serverless worker queue function for topic 'myqueue'
app.worker('myqueue', (req, res) => {
  console.log('The queue item is', req.body.payload);
  res.end(); // done with one item
});

// API route that adds to the worker queue
app.post('/addQueue', async (req, res) => {
  const conn = await Datastore.open();
  const jobId = await conn.enqueue('myqueue', { data: req.body });
  res.status(201).json({ jobId });
});

export default app.init();

Worker queue configuration​

  • Topic names are unique strings. E.g. "sendEmails"
  • Queue data are accessed through the request.body.payload object
  • Each topic is processed as a separate queue
  • QUEUE_WORKERS environment variable manage number of parallel workers. E.g. QUEUE_WORKERS=10
  • Each queue is stored in separate system collections prefixed with _queue_{TOPIC}. E.g. if the topic is sendEmail the queue would be named _queue_sendEmails
  • Read more about the JavaScript Queue API
tip

If you need more complex functionality, which uses worker queues, make sure to check out the Workflow API.