Making a scheduled job resumable after partial failure

A nightly job starts at three in the morning, works through 200,000 records, and dies at 03:40 with about 80% done. Nobody is awake. It runs again the following night from the beginning.

If that job is idempotent you lose some compute and nothing else. If it is not, you have just sent 160,000 duplicate emails, or double-charged a batch of invoices, and the morning is going to be unpleasant.

Resumability is what makes the difference, and it is mostly about where you keep three small pieces of state.

What does resumable actually mean?

A resumable job can be killed at any point and, on its next run, do only the work that is still outstanding, rather than repeating everything or abandoning the rest.

That needs three things, and none of them are complicated on their own:

• A record of what has already been done, stored somewhere that survives the crash.
• Work split into units small enough that losing one is cheap.
• Each unit safe to run twice, because at some point one of them will be.

The hard part is that these live in different places on most stacks. The cron runs on one host, the state sits in a database somewhere else, and the units of work go through a queue service that is a third thing to configure.

Why restarting is worse than it sounds

Do the arithmetic before deciding this does not matter.

A job processing 200,000 records at 100 records a second takes a little over half an hour. Fail at 80% and a naive restart repeats 160,000 units of work, so the second attempt is slower than the first and more likely to hit the same wall. If each unit makes an external API call that you pay for, you have also paid twice for 160,000 calls.

Jobs also tend to grow. The version that finishes comfortably in twenty minutes today is the one that starts overrunning its window next year, and an overrunning job that cannot resume is a job that never finishes again.

Step one: stop doing the work inline

The single most useful change is to stop processing records inside the job.

Have the scheduled job find what needs doing and queue one item per record, then finish. Workers pick items off the queue independently. A crash in the job now costs you the discovery pass, which is one query, rather than everything the job had processed so far. A crash in a worker costs you one record.

On Codehooks that discovery pass is one call:

const { ticket } = await db.enqueueFromQuery('orders', {
  synced: false
}, 'sync-order');

console.log(`queued ${ticket.count} orders`);

enqueueFromQuery enqueues one job per matching record, so the scheduled job stays short no matter how much work it finds. It resolves to a ticket carrying the job id and the number of records queued.

Step two: make each unit idempotent

Assume every unit of work will run twice at some point, because eventually it will.

The usual approach is a marker on the record itself. Check it at the top of the worker, set it at the end, and a repeat becomes a no-op rather than a duplicate.

app.worker('sync-order', async (req, res) => {
  const db = await Datastore.open();
  const { id } = req.body.payload;

  // findOneOrNull, because findOne throws when nothing matches
  const order = await db.findOneOrNull('orders', { id });
  if (!order || order.synced) return res.end();   // nothing to do

  await pushToPartner(order);
  await db.updateOne('orders', { id }, { synced: true });
  res.end();
});

Where the work is not naturally idempotent, such as sending an email or charging a card, keep the marker in the same write as the side effect where you can, or store an idempotency key that the downstream API recognises. Most payment and email providers accept one.

Step three: keep a cursor you can trust

For incremental work, store where the last successful run finished, so the next run only looks at what changed since.

Three rules matter more than the implementation.

Write the cursor only after the run has succeeded. If you update it at the start, a failed run has skipped a window permanently, and you will not notice until someone asks about a missing record months later.

Overlap the window. Subtract an hour or so when you read it back, for the clock-skew reasons covered in nightly reconciliation. Reprocessing an hour of already-done work is free if step two is in place.

Never put the window on the outstanding-work query. This is the one that catches people. A record that failed to sync a week ago still carries last week's timestamp, so a updated: { $gte: since } filter drops it after the first night and it is gone for good. The cursor belongs on the sweep for recently changed records. The outstanding query gets no time filter at all, which is fine because records leave it as soon as they succeed.

app.job('0 3 * * *', async (req, res) => {
  const db = await Datastore.open();

  // No cursor on the first run, so start from the epoch. get() returns
  // null for a missing key, and Date.parse(null) is NaN.
  const last = await db.get('sync-cursor');
  const since = last
    ? new Date(Date.parse(last) - 3600 * 1000).toISOString()
    : new Date(0).toISOString();

  // Still outstanding, at any age
  const stuck = await db.enqueueFromQuery('orders', {
    synced: false
  }, 'sync-order');

  // Changed since the last successful run
  const changed = await db.enqueueFromQuery('orders', {
    updated: { $gte: since }
  }, 'sync-order');

  console.log(
    `queued ${stuck.ticket.count} stuck, ${changed.ticket.count} changed`
  );

  await db.set('sync-cursor', new Date().toISOString());
  res.end();
});

export default app.init();

The worker from step two makes the overlap between those two queries a no-op, which is why they can be this blunt.

The cursor lives in the key-value store, the records live in the database, and the schedule runs in the same deploy as both. That matters for resumability more than it first appears. There is no separate queue service or state store to configure, so no third system whose outage can strand a run halfway through. A crash between the enqueue and the cursor write is still possible, which is what the overlap window is there to mop up.

What about retries?

Be careful what you assume here, because it is easy to believe more happens automatically than does.

A queue is persistent, so enqueued work is not lost. But if your worker throws, nothing retries it for you. That is your code's decision, and for scheduled work the simplest answer is usually to do nothing clever: catch the error, record it on the record, leave the marker unset, and let the next scheduled run pick it up. A nightly job that re-queues anything still outstanding is a retry mechanism, just a slow one.

try {
  await pushToPartner(order);
  await db.updateOne('orders', { id }, { synced: true });
} catch (err) {
  // Leave it unsynced so the next run tries again
  await db.updateOne('orders', { id }, { lastError: String(err) });
}

That is enough when a day's delay is acceptable, which for most nightly work it is. Where it falls short is anything that has to recover within the hour.

When to use the Workflow API instead

If you need retries with a count, backoff between attempts, or a multi-step process that resumes from the exact step it died on, write it as a workflow rather than assembling it from jobs and workers.

The Workflow API persists state between steps and resumes from the last completed one after a crash, and each step takes its own maxRetries. That is the difference between the two approaches: with jobs and workers you own the resume logic, and with workflows the framework owns it.

The rule of thumb we use: one kind of work over many records, reach for a job plus a queue. Several dependent steps per item where a failure halfway through leaves something inconsistent, reach for a workflow.

Summary and conclusion

A scheduled job becomes resumable when the state that says what is done outlives the process doing it. Keep the discovery pass short by queueing one item per record, make each item safe to run twice, and store a cursor you only advance after a successful run.

Retries are not automatic at the queue level, so either let the next scheduled run act as the retry, or use the Workflow API when you need retry counts and step-level resume.

The reason this is less work on Codehooks is that the schedule, the queue, the cursor and the records are all in the same deploy, which is covered in more detail on scheduled jobs.