2 posts tagged with "serverless cron"

A nightly job starts at three in the morning, works through 200,000 records, and dies at 03:40 with about 80% done. Nobody is awake. It runs again the following night from the beginning.

If that job is idempotent you lose some compute and nothing else. If it is not, you have just sent 160,000 duplicate emails, or double-charged a batch of invoices, and the morning is going to be unpleasant.

Resumability is what makes the difference, and it is mostly about where you keep three small pieces of state.

Every integration between two systems drifts, and usually not through any fault in the code. One side was unavailable for four minutes, or a request timed out after the write had already landed, or someone edited a record by hand.

The integration itself does not notice. It processed the event, got an error, retried a few times and gave up. From its point of view that is a handled failure. From the client's point of view, their stock numbers are wrong.

A reconciliation job is the thing that finds those records. It runs on a schedule, compares the two sides, and re-queues whatever does not match.