Long AI jobs need a proper tool, not a scheduled task and a prayer
Long, multi-step AI jobs fail partway through all the time. Here is why we use a tool built for that instead of stitching queues together and hoping.
The demo always works. You upload a file, the AI reads the text, works out its meaning, tags it, and files it away so you can search it. Nice and clean. Then you run it on ten thousand real files and reality shows up. A model times out. An outside service tells you to slow down. A worker restarts halfway through a job. Now what?
The fragile way
Most teams start by chaining things together. One message kicks off step one, which kicks off step two, and so on. It works until something fails halfway. Then you are stuck with hard questions:
- Which step was this file on when the worker died?
- Is it safe to try again, or will we charge someone twice or save the same thing twice?
- How do we pick up the other 400 files that were partway through?
You end up building your own tangle of retry logic and error handling in your database. You have reinvented a proper tool for this, badly.
The reliable way
A tool built for this (we use Temporal) changes how it works. You write the job as ordinary code, step one, step two, step three, and the tool makes sure it runs all the way through, exactly as you wrote it, even through crashes, restarts, and hours of waiting.
- It retries failed steps for you, without you writing any retry code.
- It picks up where it left off. If a worker dies, another one carries on from the same spot.
- You can see everything. The full history of every job is there to look at, so you are not guessing where a file got stuck.
When it is worth it
You do not need this for a contact form. You do need it the moment a job:
- Has several steps that can each fail on their own,
- Runs long enough that a restart will interrupt some of them, or
- Touches money, ownership, or anything you cannot safely do twice.
For AI jobs like these, all three are true at once. That is why we build them this way from the start. It is far cheaper than bolting reliability on after the first thing goes wrong in production.
Building something with long or AI-heavy jobs? Let us talk.