It is Tuesday morning. A backend engineer flags that the /generate endpoint took 90 seconds for a real user request. The load balancer disconnected at 60. The browser gave up at 30. The team agrees to move it to a queue. Two days later the same LLM call is running three times per user request, and the week's bill is 2.4x normal. The queue system that runs the rest of the application is doing exactly what its documentation promised: at-least-once delivery, thirty-second visibility timeout, generous retry-on-error.

The team did not misconfigure the queue. The team applied the queue architecture they knew, to a workload that violates three of the assumptions that architecture is built around. AI jobs look like regular background jobs from a distance (task in, result out, done in the background), but at close range they behave differently in ways that break the defaults and turn a routine "let's move this to a queue" into a runaway-spend incident.

Issue 008 covered the client-side streaming architecture that delivers long-running LLM responses to a browser. This issue covers the backend that produces the tokens that streaming layer serves. What follows is the three ways AI jobs are not "just longer regular jobs", the three queue patterns worth knowing, the retry rules that keep the bill sane, and the common mistakes that turn the whole thing into a story you tell in the retrospective.

Why AI jobs are not "just longer regular jobs"

Three assumptions built into every mainstream queue library break the moment the payload is an LLM call. Recognising which one your current architecture is defaulting into is the fastest way to fix the failure mode you have not seen yet.

Duration is highly variable. A /summarise endpoint might return in five seconds for a paragraph and four minutes for a thirty-page PDF, off the same route with the same handler. The distribution has fat tails: p50 might be twelve seconds, p99 might be three hundred. Fixed visibility timeouts (SQS default is 30 seconds, Celery and Sidekiq defaults sit in a similar range) drop the assumption on the floor. The worker is still generating tokens. The queue thinks the job is dead. Another worker picks it up. Now there are two LLM calls in flight for one user request, both of which will eventually return successfully, and the cost of the request has silently doubled. Multiply by the p99 tail, multiply by a week of traffic, and the bill is 2.4x normal.

Retries cost real money. In a normal background job, a retry costs a few hundred milliseconds of CPU. In an AI job, a retry costs whatever the tokens cost, anywhere from tenths of a cent to several dollars depending on the model and prompt length. The default retry-on-5xx logic that ships with every mainstream queue library will happily burn a hundred dollars in an hour if the LLM is returning content-policy refusals that look like transient failures. This is where the "queues I already know how to run" mental model breaks hardest, and it is the failure mode most likely to make the story about the queue architecture rather than about the AI feature.

Idempotency is harder than the docs suggest. LLM outputs are non-deterministic even at temperature zero, because batching effects, GPU non-determinism, and occasional silent model updates mean two calls with the same prompt can return different completions. Content-based deduplication does not work; the queue cannot tell a genuine duplicate from a legitimate re-run of the same request that happened to produce a different output. The fix is idempotency keys generated per job at enqueue time and passed through to the LLM SDK on every call. Anthropic supports an Idempotency-Key header that caches a response for 24 hours; the OpenAI SDK supports the same pattern. Send the same key on a retry, get the cached response back, no second LLM call, no doubled bill.

The three queue patterns worth knowing

Three patterns dominate production usage. Pick the simplest one that fits the workload; do not reach for a workflow orchestrator when a durable queue with correct timeouts would do.

Durable queue with correct timeouts and idempotency keys. The default for most teams and the right answer for single-call AI jobs. Use whatever queue system the rest of the application uses (Redis Streams, SQS, Postgres with LISTEN/NOTIFY), but set the visibility timeout to at least the p99 of your job duration plus a buffer (ten minutes is a reasonable starting point for LLM work), wire an idempotency key on every LLM call, and cache the first successful response for the length of the retry window. This pattern handles fire-and-forget jobs well. It handles multi-call chains poorly and offers no visibility into progress, which is the limitation the next pattern addresses.

Job with state tracking in a database. For anything user-facing where progress matters. A jobs row in Postgres records the state (pending → running → completed → failed) and lets the client poll /jobs/{id} to see progress. A separate job_tokens table, or a Redis Streams key indexed by job id, holds the streaming intermediate results the client subscribes to. This is the backend that feeds the client-side streaming architecture from Issue 008: the state row survives worker restarts, and the streaming layer survives client reconnects, so a mobile user who backgrounds the app for thirty seconds does not restart the LLM call from scratch. The added complexity is a schema and a small amount of extra write traffic. For any user-facing feature with a stream, the pattern pays back on the first client disconnect.

The diagram shows the state-tracked pattern in full. The enqueue is synchronous and returns a job id in milliseconds; the LLM work happens in the worker, which writes both the terminal state (to the job state DB) and the streaming intermediate results (to the stream store). The client reads from the streaming layer independently of the worker's lifecycle, which is what makes reconnect trivial.

Workflow orchestration (Temporal, Restate, Inngest, Trigger.dev, as of mid-2026). For multi-step chains where you need per-step retry rather than per-workflow retry. The mental model shift is that each LLM call becomes an "activity" with its own retry policy and idempotency guarantee, and the workflow engine tracks progress through the chain durably. If step four of a five-step chain fails, only step four retries; steps one through three keep their outputs and do not re-run. The operational complexity is real (another system to run, another mental model to internalise, another SDK to keep current), but for anything with three or more LLM calls the pattern pays for itself the first time step four fails and the alternative is re-running steps one and two at a dollar each.

Retry semantics for AI jobs

Retry logic that works for backend services burns money on AI jobs. Three rules make the difference between a queue that survives contact with production traffic and a queue that generates a five-figure bill overnight.

Idempotency keys on every LLM call. Generate a UUID per job at enqueue time. Pass it as the SDK's idempotency header on every LLM call inside that job. When the queue retries the job (because at-least-once delivery), the same idempotency key returns the same response instead of generating a new one. Total LLM cost stays at one call regardless of how many times the queue retries. This is the single biggest change most teams make in the first week after realising they have an AI-jobs problem.

Classify errors before retrying. Retry on transient errors (rate limits, network errors, 5xx from the provider). Do not retry on persistent errors (content policy refusals, invalid request errors, quota exceeded, authentication failures). The distinction is boring in traditional systems where every retry costs the same fraction of a cent. In AI systems, the persistent-error retry loop is the single most common runaway-spend pattern. If the queue library retries every non-2xx, wrap the LLM SDK call in a shim that catches persistent errors and re-raises them as non-retryable in whichever exception hierarchy the queue library respects.

Cap the retry budget per user request. Every user request should carry a maximum token spend, computed from a multiple of the expected cost of a single successful run. If retries push the cumulative spend past the cap, the job fails hard rather than continuing. A $0.30 LLM call retried five times is $1.80. At a thousand requests per hour, that is $1,800 per hour of runaway spend, and the graph in the observability stack from Issue 004 will not warn you until the daily aggregate lands the next morning. A hard per-request budget of, say, three times the expected cost is the difference between "we had a rough afternoon" and "we had a rough month".

Common mistakes

Four failure modes recur often enough to name directly.

Default visibility timeout. Setting the queue visibility timeout to the library default (30 seconds on SQS, similar in Celery and Sidekiq) and losing money to duplicate LLM calls when the worker is still generating past the timeout. Set the timeout to at least the p99 of your job duration plus a buffer, and re-tune it every time a new prompt template or a longer model context lands in production.

Retry on any error. Not distinguishing transient errors (5xx, rate limit, network) from persistent ones (content policy, invalid request, quota). Content-policy errors will not become transient errors on retry; they will keep burning tokens until either a budget cap catches them or a human notices the bill. The wrapper that classifies errors before letting the queue library see them is a hundred lines of code that saves months of vigilance.

No cost cap on retries. No hard ceiling on token spend per user request. A runaway retry loop can generate a five-figure bill overnight, and the aggregate cost metrics in most observability stacks are lagging enough that the alert fires after the money is gone. A per-request budget prevents this.

Restarting the whole workflow on step-three failure. The strongest argument for workflow orchestration is per-step retry. If your workflow re-runs steps one and two every time step three fails, and each step is a $0.30 LLM call, the cost of "just retrying" adds up faster than anyone expects. Once a chain has three or more LLM calls, the case for a workflow orchestrator over a plain queue gets hard to argue against.

The takeaway

AI background jobs are not just longer versions of regular background jobs. Three assumptions built into every mainstream queue library break: duration is variable enough to defeat default timeouts, retries cost real money instead of CPU cycles, and idempotency requires SDK-level keys rather than content dedup. The queue patterns that work are ordered by complexity: durable queue with correct timeouts and idempotency keys for single-call jobs, state-tracked jobs for user-facing progress that composes with the streaming architecture from Issue 008, and workflow orchestration for multi-step chains where per-step retry saves the tokens that per-workflow retry burns. The three retry rules apply regardless of pattern. Ship the simplest architecture that fits the current workload; graduate when the failure mode of the current architecture shows up in production, not before.

Production checklist

  • Set the queue visibility timeout to the p99 of job duration plus a buffer, not the queue library's default. Re-tune when a new prompt template or a longer-context model lands.

  • Generate an idempotency key per job at enqueue time. Pass it as the SDK's idempotency header on every LLM call inside the job.

  • Classify errors before retrying. Transient errors (5xx, rate limit, network) retry; persistent errors (content policy, invalid request, quota, authentication) fail hard.

  • Cap the token budget per user request. If cumulative retries push past the cap, fail the job rather than continuing to spend.

  • Record job state in a database (pending → running → completed → failed) for any user-facing workflow. Combine with the streaming layer from Issue 008 for progress display.

  • Adopt a workflow orchestrator (Temporal, Restate, Inngest, Trigger.dev, as of mid-2026) once chains have three or more LLM calls, or once "restarted from step one" appears as a recurring cost line.

  • Log per-step token cost per job, sliced by tenant and prompt template, in the observability stack from Issue 004. Cost is the first place a retry-budget bug shows.

  • Re-evaluate the queue architecture annually. Workflow orchestrators for AI jobs are the fastest-moving part of the stack; the right choice this year may not be the right choice next.

Further reading