Match each result to its stable request identifier

An original content-tagging job can assign each request a stable identifier and match every returned result to its source article. Keep successful, failed, and unfinished items separate so retries do not repeat completed work.

Submission is different from completion

Submitting a batch is not the same as completing it. Monitor job state, handle partial failures, and validate results before importing them into another system.

What the discount is buying

Interactive APIs optimize for a person or a service that needs the next token soon. Batch APIs let the provider fill spare capacity.

In return, the job may take minutes or hours, and a provider may offer a lower price per token. Read the current price sheet.

A “half price” line from a blog can be out of date, and it may exclude some models.

The queue is not a different model unless the provider says it is. You should get the same checkpoint you named.

You will not get a socket that stays open. Design the client to submit, store the job id, and poll or receive a webhook.

A laptop that sleeps midway is a bad place to hold the only copy of the job id.

How a large job fails in small ways

One poisoned prompt can sit inside ten thousand good ones. Store each input, each output, and each error code separately.

Do not mark the file “done” because the job status says completed. Completed can mean every row was attempted, including rows the model refused or the validator rejected.

Idempotency matters. A retry of the whole file will pay again for the rows that already succeeded.

Key each row. Skip keys you already accepted.

Cap the spend in the provider’s console as well as in your own counter. A loop that resubmits on every error is a bill, not a retry policy.

  • Use batch only when an hour of delay is acceptable.
  • Confirm the price and the model id on the current rate card.
  • Store a result per row, including errors.
  • Retry failed keys, not the entire successful file.

What to record before you rely on it

Store the job id, the model id, the price you believed applied, and a result row for every key. Write the retry rule: failed keys only, never the whole successful file.

Cap spend in the provider console. Inspect the first slice of outputs before you submit the rest of a new prompt.

Record the delay you actually observed, because a batch discount is not useful for a person waiting in a chat.

Keep the model id, the date, and the inputs you used beside that note, so a later change can be compared with the same test.

Common questions

Is batch output lower quality?

Not because it is batch. Quality follows the model and the prompt.

What you lose is the chance to stop a bad pattern after the first row, unless you inspect early results before submitting the rest.

Should user-facing chat go through batch?

No. A person waiting in a chat needs an interactive call. Batch fits labeling, backfills, and reports that can arrive later.

Sources & further reading

  1. Anthropic: Batch processing ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories