Retrieval provides current input without retraining weights

An original document assistant can use retrieval to supply current policies at inference time. Fine-tuning is a separate development process with its own dataset and evaluation.

Product memory or conversation storage is another separate capability.

Service data policies are a separate question

A provider may have data-use policies that allow submitted information to be retained or used later. That is a service-policy question, not a reason to assume every individual request immediately trains the model.

What training is doing

Training is the stage that adjusts weights. Pretraining predicts tokens across a huge corpus.

Fine-tuning continues that adjustment on a narrower set of examples. Preference training, such as reinforcement learning from human feedback, changes weights using judgments about which answer is better.

When training finishes, the weights are a fixed artifact until someone trains again.

Training needs data, a loss function, and a lot of compute. It also needs a decision about what the examples are allowed to teach.

A model can memorize a rare string that appeared a few times. That is why training data, not the chat window, is the place to argue about private documents and copyrighted text.

What inference is doing

Inference does not update the weights. The model reads a new prompt and produces tokens.

Temperature, tools, and a retrieved document change the input or the sampling. They do not write a new checkpoint.

If the answer is wrong, the next prompt can try a different instruction, but the underlying model is the same until you deploy a new version.

Product teams usually pay for inference per token or per machine-hour. They pay for training when they create or adapt a checkpoint.

Mixing the words hides the bill. A “training” dashboard that only shows chat usage is an inference report.

A fine-tune job that writes a new model id is training, even if it took minutes.

Training and inference are different jobs
QuestionTrainingInference
Does it change weights?Yes, if the job is allowed to update themNo. The checkpoint stays as deployed
What do you supply?Examples, a loss, and computeA prompt, optional tools, and a finished model
Where does a private document go?Into the training set if you choose thatInto that request, subject to the provider’s retention rule

What to record before you rely on it

Record whether a job writes a new checkpoint or only returns tokens. If it writes a checkpoint, name the training examples, the base model, and who is allowed to deploy the result.

If it only returns tokens, name the model id and the retention rule for that request. A dashboard that mixes the two will send a team looking for a data leak in the wrong system.

The note can be a single column on the design document: “weights change” or “weights stay.”

Keep the model id, the date, and the inputs you used beside that note, so a later change can be compared with the same test.

Common questions

Is retrieval-augmented generation a kind of training?

No. Retrieval puts extra text into the prompt for one request. The weights stay as they were.

Fine-tuning is the path that changes weights from examples.

Why does a trained model still fail on a new sentence?

Training fits the weights to the examples and the loss. A new sentence can sit outside that pattern.

Inference has no memory of your correction unless you send it again or train a new checkpoint.

Sources & further reading

  1. Hugging Face: Fine-tuning ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories