Demonstrations and preferences shape behavior

The research combines supervised instruction training with a preference-based learning process. Human judgments provide a target for behavior, but depend on the instructions, examples, and evaluation criteria given to the reviewers.

Helpful wording and factual accuracy are separate

An original evaluation should separate helpfulness, factual accuracy, and willingness to state uncertainty. A system can sound more helpful while still giving an unsupported answer.

What the authors did

They collected demonstrations and rankings from people, trained a reward model on the rankings, and used that signal to update a pretrained model.

The point of the paper was that a smaller model tuned this way could be preferred by their evaluators over a larger model that was only prompted. Those preferences are their evaluators’ judgments on their prompts.

The method has costs the paper is explicit about: people are expensive, the reward model can be gamed, and a model can become more convincing without becoming more true.

Later arguments about RLHF, including the ones in product blogs, are not this paper. If you need the claim, read Ouyang and colleagues.

What not to import

Do not import the 2022 preference win into a 2026 product announcement. The model, the raters, and the prompts were different.

Do not say RLHF “aligns” a model in the abstract. Say it optimizes a reward built from the rankings they collected.

Alignment is a conclusion you would have to test.

What to record before you rely on it

When you mention InstructGPT, link the paper and name the metric as human preference on their tests. If you are describing a current API, say so, and do not pretend the paper measured it.

Keep the model id or the paper citation, the date you read it, and the inputs you used beside that note, so a later change can be compared with the same test.

When the result is a failure, store the artifact and change the design. Do not relax the check until the model happens to pass.

Name a person who owns that failure.

Common questions

Is every chat model InstructGPT?

No. Many later models used preference data of some kind. The name belongs to that paper and those checkpoints.

Use the name only when you mean them.

Does RLHF stop hallucinations?

The paper does not show that. A reward for answers people like can reward fluent answers.

Truth is a separate test, which is why later benchmarks such as TruthfulQA kept mattering.

Sources & further reading

  1. Training language models to follow instructions with human feedback ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories