What did the Chinchilla paper find?

Jordan Hoffmann and colleagues report the result in Training Compute-Optimal Large Language Models. They trained over 400 language models, from 70 million to over 16 billion parameters, on 5 billion to 500 billion tokens.

For compute-optimal training, they find that size and tokens should scale in equal proportions. They contrast that with Kaplan and colleagues, who for a tenfold compute increase had suggested about a 5.5-times larger model and only about 1.8-times more tokens.

They then trained Chinchilla at the compute budget they attribute to Gopher, with fewer parameters and more data.

The paper says Chinchilla uniformly outperformed Gopher (280 billion), GPT-3 (175 billion), Jurassic-1 (178 billion), and Megatron-Turing NLG (530 billion) on a wide set of downstream tasks, and that it reached 67.5 percent average accuracy on MMLU, more than seven points above Gopher.

Those are the authors’ measurements.

Models as listed in the paper’s table of large dense transformers
ModelParametersTraining tokens
GPT-3175 billion300 billion
Gopher280 billion300 billion
Megatron-Turing NLG530 billion270 billion
Chinchilla70 billion1.4 trillion

What does the result not decide?

The paper is explicit that the allocated training compute is often known in advance, and the question is how to trade parameters against tokens inside that budget.

It is not an instruction to copy 20 tokens per parameter for a different architecture, a different dataset, or a model you are only serving.

The authors also say Chinchilla uses substantially less compute for fine-tuning and inference than the larger models it beat. Inference cost still depends on how you run the model after training.

Common questions

Is Chinchilla a model you can download?

In the paper, Chinchilla is the 70-billion-parameter model DeepMind trained to test the compute-optimal prediction. The paper is the source for the scaling result.

It is not a release announcement for weights.

Should every new model be trained on 1.4 trillion tokens?

No. 1.4 trillion is the token count they report for Chinchilla under the compute budget they matched to Gopher. A different budget implies a different pair of size and tokens.

Sources & further reading

  1. Training Compute-Optimal Large Language Models ↗
How this story was made

Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.

Our editorial standards
Back to all stories