Sources checked
What did the paper replace?
Ashish Vaswani, Noam Shazeer, Niki Parmar, and colleagues state the claim in Attention Is All You Need. The dominant sequence models of that moment were recurrent or convolutional networks with an encoder and a decoder.
The best of them also used attention to connect encoder and decoder. The Transformer uses attention only.
They argue this is more parallelizable and faster to train than the recurrent models they compare.
Which results are actually in the paper?
On WMT 2014 English-to-German, their model reaches 28.4 BLEU, which they describe as more than 2 BLEU above the existing best results, including ensembles.
On WMT 2014 English-to-French, they report 41.8 BLEU for a single model after training 3.5 days on eight GPUs, and they call that a fraction of the training cost of the best models in the literature they cite.
They also apply the same architecture to English constituency parsing, with both a large training set and a limited one, and say it generalizes. BLEU is a translation overlap score.
It is not a measure of whether a chatbot answer is true.
| Task | Figure in the paper | What the figure is |
|---|---|---|
| WMT 2014 English to German | 28.4 BLEU | Translation overlap, above prior results including ensembles |
| WMT 2014 English to French | 41.8 BLEU | Single model, 3.5 days on eight GPUs |
| English constituency parsing | Successful transfer | The architecture applied outside translation |
What the architecture does not carry forward
Later language models reuse attention and the Transformer block. They do not reuse this paper’s training data, its translation objective, or its BLEU scores.
Reading the 2017 experiments tells you what attention replaced and how it did on those tasks. It does not grade a product released afterward.
Common questions
What is Attention Is All You Need about?
It introduces the Transformer, a sequence model that uses attention instead of recurrence and convolution, and reports machine-translation and parsing results.
Does the paper explain why a modern chatbot is accurate?
No. Accuracy for a later model depends on that model’s data and training. This paper’s scores are BLEU on WMT 2014 and a parsing test.
Sources & further reading
How this story was made
Written by Kristian Kostov with AI assistance and checked against the linked sources. Company performance claims are attributed to the company. Analysis reflects AiLookout’s interpretation; we have not independently tested the products discussed. Cover photography is illustrative and does not depict the specific announcement or product.
Our editorial standards





