Attention Is All You Need — The Story of How Robots Learned to Really Listen Attention Is All You Need — The Paper That Rewired How AI Reads Attention Is All You Need
— explained for people who've never seen it before Attention Is All You Need: Annotated Technical Reference

Why Old Robots Were Slow Readers The Problem With Reading One Word at a Time Why did we need a new architecture? Motivating the Elimination of Recurrence and Convolution

Before this paper, computer translators worked like a kid reading a book while covering up every word except one, sliding the cover along one word at a time. They had to finish thinking about word 1 before even starting on word 2. That's called a "recurrent neural network." It worked okay, but it was slow, and by the end of a long sentence it sometimes half-forgot what happened at the beginning — like a game of telephone.

Before 2017, top translation systems used Recurrent Neural Networks (RNNs) — especially LSTMs — often paired with an attention add-on. RNNs process a sentence one word at a time, updating a running "memory" as they go. The catch: word 50 can't be processed until words 1–49 are done, so training can't be parallelized across a sequence. Information from early words also has to survive a long chain of steps to reach later ones, and tends to fade along the way.

Before 2017, the best machine translation systems were built on recurrent neural networks (RNNs) — especially LSTMs and GRUs — often paired with an attention mechanism layered on top. These models read a sentence one word at a time, carrying a "memory" (hidden state) forward from word to word.

Prior state-of-the-art sequence transduction models were dominated by recurrent architectures (LSTM/GRU) with an attached attention mechanism connecting encoder and decoder. Recurrent models factor computation along input/output symbol positions, generating hidden state h_t as a function of h_{t-1} and the input at position t — an inherently sequential dependency.

Why This Matters for You Why This Matters Beyond Translation Why this matters beyond translation Broader Significance

Letting a computer look at everything at once instead of one word at a time meant it could learn from WAY more books and websites, WAY faster. That's a big reason chatbots like Claude got so good, so fast.

Removing the one-word-at-a-time limit didn't just speed up training — it changed what was computationally realistic. Once a whole sequence can be processed in parallel, you can train on far more data with far more compute. That scaling recipe directly led to GPT, BERT, and Claude.

Removing the sequential constraint didn't just make training faster — it changed what was computationally feasible. Once you can process an entire sequence in parallel, you can train on vastly more data with vastly more compute. That scaling recipe is the direct ancestor of GPT, BERT, Claude, and virtually every modern language model.

Eliminating the sequential dependency was the structural precondition for later scaling-law-driven research: parallel training across positions is what made training runs on web-scale corpora with billions to trillions of parameters computationally tractable within realistic wall-clock budgets.

Words You Need to Know First Some Vocabulary Before We Dive In Everything you need before Section 3 Formal Preliminaries

How the Robot Is Built, Like LEGO Blocks Inside the Transformer: The Big Picture How the Transformer is built Architectural Specification: Encoder–Decoder Composition

The Transformer still has a "reader" half and a "writer" half, like older translators — but instead of reading one word at a time, both halves are built from stacked layers of "who should I pay attention to" blocks and simple thinking blocks.

The Transformer keeps the encoder–decoder shape but throws out recurrence entirely, replacing it with stacked layers of self-attention and simple feed-forward networks.

The Transformer keeps the encoder–decoder skeleton but throws out recurrence entirely, replacing it with stacked layers of self-attention and simple feed-forward networks.

The architecture retains the general encoder–decoder structure common to competitive sequence transduction models, but composes each stack entirely from self-attention and point-wise fully connected sub-layers, each wrapped in residual connections and layer normalization.

Input tokens                              Output tokens (shifted right)
     │                                              │
Input Embedding + Positional Encoding      Output Embedding + Positional Encoding
     │                                              │
┌────▼────────────┐                        ┌────────▼─────────┐
│   ENCODER (×N=6) │ ─── encoder output ──▶ │   DECODER (×N=6)  │
│  Multi-Head       │                       │  Masked Multi-Head │
│  Self-Attention    │                       │  Self-Attention    │
│  + Add & Norm       │                       │  + Add & Norm       │
│  Feed-Forward         │                       │  Encoder-Decoder     │
│  + Add & Norm           │                       │  Attention             │
└──────────────────────┘                       │  + Add & Norm             │
                                                │  Feed-Forward               │
                                                │  + Add & Norm                 │
                                                └────────────┬──────────────────┘
                                                              │
                                                    Linear + Softmax
                                                              │
                                                  Output token probabilities
  

The "Who Should I Listen To?" Trick How Attention Actually Works Scaled Dot-Product Attention & Multi-Head Attention Formal Treatment: Scaled Dot-Product and Multi-Head Attention

3.2.1 — The Basic "Who Matters?" Math 3.2.1 — Scaled Dot-Product Attention 3.2.1 — Scaled Dot-Product Attention 3.2.1 — Scaled Dot-Product Attention

Every attention step uses three things made from the input: a Question (Query), a Label (Key), and an Answer (Value). It's like: "What am I looking for?" (Query), "What do I have to offer?" (Key), and "Here's my actual info if you want it" (Value).

Every attention computation involves three learned versions of the input: Queries (Q), Keys (K), and Values (V). A Query is "what am I looking for," a Key is "what do I contain," and a Value is "what I actually offer if you find me relevant."

Every attention computation involves three learned projections of the input: Queries (Q), Keys (K), and Values (V). Intuitively: a Query is "what am I looking for," a Key is "what do I contain," and a Value is "what do I actually offer if you find me relevant."

Each attention operation projects the input into three learned representations — Queries, Keys, Values — via distinct linear maps; the output is a convex combination of Values, weighted by a compatibility function between Query and Key.

3.2.2 — Eight Brains Instead of One 3.2.2 — Multi-Head Attention 3.2.2 — Multi-Head Attention 3.2.2 — Multi-Head Attention

Instead of doing the "who matters" trick just once, the Transformer does it 8 times at once, each time from a slightly different angle, and then combines all 8 answers. It's like asking 8 different friends what they noticed about the same sentence, then putting all their notes together.

Instead of one attention computation over the full 512-number vectors, the model splits Q, K, V into 8 smaller versions (64 numbers each), runs attention on each in parallel, then combines the 8 results back into 512 numbers.

Instead of one attention computation over the full 512-dimensional vectors, the paper linearly projects Q, K, V into h = 8 different, smaller subspaces (each d_k = d_v = 64), runs scaled dot-product attention in each subspace in parallel, then concatenates and projects the results back to 512 dimensions.

Rather than a single d_model-dimensional attention function, Q, K, V are linearly projected h=8 times into d_k=d_v=64-dimensional subspaces; attention is computed in parallel per subspace, outputs concatenated and re-projected via W^O.

3.2.3 — Three Ways This Trick Gets Used 3.2.3 — Three Uses of Attention in This Model 3.2.3 — Three ways attention is used in this model 3.2.3 — Applications of Attention Within the Model

The Other LEGO Pieces Feed-Forward Layers, Embeddings & Position Tags Feed-forward, embeddings, and positional encoding Auxiliary Sub-layer Formulations

3.3 — The "Extra Thinking" Step 3.3 — Position-wise Feed-Forward Networks 3.3 — Position-wise Feed-Forward Networks 3.3 — Position-wise Feed-Forward Networks

After the "who matters" step, every word gets a private moment of extra thinking — two simple math steps with a "throw away negative numbers" rule (ReLU) in between. Every word gets this exact same treatment, just with its own info plugged in.

Applied identically to every position on its own (think: two small calculations with a ReLU "keep only positives" step in between). Input/output size is 512 numbers; the hidden step in between temporarily expands to 2048 numbers, then shrinks back.

Applied identically and independently to every position (think: two 1×1 convolutions with a ReLU in between). Input/output dimension is d_model = 512; the hidden layer in between expands to d_ff = 2048, then projects back down. Parameters differ from layer to layer even though the same shape is reused at every position within a layer.

Equivalent to two kernel-size-1 convolutions with a ReLU nonlinearity; d_model = 512 in/out, d_ff = 2048 inner dimension; parameters are shared across positions within a layer but distinct across layers.

3.4 — Turning Words Into Numbers 3.4 — Embeddings and Softmax 3.4 — Embeddings and Softmax 3.4 — Embeddings and Softmax

Words get turned into lists of 512 numbers (embeddings) that capture meaning. At the very end, the model uses almost the SAME number-lists in reverse to guess the next word — kind of like using the same dictionary to write a word as to look one up.

Input and output tokens become 512-number vectors via learned embeddings. The model reuses (ties) the same weight table for the input embeddings, output embeddings, and the final prediction layer — and scales embedding values up before adding position info.

Input and output tokens are converted to 512-dimensional vectors via learned embeddings. Notably, the paper ties the input embedding matrix, the output embedding matrix, and the pre-softmax linear layer to share the same weights — and multiplies embedding weights by √d_model before adding positional encodings, to balance their relative scale.

Shared weight matrix across input embeddings, output embeddings, and pre-softmax linear transformation (cf. Press & Wolf-style tying); embedding weights scaled by √d_model prior to summation with positional encodings, balancing relative magnitude.

3.5 — Giving Every Word a Position Tag 3.5 — Positional Encoding 3.5 — Positional Encoding 3.5 — Positional Encoding

Since the model looks at the whole sentence at once, it needs a hint about which word came first, second, third, etc. The paper gives each position its own special "wavy pattern" made from sine and cosine waves, added right onto the word's number-list.

Since attention has no inherent sense of order, position info is injected directly into the embeddings using fixed sine/cosine waves of different frequencies, one unique pattern per position.

Since attention has no inherent sense of order, the model injects position information directly into the embeddings using fixed sine/cosine functions of different frequencies:

Absolute position information is injected additively via fixed sinusoidal functions of varying frequency across the embedding dimension:

How They Taught the Robot Training the Model: The Practical Details How the model was actually trained Empirical Training Configuration

Did It Work? YES — Really Well! The Results: Beating the Old Champions Machine translation results Quantitative Results and Ablation Analysis

On the big English-to-German test, the Transformer scored 28.4 out of 100 on the BLEU "translation grade" — beating every earlier robot, even ones that combined several robots together (called "ensembles"), by more than 2 points. On English-to-French it scored 41.8, also a new record — and it trained way faster and cheaper too!

On WMT14 English-to-German, Transformer (big) scores 28.4 BLEU, beating the best previous models — including combinations of several models (ensembles) — by more than 2 BLEU points. On English-to-French, it sets a new single-model record of 41.8 BLEU. And the base model already reaches 27.3 BLEU while training on a fraction of the compute anyone else used.

On the WMT 2014 English-to-German task, the Transformer (big) model reaches 28.4 BLEU, beating the best previously reported models — including ensembles — by more than 2 BLEU, while the base model already reaches 27.3 BLEU at a fraction of typical training cost. On English-to-French, the big model sets a new single-model state of the art of 41.8 BLEU.

Transformer (big) achieves 28.4 BLEU on WMT14 En-De, exceeding prior SOTA (including ensembles) by >2.0 BLEU; base model reaches 27.3 BLEU at ~1-2 orders of magnitude lower training FLOPs than compared ensembles. On En-Fr, Transformer (big) achieves a new single-model SOTA of 41.8 BLEU.

Training cost is measured in floating-point operations; the Transformer base model uses roughly one to two orders of magnitude fewer FLOPs than the ensembled prior state-of-the-art while matching or beating its BLEU score.

6.2 — Testing What Happens If You Change Things 6.2 — Model Variations (Ablations) 6.2 — Model variations (ablations) 6.2 — Model Variations (Ablation Study)

The scientists tried tweaking one setting at a time to see what mattered. Here's what they found, in simple terms:

The authors changed one hyperparameter at a time from the base setup and re-measured translation quality. Big-picture findings:

The authors varied one hyperparameter at a time from the base configuration and re-measured English-to-German BLEU on the development set. The broad, well-established findings:

Single-factor ablations relative to the base configuration, evaluated on the En-De development set:

• Number of "brains" (heads):Number of heads:Number of heads:Attention head count: too few OR too many hurt the score — 8 was just right, proving that having several smaller "brains" beats one big one.both too few heads (just 1) and too many heads (with correspondingly smaller size each) hurt quality — 8 heads was a sweet spot, confirming multi-head beats single-head attention.both too few heads (h=1) and too many heads (with correspondingly smaller d_k, d_v) hurt quality — 8 heads was a sweet spot in their setup, confirming that single-head attention underperforms multi-head.both h=1 and excessively large h (with proportionally reduced d_k, d_v) degraded quality relative to h=8, empirically supporting the multi-subspace hypothesis.
• Query/Key size:Attention key size (d_k):Attention key size (d_k):Key dimensionality (d_k): making Q/K smaller hurt the score, showing that figuring out "who matters" isn't a trivial job.reducing d_k hurt quality, suggesting matching queries to keys isn't trivial and a smarter matching function might help — though dot-product's speed advantage was worth the trade-off.reducing d_k hurt model quality, suggesting that determining compatibility between Q and K isn't trivial and a smarter compatibility function than a dot product might help — though dot-product's speed advantage was worth the trade-off.reducing d_k degraded quality, suggesting Q-K compatibility estimation benefits from higher-dimensional representations — a nontrivial function that dot-product only crudely approximates at low d_k.
• Bigger robot brain:Model size:Model size:Model scale: bigger versions of the robot generally did better — a hint that "bigger is better" would matter a lot for AI later on.bigger models (more layers, bigger internal sizes) generally performed better, foreshadowing the scaling trends that later defined the LLM era.bigger models (more layers, larger d_model, larger d_ff) generally performed better, foreshadowing the scaling trends that later defined the LLM era.monotonic improvement with increased layers/d_model/d_ff, an early informal data point relative to later rigorous scaling-law studies.
• Forgetting on purpose (dropout):Dropout:Dropout:Dropout: removing it made the robot worse at translating new sentences it hadn't seen.removing dropout hurt performance, confirming it meaningfully helps the model generalize to new sentences.removing dropout hurt performance, confirming it meaningfully helps generalization.ablating dropout reduced held-out quality, confirming its regularizing effect.
• Position tags:Positional encoding:Positional encoding:Positional encoding scheme: the wave-pattern version and a "just memorize it" (learned) version worked about the same.learned positional embeddings performed nearly identically to the fixed sine/cosine version, backing up the choice to use waves for their extrapolation benefits.learned positional embeddings performed nearly identically to the fixed sinusoidal version (row E), validating the choice to use sinusoids for their extrapolation benefits.learned embeddings ≈ sinusoidal in measured quality (row E); sinusoidal retained for its hypothesized (not directly tested) length-extrapolation property.

6.3 — Can It Do Other Jobs Too? 6.3 — English Constituency Parsing 6.3 — English constituency parsing 6.3 — English Constituency Parsing

To check if this trick only works for translation, they tried using it to figure out the grammar structure of English sentences (like drawing a tree diagram of a sentence). It did really well — even better than special robots built ONLY for that job!

To test whether the architecture generalizes, the authors used the Transformer to parse English sentences into grammar trees. Despite little task-specific tuning, it performed competitively with — and sometimes beat — dedicated parsers built specifically for this task.

To test generalization beyond translation, the authors applied the Transformer to parsing sentences into grammatical tree structures. Despite little task-specific tuning, the Transformer performed competitively with — and in the semi-supervised setting, better than — previously reported dedicated parsers, showing the architecture wasn't a translation-only trick.

Applied to English constituency parsing with minimal task-specific tuning, the Transformer performed competitively with, and in the semi-supervised regime exceeded, prior dedicated parsing architectures — evidence of architectural generality beyond MT.

Why This Trick Beats the Old Ways Self-Attention vs. RNNs vs. CNNs: The Showdown The complexity argument, side by side Formal Complexity and Path-Length Comparison

Section 4 of the paper compares this new trick to the old ones on three things: how much work per step, how much of that work can happen at the same time, and how many steps it takes for word 1 to "talk to" word 100.

Section 4 formally compares self-attention to recurrent and convolutional layers on three criteria: compute per layer, how much of that compute can run in parallel, and the longest "distance" between any two positions (shorter is better for learning long-range relationships).

Section 4 formally compares self-attention to recurrent and convolutional layers on three criteria: compute per layer, how much of that compute can run in parallel, and the maximum path length between any two positions (shorter = easier to learn long-range dependencies).

§4 formally compares self-attention, recurrent, and convolutional layers along three desiderata: total computational complexity per layer, minimum sequential operations (parallelizability), and maximum path length between any input/output position pair.

n = sequence length, d = representation dimension, k = convolution kernel size, r = restricted attention neighborhood size.

Bonus: because you can literally look at which words a word is "paying attention to," these robots are easier to peek inside and understand than old ones.

As a side benefit, the authors note self-attention models can be more interpretable — you can inspect which words each attention head focuses on, and some heads appear to specialize in grammar or meaning patterns.

As a side benefit, the authors note self-attention models can be more interpretable: individual heads appear to specialize, with some learning to track syntactic structure and others semantic relationships — visible directly by inspecting attention weight matrices.

A noted side benefit is improved interpretability: individual heads exhibit apparent specialization toward syntactic or semantic structure, inspectable directly via attention weight matrices — though see §11 for caveats on how rigorously this claim is substantiated.

What It Still Can't Do Where the Model Struggles What the paper doesn't solve Critical Limitations and Open Problems

What They Hoped to Try Next Where the Authors Said This Could Go Where the authors said this could go Stated Future Directions

The scientists said they wanted to try this trick on pictures, sounds, and videos too, make it work well on really long inputs, and make the "writing" part less one-word-at-a-time. Guess what — all three of those things came true in the years after!

The conclusion explicitly names three future directions: extending the Transformer beyond text to images, audio, and video; investigating local/restricted attention for very large inputs; and making generation less strictly sequential. All three later became real, major areas of research.

The conclusion explicitly plans to: extend the Transformer to problems beyond text involving images, audio, and video; investigate local, restricted attention mechanisms to handle very large inputs and outputs efficiently; and pursue making generation less sequential. All three predictions came true in the following years — Vision Transformers, audio Transformers, and non-autoregressive/parallel decoding research all trace back to these stated directions.

The conclusion enumerates three research directions: (1) extension to non-textual modalities (images, audio, video); (2) local/restricted attention mechanisms for efficient handling of large inputs/outputs; (3) reduced sequentiality in generation. Subsequent work substantiated all three: Vision Transformers, audio/speech Transformers, and non-autoregressive decoding research, respectively.

Where You've Probably Already Used This Real-World Uses Why this paper reshaped the industry Deployment Contexts and Ecosystem Influence

The Good Parts and the Tricky Parts Pros and Cons A balanced assessment Strengths, Weaknesses, and Methodological Caveats

Practice Questions — Show What You Learned! 60 Practice Questions, With Answers 60 practice questions, with answers 60 Interview-Style Questions, With Model Answers

Grouped by level. Click a question to reveal the answer.

Beginner (20)

Intermediate (20)

Advanced (20)

Quiz Time! Test Yourself Test yourself Self-Assessment

Answer the questions below — feedback appears instantly. This is self-graded and doesn't save anywhere.

Word Bank — A to Z A–Z of Every Important Term A–Z of every important term Terminological Reference (A–Z)

If You Remember Five Things... If You Remember Five Things If you remember five things Five Core Takeaways

1. Paying attention beats reading one word at a time.Attention alone is enough.Attention alone is enough.Attention alone suffices. You can build a super smart translator using nothing but the "who matters to whom" trick — no reading order needed, no old-style tricks required. You can build a state-of-the-art sequence model with no recurrence and no convolution — just attention and feed-forward layers, wrapped in residuals and normalization. You can build a state-of-the-art sequence transduction model with no recurrence and no convolution — just attention and feed-forward layers, wrapped in residuals and normalization. A SOTA sequence transduction model can be constructed entirely from attention and position-wise FFN sub-layers, wrapped in residual connections and layer normalization — no recurrence or convolution required.
2. Doing things at the same time (not one-by-one) was the real magic trick.Parallelism was the unlock.Parallelism was the unlock.Parallelizability was the critical enabler. Letting the robot work on many words at once made training way faster — which is a big reason today's chatbots got so smart. Removing sequential dependence between positions made training dramatically faster and, ultimately, made today's massive-scale language models economically trainable. Removing sequential dependence between positions is what made training dramatically faster and, ultimately, made today's massive-scale language models economically trainable. Eliminating sequential inter-position dependence was the structural precondition enabling web-scale training runs underlying modern large language models.
3. Eight little brains beat one big brain.Multi-head attention gives several "angles" at once.Multi-head attention gives the model several "angles" at onceMulti-head attention enables joint attendance across representation subspaces. Splitting attention into 8 smaller versions lets each one notice different kinds of things — one big version would blur everything together. Different heads can specialize in different types of relationships between words, which a single averaged attention head cannot do. — different heads can specialize in different types of relationships between words, which a single averaged attention head cannot do. Different heads specialize toward distinct relational patterns, a capacity a single averaged head structurally lacks.
4. You have to tell it word order on purpose.Position has to be added back in explicitly.Position has to be added back in explicitlyPositional information must be injected explicitly. Since it reads the whole sentence at once, the model has no idea which word came first unless you tell it — using those wavy sine/cosine patterns. via positional encodings, because attention itself has no notion of sequence order. via positional encodings, because attention itself has no notion of sequence order. via fixed sinusoidal positional encodings, since self-attention is inherently permutation-invariant.
5. This one paper is basically the ancestor of all the AI you use today.This paper's ideas power almost everything since.This paper's ideas power almost everything since.Foundational influence on subsequent architectures. GPT-style, BERT-style, and even picture-understanding AIs — including the one writing this guide — all come from the ideas in this exact paper. GPT-style, BERT-style, and vision Transformer architectures — including the model that wrote this guide — are direct descendants of the encoder/decoder blocks described here. GPT-style, BERT-style, and vision Transformer architectures — including the model that wrote this guide — are direct descendants of the encoder/decoder blocks described here. GPT-family, BERT-family, and Vision Transformer architectures are direct descendants of the encoder/decoder sub-layer design formalized here.