Course 6, lesson 56 of 100, Ages 12+
Attention
The idea behind transformers
Like I’m 5
When you hear ‘Mia gave Sam her book’, you know ‘her’ means Mia. Attention is how AI works out which words go together.
The big idea
Attention lets a model decide which earlier words matter most for understanding each new word. In 'The cat sat on the mat because it was tired', attention helps link 'it' to 'cat'.
Transformers, the design behind modern LLMs, use attention in every layer, with many 'heads' looking for different kinds of links, like grammar, meaning or who's who in a story.
Examples
- Pronouns: Linking 'she' to the right person in a story.
- Translation: Matching words that mean the same thing across languages.
- Long documents: Connecting a question to the paragraph that answers it.
How it works
- To understand a sentence, you need to know which words connect to which.
- In ‘Mia gave Sam her book’, the AI has to work out who ‘her’ means.
- Attention lets the model focus on the words that matter most. It’s the key idea behind today’s chatbots.
Check your understanding
- What does attention help a model do?
- Options: Focus on the words that matter; Type faster; Use less electricity.
Answer: Focus on the words that matter. Attention links each word to the other words that help explain it. - What does attention help a model do?
- Options: Focus on the words that matter for understanding each word; Make text bold; Count pages.
Answer: Focus on the words that matter for understanding each word. Attention weighs how relevant each word is to the others.
Remember
Attention helps AI connect the words that matter.
Talk about it
In a busy room, how do you listen to just one voice?
Go deeper
Transformers, introduced in 2017, use self-attention: every token weighs how relevant every other token in the context is. Stacking many attention layers lets models track grammar, meaning and references across long passages, and the maths runs efficiently on modern chips.