Every stage of the Transformer computed by hand on a single English → Arabic sentence: embeddings, positional encoding, attention, the decoder and the final softmax.
Intermediate ~30 min AI ApplicationsFirst step: each word is converted to a vector (list of numbers). We'll assume embedding dimension = 4 (in practice it's 512).
"I" → [0.2, 0.5, 0.1, 0.8] "love" → [0.9, 0.3, 0.7, 0.2] "learning" → [0.4, 0.6, 0.5, 0.9]
These numbers are learned during training — words with similar meanings end up with similar vectors.
Transformers process all words in parallel (not one by one), so they don't know that "I" is the 1st word and "learning" is the 3rd.
Add a unique vector to each position using sin and cos functions:
Where: pos = word position (0, 1, 2, ...) | i = dimension index | d = embedding size (= 4)
This gives every position a unique fingerprint.
Position 0 ("I"):
PE(0,0) = sin(0 / 10000^(0/4)) = sin(0) = 0.00 PE(0,1) = cos(0 / 10000^(0/4)) = cos(0) = 1.00 PE(0,2) = sin(0 / 10000^(2/4)) = sin(0) = 0.00 PE(0,3) = cos(0 / 10000^(2/4)) = cos(0) = 1.00 → PE₀ = [0.00, 1.00, 0.00, 1.00]
Position 1 ("love"):
PE(1,0) = sin(1) = 0.84 PE(1,1) = cos(1) = 0.54 PE(1,2) = sin(0.01) = 0.01 PE(1,3) = cos(0.01) = 1.00 → PE₁ = [0.84, 0.54, 0.01, 1.00]
Position 2 ("learning"):
PE(2,0) = sin(2) = 0.91 PE(2,1) = cos(2) = -0.42 PE(2,2) = sin(0.02) = 0.02 PE(2,3) = cos(0.02) = 1.00 → PE₂ = [0.91, -0.42, 0.02, 1.00]
"I" = [0.2, 0.5, 0.1, 0.8] + [0.00, 1.00, 0.00, 1.00] = [0.20, 1.50, 0.10, 1.80] "love" = [0.9, 0.3, 0.7, 0.2] + [0.84, 0.54, 0.01, 1.00] = [1.74, 0.84, 0.71, 1.20] "learning" = [0.4, 0.6, 0.5, 0.9] + [0.91,-0.42, 0.02, 1.00] = [1.31, 0.18, 0.52, 1.90]
Each word asks: "Which other words are important for understanding my meaning?"
For each word, multiply by three learned weight matrices:
Q = X × Wq (Query — "What am I looking for?") K = X × Wk (Key — "What do I contain?") V = X × Wv (Value — "What information do I give?")
After multiplication (simplified to size 3):
Q K V
"I" [1, 0, 1] [0, 1, 1] [1, 0, 0]
"love" [0, 1, 0] [1, 1, 0] [0, 1, 0]
"learn" [1, 1, 0] [0, 0, 1] [0, 0, 1]
a) Multiply Q × KT (every query with every key):
Example: scores for word "I" with all words:
"I" with "I": Q_I · K_I = [1,0,1]·[0,1,1] = 0+0+1 = 1 "I" with "love": Q_I · K_love = [1,0,1]·[1,1,0] = 1+0+0 = 1 "I" with "learn": Q_I · K_learn = [1,0,1]·[0,0,1] = 0+0+1 = 1
All scores:
"I" "love" "learn"
"I" → [ 1, 1, 1 ]
"love" → [ 1, 1, 0 ]
"learn" → [ 0, 2, 1 ]
b) Divide by √dk (scaling):
d_k = 3 → √3 ≈ 1.73
"I" "love" "learn"
"I" → [0.58, 0.58, 0.58]
"love" → [0.58, 0.58, 0.00]
"learn" → [0.00, 1.15, 0.58]
c) Apply Softmax (convert to probabilities):
Each row becomes probabilities that sum to 1:
"I" "love" "learn"
"I" → [0.33, 0.33, 0.33] ← attends equally to all words
"love" → [0.39, 0.39, 0.22] ← attends more to "I" and itself
"learn" → [0.19, 0.59, 0.22] ← attends most to "love"!
d) Multiply by V (get weighted output):
output_I = 0.33×[1,0,0] + 0.33×[0,1,0] + 0.33×[0,0,1] = [0.33, 0.33, 0.33] output_love = 0.39×[1,0,0] + 0.39×[0,1,0] + 0.22×[0,0,1] = [0.39, 0.39, 0.22] output_learn = 0.19×[1,0,0] + 0.59×[0,1,0] + 0.22×[0,0,1] = [0.19, 0.59, 0.22]
Instead of one attention, run 8 in parallel — each one called a head.
Each head learns a different type of relationship:
1. For each head: run Self-Attention independently (with different weights) head₁ = Attention(Q×W₁Q, K×W₁K, V×W₁V) head₂ = Attention(Q×W₂Q, K×W₂K, V×W₂V) ... head₈ = Attention(Q×W₈Q, K×W₈K, V×W₈V) 2. Concatenate all results: MultiHead = Concat(head₁, head₂, ..., head₈) × Wᴼ
Head 1 output for "learn": [0.19, 0.59, 0.22] (learned semantic relations) Head 2 output for "learn": [0.45, 0.10, 0.45] (learned positional relations) Concat: [0.19, 0.59, 0.22, 0.45, 0.10, 0.45] × Wᴼ → [0.30, 0.40, 0.30] (final combined output)
The original input is added back to the sublayer's output.
Where: μ = mean | σ² = variance | γ, β = learnable parameters
x (original input for "learn"): [1.31, 0.18, 0.52]
attention output: [0.19, 0.59, 0.22]
After Residual: [1.31+0.19, 0.18+0.59, 0.52+0.22] = [1.50, 0.77, 0.74]
μ = (1.50 + 0.77 + 0.74) / 3 = 1.00
σ² = ((0.50)² + (-0.23)² + (-0.26)²) / 3 = 0.12
√(σ² + ε) ≈ 0.35
LayerNorm = [(1.50-1.00)/0.35, (0.77-1.00)/0.35, (0.74-1.00)/0.35]
= [1.43, -0.66, -0.74]
After attention, each position passes through a small neural network:
input: [1.43, -0.66, -0.74] After W₁ + b₁: [2.1, -0.3, 1.5, -1.2] After ReLU (max 0): [2.1, 0.0, 1.5, 0.0] ← negatives become 0 After W₂ + b₂: [0.8, 0.5, 0.3] ← back to original size
The Decoder works one word at a time. Assume we're generating the 3rd word and have:
Decoder inputs so far: ["أنا", "أحب", ???]
Same as Self-Attention — but with a mask:
Scores before mask:
"أنا" "أحب" ???
"أنا" → [0.8, 0.5, 0.3]
"أحب" → [0.6, 0.9, 0.4]
??? → [0.3, 0.7, 0.8]
Apply Mask (lower triangle only):
"أنا" "أحب" ???
"أنا" → [0.8, -∞, -∞ ] ← can only see itself
"أحب" → [0.6, 0.9, -∞ ] ← can see "أنا" and itself
??? → [0.3, 0.7, 0.8] ← can see everything before it
After Softmax:
"أنا" → [1.00, 0.00, 0.00]
"أحب" → [0.43, 0.57, 0.00]
??? → [0.17, 0.37, 0.46]
Here the Decoder looks at the Encoder's output:
Q = from Decoder (the word we're generating)
K = from Encoder ("I", "love", "learning")
V = from Encoder ("I", "love", "learning")
Q for ???: [0.5, 0.8, 0.3] Scores with each English word: ??? with "I": 0.5×0.2 + 0.8×0.5 + 0.3×0.1 = 0.53 ??? with "love": 0.5×0.9 + 0.8×0.3 + 0.3×0.7 = 0.90 ??? with "learning": 0.5×0.4 + 0.8×0.6 + 0.3×0.5 = 0.83 After softmax: [0.22, 0.40, 0.38]
Final step — the Decoder output passes through:
1. Linear Layer: projects vector to vocabulary size (e.g., 50,000 words) 2. Softmax: converts to probabilities Decoder output for ???: [0.25, 0.60, 0.15] After Linear (simplified to 5 words): ["أنا": 0.02, "أحب": 0.05, "التعلم": 0.85, "كتاب": 0.03, "بيت": 0.05] Highest probability → "التعلم" (0.85) ✅
┌─────────────────────────── ENCODER ───────────────────────────┐ │ │ │ "I love learning" │ │ ↓ │ │ [Input Embedding] → vector for each word │ │ ↓ │ │ [+ Positional Encoding] → add position information │ │ ↓ │ │ [Multi-Head Self-Attention] → each word attends to all others │ │ ↓ │ │ [+ Residual + LayerNorm] │ │ ↓ │ │ [Feed-Forward Network] → neural net per position │ │ ↓ │ │ [+ Residual + LayerNorm] │ │ ↓ │ │ ══► Encoder Output (sent to Decoder) │ │ │ │ × 6 layers │ └────────────────────────────────────────────────────────────────┘ ┌─────────────────────────── DECODER ───────────────────────────┐ │ │ │ "أنا أحب" (words generated so far) │ │ ↓ │ │ [Output Embedding + Positional Encoding] │ │ ↓ │ │ [Masked Multi-Head Self-Attention] → can't see future │ │ ↓ │ │ [+ Residual + LayerNorm] │ │ ↓ │ │ [Cross-Attention] → Q from here, K+V from Encoder │ │ ↓ │ │ [+ Residual + LayerNorm] │ │ ↓ │ │ [Feed-Forward Network] │ │ ↓ │ │ [+ Residual + LayerNorm] │ │ ↓ │ │ [Linear → Softmax] → probability over vocabulary │ │ ↓ │ │ ══► "التعلم" ← highest probability word │ │ │ │ × 6 layers │ └────────────────────────────────────────────────────────────────┘
| Parameter | Value |
|---|---|
| Encoder layers (original paper) | 6 |
| Decoder layers | 6 |
| Embedding size (d_model) | 512 |
| Number of heads | 8 |
| Size per head (d_k) | 512 / 8 = 64 |
| Feed-Forward inner dimension | 2048 |
| Original paper | "Attention Is All You Need" (2017) |
| Authors | Vaswani et al. (Google) |
| Optimizer | Adam (with warmup + decay schedule) |
| Training loss | Cross-Entropy |
| Decoder training trick | Teacher Forcing (feed correct previous token) |
| Type | Examples | Best For | How It Works |
|---|---|---|---|
| Encoder-only | BERT, RoBERTa, DistilBERT | Classification, NER, understanding | Bidirectional — sees all words at once |
| Decoder-only | GPT, GPT-2, GPT-3, GPT-4 | Text generation | Autoregressive — predicts next token left-to-right |
| Encoder-Decoder | BART, T5, Original Transformer | Translation, summarization | Full sequence-to-sequence |
Prepared by Dr. Abdulkarim Albanna — AI Applications Course