🧠 Transformer — Questions & Answers

A Transformer is a deep learning architecture introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. (Google).

It processes entire sequences in parallel using a mechanism called self-attention — no recurrence (RNN) or convolution (CNN) needed.

It consists of an Encoder (understands the input) and a Decoder (generates the output).

Self-Attention allows each word to look at all other words in the sentence to understand its context.

Example: In "The cat sat on the mat because it was soft"

• What does "it" refer to? The model needs to attend to "mat" (not "cat").
• Self-Attention computes a score between "it" and every other word.
• "mat" gets the highest score → the model understands "it" = "mat".

Steps:

1. Create Q (Query), K (Key), V (Value) for each word
2. Score = Q × K^T (how relevant is each word?)
3. Scale by √d_k
4. Apply Softmax → probabilities
5. Multiply by V → weighted output

Why three? Separating them allows the model to learn different representations for "what to search for" vs "what to match against" vs "what to output". Using one vector for all three would be much less expressive.

Multi-Head Attention runs multiple self-attention operations in parallel, each with different learned weight matrices.

Why multiple heads? Each head learns different relationships:

• Head 1: might learn grammar (subject-verb)
• Head 2: might learn proximity (nearby words)
• Head 3: might learn meaning (semantic similarity)

Original paper: 8 heads, each with dimension 64 (512/8).

Transformers process all words simultaneously — they have no inherent sense of word order.

Without positional encoding:

"The cat ate the fish"  =  "The fish ate the cat"

Same words, same embeddings — but completely different meanings!

Positional Encoding adds a unique vector to each position so the model knows word order.

Why sin and cos?

Same as Self-Attention, but future positions are masked out (set to -∞ before softmax).

When generating "love" in "I love learning":
  Can see: "I"        ✅
  Can see: "love"     ✅ (itself)
  Cannot see: "learning" ❌ (future — not generated yet)

Why? During generation, the model produces words one at a time. If it could see future words, it would be cheating — copying the answer instead of learning to predict.

The mask sets future positions to -∞ → after softmax they become 0 → ignored.

Example (English → Arabic): Generating "التعلم":

• Q = from Decoder ("what Arabic word am I generating?")
• K, V = from Encoder ("I", "love", "learning")
• Attends most to "learning" → generates "التعلم"

The original input is added directly to the sublayer's output.

Why?

1. Prevents vanishing gradients — gradients flow directly through the skip connection
2. Preserves information — original input is never lost
3. Enables deeper networks — without them, 6+ layers would be very hard to train

Purpose: Normalizes values to mean ≈ 0, variance ≈ 1, which stabilizes training, allows higher learning rates, and reduces sensitivity to initialization.

• Two linear layers with ReLU activation in between
• Inner dimension = 4× model dimension (512 → 2048 → 512)
• Applied to each position independently

Why needed? Self-attention only computes weighted averages (linear). The FFN adds non-linear transformation, allowing the model to learn complex patterns.

During training, instead of feeding the model its own (possibly wrong) predictions, we feed it the correct previous token.

Example: Target = "أنا أحب التعلم"

Step 1: Input = [START]              → predict "أنا"
Step 2: Input = [START, أنا]         → predict "أحب"     (correct "أنا", not model's guess)
Step 3: Input = [START, أنا, أحب]    → predict "التعلم"

Why? Speeds up training and prevents error accumulation (one wrong prediction causing all subsequent ones to be wrong).

Translating "I love learning" → "أنا أحب التعلم":

Encoder:

1. Embed each English word → vectors
2. Add Positional Encoding
3. Pass through 6 encoder layers (Self-Attention → FFN)
4. Output: contextualized representations

Decoder (one word at a time):

1. Start with [START] token
2. Embed + Positional Encoding
3. Masked Self-Attention (can't see future)
4. Cross-Attention (attend to Encoder output)
5. FFN → Linear → Softmax → predict "أنا"
6. Repeat with [START, أنا] → predict "أحب"
7. Repeat → predict "التعلم"
8. Repeat → predict [END] → stop

Training: Parallel ✅ — we have the full target sentence, so we use teacher forcing and process all positions at once (with masking).

Inference (generation): Sequential ❌ — we don't know the next word until we predict it, so we must generate one token at a time.