The architecture that taught machines to see — convolutions, pooling, and how a handful of small filters replace millions of weights.
Module 8 · Lecture notes by Dr. Abdulkarim Albanna
Core Concept Computer Vision ~50 minEverything so far has used fully connected (dense) layers, where every input connects to every neuron. That works for small vectors. It falls apart for images.
Take a modest colour photo of \(224 \times 224\) pixels with 3 colour channels. Flattened, that is one input vector of length \(224 \times 224 \times 3 = 150{,}528\). Wire it into a single hidden layer of just 1000 neurons and you already need:
\[ 150{,}528 \times 1000 + 1000 \approx 1.5 \times 10^{8} \text{ weights} \]
Over 150 million parameters in one layer — before you have learned anything useful. Such a network overfits easily, is slow to train, and needs enormous data.
Worse, flattening throws away spatial structure: pixel \((0,0)\) and its right-hand neighbour end up as unrelated entries in a long vector, and a cat shifted three pixels to the right looks like a completely different input. CNNs fix all of this with three ideas:
A neuron looks at only a small patch of the image (say \(3 \times 3\)), not the whole thing — because meaningful visual features (edges, corners, textures) are local.
The same small set of weights (a filter) is reused at every location. One \(3 \times 3\) edge detector is useful everywhere in the image, so we learn it once and slide it across.
Because the same filter is applied everywhere, a feature is detected no matter where it appears. A cat in the corner fires the same detectors as a cat in the centre.
A convolution slides a small grid of weights — the kernel or filter — across the input. At each position it multiplies the overlapping numbers element-by-element and sums them into a single output value. Sweep over every position and the outputs form a feature map.
For a 2D input \(I\) and a \(k \times k\) kernel \(K\), the value at output position \((i,j)\) is:
\[ S(i,j) = \sum_{m=0}^{k-1} \sum_{n=0}^{k-1} I(i+m,\; j+n)\, K(m,n) \]
(Deep learning libraries implement this cross-correlation form — no kernel flip — and call it convolution. We follow that convention.)
Take a \(5 \times 5\) input that is bright on the left and dark on the right, and a \(3 \times 3\) vertical-edge kernel. We use valid convolution (no padding, stride 1), so the output is \(3 \times 3\).
Input \(I\) (5×5)
| 10 | 10 | 10 | 0 | 0 |
| 10 | 10 | 10 | 0 | 0 |
| 10 | 10 | 10 | 0 | 0 |
| 10 | 10 | 10 | 0 | 0 |
| 10 | 10 | 10 | 0 | 0 |
Kernel \(K\) (3×3)
| 1 | 0 | −1 |
| 1 | 0 | −1 |
| 1 | 0 | −1 |
Output cell \((0,0)\) — overlay the kernel on the top-left \(3\times3\) patch (all values 10):
\[ (10\cdot1 + 10\cdot0 + 10\cdot(-1)) \times 3 \text{ rows} = 0 \]
The patch is uniform, so the \(+1\) and \(-1\) columns cancel: no edge here.
Output cell \((0,1)\) — shift one column right; the patch now straddles the bright/dark boundary (columns \(10,10,0\)):
\[ (10\cdot1 + 10\cdot0 + 0\cdot(-1)) \times 3 \text{ rows} = 10 \times 3 = 30 \]
A large response — the filter has found the vertical edge. Sweeping over all nine positions (every row is identical) gives the feature map:
Feature map (3×3)
| 0 | 30 | 30 |
| 0 | 30 | 30 |
| 0 | 30 | 30 |
The output is near zero over flat regions and spikes where brightness changes horizontally. A single \(3\times3\) filter — nine numbers — became a reusable edge detector. Different kernels detect different things: box/averaging kernels blur, and Laplacian-style kernels sharpen. In a CNN we don't hand-pick these numbers — they are learned by backprop.
Two knobs control how the kernel sweeps the input:
How many pixels the kernel jumps each step. \(S=1\) visits every position; \(S=2\) skips every other one, roughly halving the output size (a cheap way to downsample).
A border of zeros added around the input. Without it, every convolution shrinks the image and the edge pixels are under-sampled. “Valid” means \(P=0\) (output shrinks); “same” padding is chosen so the output matches the input size.
For an input of width \(W\), kernel \(K\), padding \(P\), and stride \(S\), the output width is:
\[ O = \left\lfloor \frac{W - K + 2P}{S} \right\rfloor + 1 \]
(The same formula applies to height. For square inputs and kernels, height and width come out equal.)
| Input \(W\) | Kernel \(K\) | Padding \(P\) | Stride \(S\) | Output \(O\) | Note |
|---|---|---|---|---|---|
| 32 | 5 | 0 | 1 | (32−5+0)/1 + 1 = 28 | valid — shrinks |
| 32 | 5 | 2 | 1 | (32−5+4)/1 + 1 = 32 | “same” — size preserved |
| 32 | 5 | 0 | 2 | ⌊27/2⌋ + 1 = 14 | stride 2 — downsampled |
| 28 | 3 | 1 | 1 | (28−3+2)/1 + 1 = 28 | 3×3 “same” (\(P=1\)) |
A \(3 \times 3\) kernel with \(P=1\) and \(S=1\) always preserves the spatial size. That is why modern architectures (VGG, ResNet) stack many \(3\times3\), \(P=1\) convolutions and downsample only at pooling or strided layers.
Real inputs have depth: a colour image is \(3\) channels (R, G, B). A convolutional filter always spans the full depth of its input. So a “\(3\times3\) filter” on an RGB image is really a \(3 \times 3 \times 3\) block of weights — it slides in 2D (across height and width) but reaches through all channels at each stop, producing one 2D feature map.
To detect many features, a layer uses many filters. \(F\) filters produce \(F\) feature maps, stacked into an output of depth \(F\) — which becomes the input depth of the next layer.
A conv layer with \(C_{\text{in}}\) input channels, \(F\) filters of size \(k \times k\), has:
\[ \text{params} = (C_{\text{in}} \cdot k \cdot k + 1) \cdot F \]
The \(+1\) is one bias per filter. For our first layer — 3 input channels, 32 filters, \(3\times3\):
\[ (3 \cdot 3 \cdot 3 + 1) \cdot 32 = 28 \cdot 32 = 896 \text{ parameters} \]
The dense layer at the top of this page needed ~150,000,000 weights to touch a \(224\times224\times3\) image once. A conv layer with 32 filters sees the same image with 896 — and, thanks to parameter sharing, applies them at every location. That five-orders-of-magnitude saving is the whole point of a CNN.
A deeper layer with 32 input channels and 64 filters of \(3\times3\) has \((3\cdot3\cdot32 + 1)\cdot64 = 289 \cdot 64 = 18{,}496\) parameters — still tiny by dense-layer standards.
After a convolution we often downsample with a pooling layer: slide a small window (usually \(2\times2\), stride 2) over each feature map and replace it with a single summary value. This shrinks the maps, cuts computation, and adds a little robustness to small shifts.
Max pooling keeps the largest value in each window (the strongest activation); average pooling takes the mean. Max pooling is the common default.
A \(4\times4\) feature map, pooled with a \(2\times2\) window at stride 2, yields a \(2\times2\) output — four non-overlapping windows:
Input (4×4)
| 1 | 3 | 2 | 4 |
| 5 | 6 | 1 | 2 |
| 7 | 2 | 3 | 0 |
| 1 | 2 | 4 | 8 |
Max-pooled (2×2)
| 6 | 4 |
| 7 | 8 |
Top-left window \(\{1,3,5,6\}\to 6\); top-right \(\{2,4,1,2\}\to 4\); bottom-left \(\{7,2,1,2\}\to 7\); bottom-right \(\{3,0,4,8\}\to 8\). (Average pooling the top-left window would instead give \((1+3+5+6)/4 = 3.75\).)
Pooling has zero learnable parameters — it is a fixed operation. Many modern networks drop it in favour of strided convolutions (a conv with \(S=2\) downsamples and learns how), but max pooling remains simple, effective, and everywhere in classic architectures.
A convolutional network is a stack of the same building block — CONV → ReLU → POOL — repeated a few times. Early layers learn simple features (edges, colours); deeper layers combine them into textures, parts, and whole objects. After the convolutional stack, the maps are flattened and fed to one or two dense layers ending in a softmax classifier.
Every \(3\times3\) conv uses \(P=1\) (size-preserving); every pool is \(2\times2\), stride 2 (halves H and W). Follow the shape and parameter count layer by layer:
| Layer | Output shape (C×H×W) | Parameters |
|---|---|---|
| Input | 1 × 28 × 28 | 0 |
| Conv 8 filters, 3×3, P=1 | 8 × 28 × 28 | (1·9+1)·8 = 80 |
| ReLU | 8 × 28 × 28 | 0 |
| MaxPool 2×2 | 8 × 14 × 14 | 0 |
| Conv 16 filters, 3×3, P=1 | 16 × 14 × 14 | (8·9+1)·16 = 1168 |
| ReLU | 16 × 14 × 14 | 0 |
| MaxPool 2×2 | 16 × 7 × 7 | 0 |
| Flatten | 784 | 0 |
| Linear 784 → 10 | 10 | 784·10 + 10 = 7850 |
| Total | 10 logits | 9098 |
Just 9,098 parameters classify handwritten digits — and most of them live in the final dense layer, not the convolutions.
The shape trace above translates almost line-for-line into a nn.Module:
A quick sanity check that the layer really has the parameter count we traced by hand:
PyTorch expects images as (N, C, H, W) — batch, channels, height, width. Conv2d(in_channels, out_channels, kernel_size) mirrors our parameter formula exactly: out_channels is the number of filters \(F\), and each filter spans in_channels.
Implement a convolution, a max-pool, and this small CNN from scratch, then check them against PyTorch — instant feedback, reference solutions, no GPU needed.
Open TorchCodeWork each problem by hand before revealing the solution.
A conv layer receives a \(3 \times 64 \times 64\) input (3 channels) and applies 16 filters of size \(5\times5\) with padding \(P=2\), stride \(S=1\). Give the output shape \((C\times H\times W)\) and the number of parameters.
Output size: \(O = \lfloor (64 - 5 + 2\cdot2)/1 \rfloor + 1 = \lfloor 63 \rfloor + 1 = 64\). With 16 filters the output is \(16 \times 64 \times 64\) (a “same” convolution).
Parameters: \((C_{\text{in}}\cdot k\cdot k + 1)\cdot F = (3\cdot5\cdot5 + 1)\cdot 16 = 76 \cdot 16 = 1{,}216\).
Compute the full valid (\(P=0\), \(S=1\)) convolution of this \(4\times4\) input with the \(3\times3\) vertical-edge kernel \(\begin{smallmatrix}1&0&-1\\1&0&-1\\1&0&-1\end{smallmatrix}\). Then state the output shape if instead \(S=2\).
| 1 | 2 | 0 | 1 |
| 0 | 1 | 3 | 2 |
| 2 | 1 | 0 | 1 |
| 1 | 0 | 2 | 3 |
Valid output size: \(\lfloor(4-3)/1\rfloor + 1 = 2\), so a \(2\times2\) map. Each cell is (left column) − (right column), summed over the three rows:
\((0,0)\): \((1-0)+(0-3)+(2-0) = 1-3+2 = 0\)
\((0,1)\): \((2-1)+(1-2)+(1-1) = 1-1+0 = 0\)
\((1,0)\): \((0-2)+(2-0)+(1-2) = -2+2-1 = -2\)
\((1,1)\): \((1-2)+(1-1)+(0-3) = -1+0-3 = -4\)
Feature map: \(\begin{smallmatrix}0&0\\-2&-4\end{smallmatrix}\)
With \(S=2\): \(\lfloor(4-3)/2\rfloor + 1 = 1\), so a single \(1\times1\) output — just the top-left value, \(0\).
Starting from a \(1 \times 32 \times 32\) input, give the output shape after each layer, and the flattened length at the end:
(a) Conv 8 filters \(3\times3\), \(P=1\) → (b) MaxPool \(2\times2\) → (c) Conv 16 filters \(3\times3\), \(P=1\) → (d) MaxPool \(2\times2\) → (e) Flatten.
(a) \(3\times3\), \(P=1\) preserves size → \(8 \times 32 \times 32\)
(b) pool halves H, W → \(8 \times 16 \times 16\)
(c) size-preserving conv, 16 filters → \(16 \times 16 \times 16\)
(d) pool halves again → \(16 \times 8 \times 8\)
(e) flatten → \(16 \cdot 8 \cdot 8 = \) 1024 values.
(Parameter counts, for reference: conv (a) \((1\cdot9+1)\cdot8 = 80\); conv (c) \((8\cdot9+1)\cdot16 = 1168\); pooling adds none.)
Why dense layers can't scale to images, and how CNNs win with local receptive fields, parameter sharing, and translation invariance. You can compute a convolution by hand, use the output-size formula \(O = \lfloor (W-K+2P)/S \rfloor + 1\), count a conv layer's parameters with \((C_{\text{in}} k^2 + 1)F\), apply max pooling, and trace shapes through a full CONV→ReLU→POOL→dense architecture — in maths and in PyTorch.
Next up: Module 9 — Sequence Models. Images are grids; language and time series are sequences. We'll turn to RNNs, LSTMs, and the attention mechanism that powers modern models.
