Module 8 · Hands-on Lab · Dr. Abdulkarim Albanna
Every line of code is explained. Follow the steps in order, or open the whole notebook in Colab.
Each numbered step is one runnable code cell, shown line by line with an explanation under every line, followed by the output you should expect (sample images, the drawn architecture, training logs and predictions). The labs are built to run on Google Colab — click the button above to open the full notebook, or copy each cell with the button and run the steps in order.
Deep Learning course · Dr. Abdulkarim Albanna · albanna-tutorials.com
Feature extraction (Lab 2) froze the backbone. Fine-tuning goes further: we let the pretrained weights keep learning — but gently, with a small learning rate — so the features themselves adapt to our task.
Same *small, public* dataset as Lab 2: Hymenoptera (ants vs. bees). This lets you compare the two strategies directly.
| Backbone weights | Learning rate | When to prefer | |
|---|---|---|---|
| Feature extraction | frozen | normal | very little data / very different-but-generic task |
| Fine-tuning | trainable | small (e.g. 1e-4) | enough data / task close to the source domain |
A common recipe: warm up the head first (like Lab 2), *then* unfreeze and fine-tune everything at a low LR.
import os, zipfile, urllib.requestStandard library: paths, unzip, download.import torchCore PyTorch.import torch.nn as nnLayers and losses.from torchvision import datasets, transforms, modelsDatasets, transforms, and pretrained models.from torch.utils.data import DataLoaderBatches the dataset.import matplotlib.pyplot as pltPlotting.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')GPU if available, otherwise CPU.print('Using device:', device)Show the device.Using device: cpu
URL = 'https://download.pytorch.org/tutorial/hymenoptera_data.zip'URL of the small ants/bees dataset.if not os.path.isdir('hymenoptera_data'):Download only if it isn't already present. urllib.request.urlretrieve(URL, 'hymenoptera_data.zip')Download the zip. with zipfile.ZipFile('hymenoptera_data.zip') as z:Open it. z.extractall('.')Extract into the current folder.mean, std = [0.485, 0.456, 0.406], [0.229, 0.224, 0.225]ImageNet normalisation statistics.data_tf = {A dict of transform pipelines per split. 'train': transforms.Compose([transforms.RandomResizedCrop(224), transforms.RandomHorizontalFlip(),Train: random resized crop to 224 and a horizontal flip… transforms.ToTensor(), transforms.Normalize(mean, std)]),…then convert to a tensor and ImageNet-normalise. 'val': transforms.Compose([transforms.Resize(256), transforms.CenterCrop(224),Val: resize to 256 and centre-crop to 224… transforms.ToTensor(), transforms.Normalize(mean, std)]),…to tensor and normalise (no augmentation).}End the transforms dict.root = 'hymenoptera_data'Dataset root folder.image_sets = {s: datasets.ImageFolder(os.path.join(root, s), data_tf[s]) for s in ['train','val']}Build the train/val datasets from the class subfolders.loaders = {s: DataLoader(image_sets[s], batch_size=32, shuffle=(s=='train'), num_workers=0) for s in ['train','val']}Batch each split; shuffle training only. num_workers=0 for limited CPUs.class_names = image_sets['train'].classesClass names: ['ants','bees'].print('classes:', class_names, '|', {s: len(image_sets[s]) for s in image_sets})Show the classes and per-split image counts.classes: ['ants', 'bees'] | {'train': 244, 'val': 153}This time we do not freeze the parameters. Every weight can update during training.
model = models.resnet18(weights=models.ResNet18_Weights.DEFAULT)Load the ImageNet-pretrained ResNet-18.model.fc = nn.Linear(model.fc.in_features, len(class_names))Swap in a fresh 2-class head.model = model.to(device)Move the model to the device.print('Trainable tensors:', sum(p.requires_grad for p in model.parameters()),Count trainable tensors — note NOTHING is frozen this time… '/ total:', len(list(model.parameters())))…so every parameter (backbone + head) will be fine-tuned.Trainable tensors: 62 / total: 62
The diagram below shows the whole network and how the tensor shape changes at each stage.
A key trick is discriminative learning rates: give the freshly-initialised head a larger LR than the pretrained backbone, which only needs gentle nudging. We also add a step scheduler.
criterion = nn.CrossEntropyLoss()Classification loss.backbone_params = [p for n, p in model.named_parameters() if not n.startswith('fc.')]All pretrained parameters except the new head.head_params = [p for n, p in model.named_parameters() if n.startswith('fc.')]Just the new head's parameters.optimizer = torch.optim.SGD([SGD with two parameter groups — this is the discriminative-learning-rate trick. {'params': backbone_params, 'lr': 1e-4}, # gentle on pretrained featuresBackbone: a tiny LR (1e-4) so the pretrained features barely move. {'params': head_params, 'lr': 1e-3}, # faster on the new headHead: a larger LR (1e-3) since it starts from random.], momentum=0.9)Momentum 0.9 smooths the updates.scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.1)Every 5 epochs, drop the learning rate by 10×.EPOCHS = 6 # fine-tuning is heavier; keep epochs low on CPUNumber of epochs — kept low because fine-tuning is heavier on CPU.def run_epoch(phase):One pass over 'train' or 'val'. model.train() if phase == 'train' else model.eval()Set the matching mode (train vs eval). total_loss = correct = total = 0Running totals. for x, y in loaders[phase]:Iterate the split's batches. x, y = x.to(device), y.to(device)Move to the device. with torch.set_grad_enabled(phase == 'train'):Gradients on only for training. out = model(x)Forward pass. loss = criterion(out, y)Compute the loss. if phase == 'train':Update weights only while training. optimizer.zero_grad(); loss.backward(); optimizer.step()Zero gradients, backpropagate, step the optimiser. total_loss += loss.item() * x.size(0)Accumulate loss. correct += (out.argmax(1) == y).sum().item()Count correct predictions. total += y.size(0)Count samples. return total_loss/total, correct/totalReturn average loss and accuracy.best_val = 0.0Track the best validation accuracy seen so far.for epoch in range(1, EPOCHS+1):Epoch loop. tl, ta = run_epoch('train')Train one epoch. vl, va = run_epoch('val')Validate. scheduler.step()Advance the learning-rate schedule. best_val = max(best_val, va)Update the best validation accuracy. print(f'Epoch {epoch:2d} | train {ta*100:5.1f}% | val {va*100:5.1f}% | lr {scheduler.get_last_lr()[0]:.1e}')Log accuracy and the current learning rate.print(f'Best val accuracy: {best_val*100:.1f}%')Report the best result at the end.Epoch 1 | train 50.4% | val 64.1% | lr 1.0e-04 Epoch 2 | train 66.8% | val 85.6% | lr 1.0e-04 Epoch 3 | train 79.5% | val 91.5% | lr 1.0e-04 Epoch 4 | train 83.2% | val 93.5% | lr 1.0e-04 Epoch 5 | train 86.9% | val 94.1% | lr 1.0e-05 Epoch 6 | train 93.0% | val 94.8% | lr 1.0e-05 Best val accuracy: 94.8%
def denorm(x):Reverse normalisation for display. m = torch.tensor(mean).view(3,1,1); s = torch.tensor(std).view(3,1,1)Reshape mean/std to broadcast over the image. return (x*s + m).clamp(0,1)Undo normalisation and clip to 0–1.imgs, labels = next(iter(loaders['val']))One validation batch.model.eval()Eval mode.with torch.no_grad():No gradients for inference. preds = model(imgs.to(device)).argmax(1).cpu()Predict and move results to the CPU.fig, axes = plt.subplots(1, 6, figsize=(14, 3))A row of 6 sub-plots.for i, ax in enumerate(axes):Loop the 6 images. ax.imshow(denorm(imgs[i]).permute(1,2,0).numpy())Show de-normalised image i. ok = preds[i] == labels[i]True if the prediction is correct. ax.set_title(class_names[preds[i]], color=('green' if ok else 'red')); ax.axis('off')Predicted label, green if right/red if wrong; hide ticks.plt.tight_layout(); plt.show()Render the predictions.
conv1, bn1, layer1) but fine-tune the rest. Does it help or hurt?models.resnet18(weights=None) (no pretraining) with the same recipe. How much does ImageNet pretraining buy you on this tiny dataset?Fine-tuning adapts the pretrained features to your task with a small learning rate and (optionally) discriminative LRs. On small datasets close to the source domain it usually edges out pure feature extraction — at the cost of more compute and a higher risk of overfitting.
# !pip install torch torchvision matplotlib --quietUncomment to install the libraries on a fresh environment.