Module 3: Convolutional networks

A colour image of 224 × 224 pixels is 224 × 224 × 3 = 150,528 numbers. Flattened, it can be fed to the multilayer perceptron of Module 02, but at this size it fails twice: on cost and on statistics.

The statistical failure is worse. Each weight belongs to one input position, so the layer must learn separately that an edge at the top left and the same edge at the bottom right are the same thing, from examples that show an edge in each place. Two properties of images remove both problems.

Locality

A pixel’s meaning depends far more on its neighbours than on distant pixels: edges, strokes and textures are relations between pixels a few positions apart. This can be measured.

import numpy as np
from sklearn.datasets import load_digits

X = load_digits().data                       # (1797, 64), grey levels 0..16
keep = np.flatnonzero(X.std(axis=0) > 0.5)   # skip pixels that are almost always blank
R = np.corrcoef(X[:, keep], rowvar=False)    # correlation of every pair of pixels
row, col = np.divmod(keep, 8)                # position of each kept pixel on the grid
dist = np.hypot(row[:, None] - row, col[:, None] - col)
iu = np.triu_indices(len(keep), k=1)         # each pair once
r, d = R[iu], dist[iu]
for lo, hi in [(0, 1), (1, 1.5), (1.5, 2), (2, 3), (3, 4.5), (4.5, 12)]:
    sel = (d > lo) & (d <= hi)
    print(f"distance ({lo}, {hi}]: mean r = {r[sel].mean():+.2f}  ({sel.sum()} pairs)")

distance (0, 1]: mean r = +0.42  (88 pairs)
distance (1, 1.5]: mean r = +0.17  (76 pairs)
distance (1.5, 2]: mean r = -0.04  (72 pairs)
distance (2, 3]: mean r = -0.06  (236 pairs)
distance (3, 4.5]: mean r = -0.04  (406 pairs)
distance (4.5, 12]: mean r = -0.03  (448 pairs)

Stationarity

The second property is stationarity: the statistics of small patches are the same everywhere in the image. An edge looks the same at the top left as at the bottom right, so a detector useful at one position is useful at every position. That licenses weight sharing: one small set of weights, applied everywhere.

One layer, three parameter counts

Take a layer from a 224 × 224 × 3 image to a 224 × 224 × 64 output (3,211,264 units, 64 at each of 50,176 positions) and impose the two assumptions one at a time; Figure 3.1 draws the three cases for one output unit.

Weight sharing also pools statistical strength. A shared weight acts at every position, so the chain rule sums its gradient over all 50,176 positions of every training image, where an untied weight hears only from its own. Neighbouring positions are correlated, so one image is not worth 50,176 independent examples, but it is worth far more than one.

Equivariance and invariance

Write T_{\boldsymbol{\tau}} for the translation that shifts an image by \boldsymbol{\tau} = (\tau_1, \tau_2) pixels, (T_{\boldsymbol{\tau}} X)_{i,j} = X_{i-\tau_1,\, j-\tau_2}. A function f of images is

• translation equivariant if f(T_{\boldsymbol{\tau}} X) = T_{\boldsymbol{\tau}} f(X): shifting the input shifts the output by the same amount;
• translation invariant if f(T_{\boldsymbol{\tau}} X) = f(X): shifting the input leaves the output unchanged.

A convolution is equivariant (Section 2 proves it); pooling and global averaging give approximate invariance (Section 5). The task decides which is wanted. A label such as “contains a crack” does not move when the crack moves, so a classifier should be invariant; a segmentation mask or a detection box must move with the object, so those networks should be equivariant.

The inductive bias, and where it does not hold

A convolution is a dense layer whose weight matrix is constrained: most entries are zero (locality) and the rest are tied (weight sharing), as Section 2 shows. It represents fewer functions than a dense layer with the same input and output, and that is the point: the functions it gives up are ones images do not need, and every parameter it lacks need not be estimated from data. An assumption built into an architecture in this way is an inductive bias. Lab 2 tests it from both sides: shuffling the 64 pixels of every digit with one fixed permutation leaves an MLP’s accuracy unchanged, since it never knew which pixels were neighbours, and lowers a CNN’s from about 98.4% to about 96%.

The bias pays on grids with local, stationary statistics: images, audio and sensor signals, spectra, CT and microscopy volumes (Section 13). It does not help on tabular data, whose columns (age, pressure, alloy grade) have no neighbours and an arbitrary order.

For a single-channel image X \in \R^{H \times W}, a kernel K \in \R^{k \times k} and a bias b, a convolution layer computes

Indices start at 0, and (i, j) is the top-left corner of the window the kernel covers: lay the kernel over the window, multiply the overlapping numbers, add the products and the bias, slide one place and repeat. The window fits at H - k + 1 positions per axis, so without padding a 3 × 3 kernel turns a 5 × 5 image into a 3 × 3 output; Section 3 deals with the border.

Cross-correlation or convolution

Strictly, the operation above is a cross-correlation, written K \star X in this module when the distinction matters. The convolution of mathematics and signal processing flips the kernel:

Here the output index sits at the window’s bottom-right corner. Move it to the top-left (i \to i + k - 1, j \to j + k - 1) and substitute u' = k - 1 - u, v' = k - 1 - v: the sum becomes \sum_{u',v'} K_{k-1-u',\,k-1-v'}\, X_{i+u',\, j+v'}, a cross-correlation with the kernel rotated by 180°. The two differ only by the flip K_{u,v} \to K_{k-1-u,\,k-1-v}. For a learned kernel this is immaterial, because the network simply learns the flipped kernel, and every deep-learning framework computes the cross-correlation and calls it convolution. Three consequences:

• scipy.signal.convolve2d flips the kernel, so it disagrees with torch.nn.functional.conv2d, while correlate2d agrees, as does convolve2d given K[::-1, ::-1] (Lab 1).
• A kernel unchanged by the rotation, such as a blur, gives identical results either way.
• An antisymmetric kernel changes sign. Rotating the edge kernel above negates it, so the true convolution of the example is -Y, with rows (1, 4, 0), (3, 2, −4) and (1, 2, −1).

Hand-designed kernels

Before kernels were learned they were designed, and the classic designs show what a 3 × 3 kernel can do. One rule organises them. On a constant region, X_{i,j} = c, the output is b + c \sum_{u,v} K_{u,v}: kernels whose weights sum to 1 reproduce flat regions and change only the structure on top of them, while kernels whose weights sum to 0 output zero on flat regions and respond only to change.

The filters a network learns in its first layer look much like these, oriented edge detectors and blobs (Lab 2, step 7, plots the eight learned by its digit CNN, and Section 14 discusses them); deeper layers combine them into detectors nobody would design by hand.

Translation equivariance, proved

Section 1 claimed that convolution is translation equivariant. With (T_{\boldsymbol{\tau}} X)_{i,j} = X_{i-\tau_1,\, j-\tau_2} the proof takes two lines:

The first line substitutes the shifted image; the second reads the same sum as the output at (i - \tau_1, j - \tau_2). The bias is the same everywhere and passes through. The proof assumes that every X_{i+u-\tau_1,\, j+v-\tau_2} exists, which holds on an infinite grid and with circular padding, where the image wraps around like a torus. With zero padding, the pixels a shift pushes in at one border are zeros rather than those pushed out at the other, so equivariance holds exactly away from the borders and fails near them; Lab 1 measures both cases.

The convolution as a matrix

A convolution is linear in X, so it is a matrix product. In one dimension, a length-5 input and the kernel (1, 2, 3) give three outputs, and writing each window’s weights as a row gives

Each row is the previous one shifted one place right: \mathbf{M} is a banded Toeplitz matrix, constant along its diagonals. In two dimensions, with the image flattened row by row, the matrix is doubly block-Toeplitz: Toeplitz blocks, one per kernel row, arranged in a Toeplitz pattern. This is the precise sense of Section 1’s claim: a convolution layer is a dense layer whose matrix has k^2 nonzeros per row and the same k^2 values in every row.

The backward pass, and the transposed convolution

If \mathbf{y} = \mathbf{M}\mathbf{x} and the gradient \mathbf{g} = \partial \mathcal{L}/\partial \mathbf{y} arrives from the layer above, the chain rule (Module 02, Section 3) gives

Column m of \mathbf{M} holds the weights with which x_m entered each output, so \mathbf{M}^\top scatters every g_i back through the kernel onto the inputs its window read. The result is a “full” convolution of \mathbf{g} with the kernel, over every overlap including those that hang over the ends. Applying the transpose of a convolution’s matrix is a transposed convolution; Section 12 runs it forwards to upsample. The weight gradient follows because K_{u,v} enters Y_{i,j} only through the term K_{u,v} X_{i+u,j+v}:

a cross-correlation of the input with the output gradient. The backward pass is therefore two more convolutions, each with as many multiply-accumulates as the forward one: about twice the forward cost, as for the dense layers of Module 02.

import numpy as np

w = np.array([1.0, 2.0, 3.0])                # kernel
x = np.array([1.0, 0.0, 2.0, 1.0, 3.0])      # input of length 5
M = np.zeros((3, 5))
for i in range(3):
    M[i, i:i + 3] = w                        # row i: the kernel shifted i places
print(M @ x, np.correlate(x, w, "valid"))    # forward pass, two ways
g = np.array([1.0, -1.0, 2.0])               # dL/dy arriving from above
print(M.T @ g, np.convolve(g, w, "full"))    # dL/dx: the transposed convolution
print(np.correlate(x, g, "valid"))           # dL/dw: input correlated with dL/dy

[ 7.  7. 13.] [ 7.  7. 13.]
[1. 1. 3. 1. 6.] [1. 1. 3. 1. 6.]
[5. 0. 7.]

Every architecture diagram is a list of shapes, each following from one formula, and a recurrence says how much of the image each unit can see. Both rest on three settings. Padding p adds p cells at each end of an axis; stride s is the step between windows; dilation d spaces the kernel’s taps d apart, so a 3 × 3 kernel with d = 2 reads every second pixel of a 5 × 5 area. Dilation keeps the k^2 weights while the span grows to d(k-1) + 1.

The output size

Count the window positions along one axis. After padding there are H + 2p positions, numbered 0 to H + 2p - 1. A window starting at t reads t, t + d, \dots, t + (k-1)d, so it fits if t + d(k-1) \le H + 2p - 1. Windows start at t = 0, s, 2s, \dots, and the number of multiples of s in [0,\, H + 2p - d(k-1) - 1] is

At d = 1 this is \lfloor (H + 2p - k)/s \rfloor + 1, and with p = 0, s = 1 it is Section 2’s H - k + 1. Widths follow the same formula.

“Same” padding. With s = 1 and p = d(k-1)/2 the numerator is H - 1, so H_{\text{out}} = H. This needs d(k-1) even, which at d = 1 means an odd kernel (3 × 3 with p = 1, 5 × 5 with p = 2). An even kernel cannot be padded symmetrically; PyTorch’s padding="same" then puts the extra zero on the right and bottom, and refuses strides above 1.

Stride 2. With k = 3, p = 1, s = 2 the formula gives \lfloor (H - 1)/2 \rfloor + 1: H/2 for even H, (H + 1)/2 for odd. “Stride 2 halves the size” is exact for even sizes; 7 becomes 4.

Padding modes. Zero padding is the default; reflect (mirror), replicate (repeat the edge pixel) and circular (wrap around) fill the border from the image instead. The choice matters: with zero padding a kernel near the border sees an artificial dark frame, so the network can learn where the border is, and through it the absolute position of a feature, which an equivariant layer is not supposed to know.

The receptive field

The receptive field of a unit is the set of input positions that can influence it; along one axis its size is r. Track two quantities through the layers: r_l, the field of a unit of layer l, and the jump \Delta_l, the distance in input pixels between adjacent units of layer l, starting from r_0 = 1 and \Delta_0 = 1. A unit of layer l reads k_l units of layer l - 1 spaced d_l apart, so its outermost inputs lie (k_l - 1)\,d_l\,\Delta_{l-1} input pixels apart, each bringing a field of r_{l-1}; its own stride multiplies the spacing of its units:

Pooling layers enter the same way, with their window as k_l, their stride as s_l and d_l = 1. Padding does not change r; it decides only how far the field hangs over the border.

Small kernels, stacked. Each 3 × 3 layer with stride 1 adds 2, so n of them reach r = 2n + 1. With C channels in and out, two stacked 3 × 3 layers see what one 5 × 5 sees with 18C^2 weights instead of 25C^2 (28% fewer), and three see what one 7 × 7 sees with 27C^2 instead of 49C^2 (45% fewer), with a nonlinearity between each layer. This is the whole design of VGG (Section 7).

Downsampling multiplies the jump. After one stride-2 layer every further 3 × 3 adds 4 pixels instead of 2; after five, 64. That is how a network reaches whole-image context in a few dozen layers.

Dilation grows the field exponentially. A 3 × 3 stack with d = 1, 2, 4, 8 adds 2d per layer, r = 3, 7, 15, 31, where an undilated stack needs 15 layers to reach 31, and no resolution is lost. Dilations sharing a common factor leave gaps: three layers at d = 2 read only every second input, a pattern called gridding, which starting at d = 1 avoids. Figure 3.4 sets the three ways of growing the field side by side: stacking, striding and dilating.

The effective receptive field

The theoretical field says which pixels can influence a unit, not how much. Count the paths: in three stacked 3-tap layers the centre input reaches the top unit along 7 paths and each outermost input along 1, the per-axis counts 1, 3, 6, 7, 6, 3, 1 being the triple self-convolution of (1, 1, 1). With n layers the counts are the kernel’s n-fold self-convolution, which by the central limit theorem tends to a Gaussian whose width grows as \sqrt{n}, while the field grows as n. Luo et al. (2016) measured this effective receptive field, where the gradient of an output unit actually lands, and found it roughly Gaussian and much smaller than the theoretical one. For SmallResNet’s last stage (Section 8), theoretical field 49 × 49, Lab 3 measures about 44% of the input-gradient mass inside the central 9 × 9 and 89% inside 25 × 25; half of it lies within about 11 × 11, a twentieth of the theoretical area. Step 5 of that lab plots the gradient map with the theoretical window and the contours that enclose 50% and 90% of the mass.

This matters when objects are large. A network whose effective field is smaller than an object can judge it only by local texture, and ImageNet-trained CNNs show the bias: given images whose shape says one class and whose texture another, they mostly answer with the texture (Geirhos et al. 2019). The recurrence makes computing the field a five-minute task; Lab 3 shows how to look at the effective one.

Real inputs have several channels (red, green and blue, or the fields of a simulation), and every layer after the first produces many. The single-channel layer generalises by summing over input channels:

The weight tensor has shape (C_{\text{out}}, C_{\text{in}}, k, k), and PyTorch stores activations as (B, C, H, W), channels first; TensorFlow defaults to channels last, (B, H, W, C). Each output channel c has one filter K_{c,\cdot,\cdot,\cdot} of shape C_{\text{in}} \times k \times k, which looks at a k \times k patch through every input channel at once and sums the results into one number. The map a filter produces as it slides is one feature map; a layer is C_{\text{out}} filters producing C_{\text{out}} feature maps, the next layer’s channels (Figure 3.5).

The layer as one matrix multiply

For a fixed output position, the sum over (c', u, v) is a dot product between the filter, flattened to length C_{\text{in}}k^2, and the window, flattened the same way. Collect every window as one column of a matrix \mathbf{X}_{\text{col}} of shape C_{\text{in}}k^2 \times H_{\text{out}}W_{\text{out}}, a step called im2col (“image to columns”), and stack the C_{\text{out}} flattened filters as the rows of \mathbf{W}. The whole layer is then one matrix product,

plus the bias, reshaped to (C_{\text{out}}, H_{\text{out}}, W_{\text{out}}). This is how CPU libraries and many GPU kernels compute convolutions, because matrix multiplication is the most heavily optimised routine in numerical computing; FFT-based and Winograd algorithms exist for some shapes. The price is memory: at stride 1 each input value is copied into up to k^2 columns, so a 3 × 3 layer’s column matrix is about nine times its input. PyTorch exposes im2col as F.unfold:

import torch
import torch.nn.functional as F

torch.manual_seed(0)
x = torch.randn(1, 64, 56, 56)                  # one image with 64 channels
w = torch.randn(128, 64, 3, 3)                  # 128 filters, each 64 x 3 x 3
cols = F.unfold(x, kernel_size=3, padding=1)    # im2col: one column per position
y = (w.reshape(128, -1) @ cols).reshape(1, 128, 56, 56)
print(tuple(cols.shape), cols.numel() / x.numel())
print(torch.allclose(y, F.conv2d(x, w, padding=1), atol=1e-3))

(1, 576, 3136) 9.0
True

Counting parameters, MACs and FLOPs

Read the counts off the equation. Each output channel has k^2 C_{\text{in}} weights and a bias, so a layer has k^2 C_{\text{in}} C_{\text{out}} + C_{\text{out}} parameters. Each output value costs k^2 C_{\text{in}} multiply-accumulates (MACs), one multiplication and one addition each, and there are C_{\text{out}} H_{\text{out}} W_{\text{out}} output values:

The factor 2 is the counting behind the “2N FLOPs per forward pass” of Module 02, Section 2.

Where the parameters and the compute live

Parameters, k^2 C_{\text{in}}C_{\text{out}} + C_{\text{out}}, do not depend on H and W; compute is proportional to H_{\text{out}}W_{\text{out}}. Early layers, at high resolution with few channels, dominate the compute. Late layers, with many channels at low resolution, and above all dense heads, dominate the parameters.

# VGG-16 on a 224x224 image, counted from its layer list
cfg = [64, 64, "M", 128, 128, "M", 256, 256, 256, "M",
       512, 512, 512, "M", 512, 512, 512, "M"]       # "M" is a 2x2 max pool
H, C, params, macs = 224, 3, 0, 0
for v in cfg:
    if v == "M":
        H //= 2                                    # pooling halves the map
        continue
    params += 3 * 3 * C * v + v                    # weights + biases
    macs += 3 * 3 * C * v * H * H                  # padding 1: output is H x H
    C = v
conv_params, conv_macs = params, macs
for n_in, n_out in [(512 * 7 * 7, 4096), (4096, 4096), (4096, 1000)]:
    params += n_in * n_out + n_out                 # fc6, fc7, fc8
    macs += n_in * n_out
print(f"parameters {params:,}, of which dense {params - conv_params:,}")
print(f"MACs: convolutions {conv_macs / 1e9:.2f} G, dense {(macs - conv_macs) / 1e9:.2f} G")

parameters 138,357,544, of which dense 123,642,856
MACs: convolutions 15.35 G, dense 0.12 G

Memory, the backward pass and the bias

Activation memory. An output of shape (C, H, W) in float32 takes 4CHW bytes per image, and training keeps it until the backward pass has used it, which is why training memory scales with depth times batch (Module 02, Section 3). The 64 → 128 layer’s output is 128 \times 56 \times 56 \times 4 = 1{,}605{,}632 bytes, 1.53 MiB per image and 392 MiB (411 MB) for a batch of 256, against 0.28 MiB for the layer’s parameters.

Backward cost. The backward pass computes two convolutions with as many MACs as the forward one (Section 2), the input gradient and the weight gradient, so a training step costs about three times the forward FLOPs: about 1.39 GFLOPs per image for the layer above, slightly less for a first layer, whose input gradient is not needed.

Bias and batch normalisation. A convolution followed directly by batch normalisation needs no bias: batch normalisation subtracts each channel’s mean over the batch, which removes any constant b_c added before it, and then adds its own learned shift. Hence bias=False in the convolutions of Section 8’s code; the normalisation itself is Module 02, Section 10’s.

A classifier must turn a 224 × 224 map into one label, and it should not pay full-resolution prices all the way. Both needs are met by reducing resolution as the network deepens.

Max pooling takes the maximum over each 2 × 2 window with stride 2: it halves H and W, keeps each window’s strongest response and has no parameters. Since \max(x_1, \dots, x_4) has derivative 1 with respect to its largest input and 0 with respect to the others, the backward pass sends each window’s gradient entirely to the position that held the maximum (PyTorch picks the first on a tie). Average pooling averages each window and passes a quarter of the gradient to each position.

Global average pooling (GAP) averages each channel’s entire map to one number, turning C \times H \times W into C. It replaces the giant dense layers that ended early networks and lets a network accept any input size.

Strided convolution also halves the map, but its weights decide what to keep, where a max pool always keeps the largest value, and modern networks often downsample this way. ResNet moves between stages with stride-2 convolutions, a strided 1 × 1 on the shortcut (Section 8); ConvNeXt uses separate downsampling layers, a normalisation and a 2 × 2 convolution with stride 2 (Section 9).

Invariance is only approximate

Pooling is often said to make a network translation invariant. It gives only a little invariance, and only for some shifts. A stride-2 operation is equivariant only to shifts by multiples of 2: a shift of two input pixels moves the output by one cell, but a shift of one pixel changes which pixels share a window.

This is aliasing, as in signal processing: subsampling a signal that contains frequencies above half the new sampling rate folds them into spurious low frequencies, so the result depends on where the samples fall. The classical remedy is a low-pass filter before subsampling. Zhang (2019) blurs before each strided operation (“anti-aliased” pooling) and reports more shift-stable outputs; Azulay and Weiss (2019) document modern CNNs whose predictions change under one-pixel shifts.

Lab 2 measures the effect on 8 × 8 digits. A CNN with two 2 × 2 max pools and a flatten-and-dense head scores about 98.4% on the centred test images and about 66% when they move one pixel to the right; an MLP of the same size falls from about 97% to 44%. On digits placed at random on a 16 × 16 canvas, the same convolutions with a third layer reach about 93% with GAP and 89% with a flatten head (Figure 3.6 contrasts the two heads). Equivariant layers do not by themselves make an invariant classifier: the head and the training data decide.

Why downsample

Each halving of H and W cuts the FLOPs of every later layer by 4 (Section 4). It doubles the jump, so later layers grow the receptive field twice as fast (Section 3). And the usual convention of halving H and W while doubling C keeps the compute per layer roughly constant across stages, because doubling C multiplies C_{\text{in}} C_{\text{out}} by 4 while halving the map divides H_{\text{out}} W_{\text{out}} by 4.

A standard k \times k convolution does two jobs at once. It filters in space, over a k \times k window, and it mixes channels, because every output channel reads every input channel. Its cost per output position, k^2 C_{\text{in}} C_{\text{out}} multiply-accumulates (Section 4), is the product of the two. The efficient architectures of Sections 7 to 9 are built by doing the two jobs separately, or by doing the expensive one on fewer channels. This section derives what each piece costs and what it gives up.

The 1 × 1 convolution

With k = 1 the window is a single pixel and the multi-channel equation of Section 4 becomes

At every position the vector of C_{\text{in}} channel values is multiplied by the same C_{\text{out}} \times C_{\text{in}} matrix \mathbf{W}. A 1 × 1 convolution is a dense layer applied independently at every pixel, with C_{\text{in}} C_{\text{out}} + C_{\text{out}} parameters, and it never looks at a neighbour. It does two useful things: it changes the number of channels cheaply, and, followed by a ReLU, it adds a nonlinearity per pixel. Lin et al. (2014) followed every spatial convolution with a stack of such layers, a small MLP at each pixel (“Network in Network”); the same paper introduced global average pooling.

Bottlenecks

At a fixed output size a k \times k convolution costs k^2 C_{\text{in}} C_{\text{out}} H_{\text{out}} W_{\text{out}} MACs, proportional to the product of its two channel counts. Divide both counts by r and the cost falls by r^2. A bottleneck exploits this: a 1 × 1 convolution reduces the channels before the expensive k \times k one and, where the width must be restored, a second 1 × 1 expands them after it. Inception used bottlenecks to afford 5 × 5 branches; ResNet-50 and its deeper relatives use one in every block (Section 8).

Grouped and depthwise convolutions

A grouped convolution splits the C_{\text{in}} input channels into g groups and gives each group its own C_{\text{out}}/g filters, which read only that group’s channels. Each filter now spans C_{\text{in}}/g channels, so the layer has k^2 C_{\text{in}} C_{\text{out}}/g weights and costs g times less than the full layer. AlexNet’s grouped layers were a hardware split: half of the filters lived on each of two GPUs and read only the maps on their own GPU (Section 7). ResNeXt (Xie et al. 2017) made the number of groups a design dimension in its own right.

At the extreme, g = C_{\text{in}} = C_{\text{out}} = C, every group is a single channel. This is the depthwise convolution: one k \times k filter per channel, k^2 C weights, and no mixing of channels at all.

Depthwise-separable convolution

A depthwise-separable convolution does the two jobs one after the other: a depthwise k \times k convolution filters each channel in space, then a pointwise (1 × 1) convolution mixes the channels (Figure 3.8). Put any stride in the depthwise layer, so that both versions produce the same output size, and count the MACs per output position:

The weights obey the same ratio, because without biases each layer’s weight count equals its MACs per position. On a wide layer the first term is negligible, and a 3 × 3 separable layer costs about a ninth of the standard one: the saving is a factor of roughly k^2.

What the factorisation gives up

Compose the two layers. The depthwise filter D_{c'} turns channel c' into a filtered map, and the pointwise weights P_{c,c'} add those maps into output channel c (biases omitted):

A separable layer is therefore a standard convolution whose kernel is forced into the form K_{c,c',u,v} = P_{c,c'} D_{c',u,v}. On input channel c', every output channel applies a scaled copy of the same spatial filter D_{c'}, where a standard layer can give each of its C_{\text{in}} C_{\text{out}} channel pairs its own spatial pattern. MobileNet’s Table 4 (Howard et al. 2017) puts a price on the restriction. On ImageNet the depthwise-separable network reaches 70.6% top-1 with 569 million multiply-adds and 4.2 million parameters; the same network with full convolutions reaches 71.7% with 4,866 million and 29.3 million. About one point of accuracy buys 8.5 times less compute.

FLOPs are not run time

A FLOP count measures arithmetic, and a depthwise layer does little arithmetic per byte it moves. On the 64-channel 56 × 56 map above, the depthwise 3 × 3 performs 9 \times 64 \times 3{,}136 = 1.8 million MACs while reading and writing 1.53 MiB of activations, about 1.1 MACs per byte. The standard 3 × 3 layer performs 231 million MACs on 2.58 MiB of activations and weights, about 85 per byte. A GPU needs tens of operations per byte or more to keep its arithmetic units busy (the roofline of Module 10, Section 2), so depthwise layers run far below peak and a separable network’s measured speed-up is smaller than its FLOP ratio. Measure on the hardware you will deploy on; Lab 3’s extension asks why its separable network does not train 6.6 times faster.

The history of convolutional architectures reads best as a sequence of single ideas, each added to the network before it. Every number in their papers can be checked with the tools of Sections 3 to 6, and checking them is the quickest way to understand a design.

Reading an architecture table

An architecture table lists, for every layer, the output shape, the kernel and stride, and often the parameters and multiply-adds. Check each row: the output size with Section 3’s formula, the parameters and MACs with Section 4’s. A row that does not follow from the formulas is a misprint or hides something worth knowing: a partial connection, a bias convention, or multiply-adds reported as FLOPs.

LeNet-5: the pattern

LeNet-5 (LeCun et al. 1998) fixed the pattern every later network elaborates: convolution, subsampling, convolution, subsampling, dense layers. It reads a 32 × 32 image in which the digit occupies at most the central 20 × 20, so that strokes near the edge of a digit still fall in the middle of the top detectors’ fields. C1 applies six 5 × 5 filters (6 maps of 28 × 28). S2 subsamples each map to 14 × 14: a unit adds the four values of a 2 × 2 window, multiplies the sum by one trainable coefficient per map, adds one trainable bias per map and applies a sigmoid. C3 applies 5 × 5 filters to give 16 maps of 10 × 10, but each map reads only 3, 4 or 6 of the six S2 maps, following a connection table; the authors give two reasons, to keep the number of connections in bounds and to break the symmetry between maps. S4 subsamples to 16 maps of 5 × 5. C5 is a 5 × 5 convolution with 120 filters on a 5 × 5 input, so each of its maps is 1 × 1: in effect a dense layer. F6 is a dense layer of 84 units, followed by 10 output units. A cheque-reading system built on this network was deployed commercially in the mid-1990s.

A modern “LeNet-5” connects C3 fully, pools without weights and ends in a linear 10-way layer. It has 156 + 2{,}416 + 48{,}120 + 10{,}164 + 850 = 61{,}706 parameters, the number most tutorials print:

import torch, torch.nn as nn

lenet5 = nn.Sequential(                               # full C3, pooling without weights
    nn.Conv2d(1, 6, 5), nn.Tanh(), nn.AvgPool2d(2),    # C1, S2 -> 6 x 14 x 14
    nn.Conv2d(6, 16, 5), nn.Tanh(), nn.AvgPool2d(2),   # C3, S4 -> 16 x 5 x 5
    nn.Conv2d(16, 120, 5), nn.Tanh(), nn.Flatten(),    # C5 -> 120
    nn.Linear(120, 84), nn.Tanh(), nn.Linear(84, 10))  # F6, output
sizes = [sum(p.numel() for p in m.parameters()) for m in lenet5]
print([n for n in sizes if n > 0], sum(sizes))
print(lenet5(torch.zeros(1, 1, 32, 32)).shape)

[156, 2416, 48120, 10164, 850] 61706
torch.Size([1, 10])

AlexNet: scale, ReLU and regularisation

AlexNet (Krizhevsky et al. 2012) kept the pattern and scaled it to ImageNet: five convolutional layers and three dense layers, about 60 million parameters, trained on two GPUs, whose split survives as its grouped convolutions (Section 6). Most of the parameters sit in the dense layers: the first maps the 6 \times 6 \times 256 = 9{,}216 features of the last pooled map to 4,096 units, 37.7 million weights. Four ingredients made it train and generalise:

• ReLU instead of tanh. Their Figure 1 shows a four-layer CNN reaching 25% training error on CIFAR-10 six times faster with ReLUs than with tanh units.
• Dropout with probability 0.5 in the first two dense layers (Module 02, Section 11).
• Augmentation: random 224 × 224 crops of 256 × 256 images, horizontal flips, and colour jitter along the principal components of the RGB values.
• Overlapping max pooling: 3 × 3 windows with stride 2.

In the ILSVRC-2012 competition its top-5 test error was 15.3%, against 26.2% for the next entry. It started the field’s move to deep networks.

VGG: only 3 × 3

VGG (Simonyan and Zisserman 2015) asked what depth alone buys. It uses only 3 × 3 convolutions with stride 1 and padding 1, which keep the map size, and 2 × 2 max pools with stride 2, which halve it; the channels double from 64 to 512 across the pools, and the networks it compares have 11 to 19 weight layers, of which VGG-16 and VGG-19 are the ones still in use. Section 3 gave the argument: two stacked 3 × 3 layers see a 5 × 5 window with 18 instead of 25 weights per channel pair, and with a nonlinearity between them. VGG-16 has 138 million parameters, 89% of them in its three dense layers, and costs about 15.5 G multiply-adds per 224 × 224 image (Section 4). It placed second to GoogLeNet in the 2014 classification task, and because it is simple it is still a common backbone and feature extractor.

Inception: parallel branches

GoogLeNet (Szegedy et al. 2015) asked which kernel size to use and answered “all of them”. An inception module runs 1 × 1, 3 × 3 and 5 × 5 convolutions and a 3 × 3 max pool in parallel and concatenates their outputs along the channel axis (Figure 3.7). Section 6’s 1 × 1 bottlenecks keep it affordable: module (3a) as a whole has 163,328 weights and costs 128 M MACs, of which its reduced 5 × 5 branch takes 12.4 M. The network is 22 layers deep, counting layers with parameters, and the paper states that it uses 12 times fewer parameters than AlexNet. It won the ILSVRC-2014 classification task with 6.67% top-5 error. Two auxiliary classifiers, small heads on intermediate layers whose losses were added with weight 0.3 during training, were meant to push gradient into the early layers: a fix for depth from before residual connections.

How the headline numbers are defined

The ImageNet numbers in this module come from the ILSVRC classification task (Russakovsky et al. 2015): 1.2 million training images in 1,000 classes, 50,000 validation images and 100,000 test images. Top-5 error counts a prediction as correct when the true class is among the five highest-scoring classes; top-1 requires it to be the highest. The two cannot be compared with each other. Competition entries were often ensembles of several networks evaluated on many crops, so a competition result is not the accuracy of one network on one view.

The thread

Each network is its predecessor plus one idea: LeNet’s pattern, AlexNet’s scale and regularisation, VGG’s uniform small kernels, Inception’s cheap width. Top-5 error fell from 15.3% in 2012 to 6.67% in 2014, and the parameter count did not have to rise to get there: VGG-16 has 138 million, GoogLeNet a few million. Depth was the obstacle left, and Section 8 removes it.

By 2014 depth was paying: VGG reached 19 layers and GoogLeNet 22, with auxiliary classifiers to get the gradient through. Going much deeper made networks worse, and not because they overfitted.

The degradation problem

He et al. (2016) trained plain networks, stacks of 3 × 3 convolutions with batch norm and ReLU, with 20 and with 56 layers on CIFAR-10. Their Figure 1 shows the 56-layer network with the higher error, on the test set and also on the training set. Higher training error rules out overfitting: the deeper network does not even fit the data it sees. Nor is it short of capacity. Copy the trained 20-layer network into the first 20 layers of the deeper one and make the other 36 compute the identity: the result has exactly the shallower network’s training error. A solution at least as good exists, and the optimiser does not find it. This is the degradation problem, an optimisation failure. Lab 3 reproduces it on 8 × 8 digits and plots the training curves.

The residual block

The fix is to make the identity the default. A residual block computes

where the branch F is a few convolution layers. The block learns only the change to its input; if nothing needs to change, F = 0 and the input passes through. In the basic block F is conv 3 × 3, batch norm, ReLU, conv 3 × 3, batch norm, and the original design applies a ReLU after the addition. When the block changes the shape, by a stride or a new channel count, the shortcut becomes a strided 1 × 1 convolution with batch norm (a projection), so that the two terms can be added (Figure 3.9, left). The code below is that block and a four-stage network built from it, with its arithmetic printed.

import torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):                      # conv-BN-ReLU twice, with a residual path
    def __init__(self, cin, cout, stride=1):
        super().__init__()
        self.c1 = nn.Conv2d(cin, cout, 3, stride, 1, bias=False)
        self.b1 = nn.BatchNorm2d(cout)
        self.c2 = nn.Conv2d(cout, cout, 3, 1, 1, bias=False)
        self.b2 = nn.BatchNorm2d(cout)
        self.skip = nn.Identity() if stride == 1 and cin == cout else nn.Sequential(
            nn.Conv2d(cin, cout, 1, stride, bias=False), nn.BatchNorm2d(cout))
    def forward(self, x):
        y = F.relu(self.b1(self.c1(x)))
        y = self.b2(self.c2(y))
        return F.relu(y + self.skip(x))

class SmallResNet(nn.Module):
    def __init__(self, classes=10):
        super().__init__()
        self.stem = nn.Sequential(
            nn.Conv2d(3, 32, 3, 1, 1, bias=False), nn.BatchNorm2d(32), nn.ReLU())
        self.stages = nn.Sequential(
            Block(32, 32), Block(32, 64, 2), Block(64, 128, 2), Block(128, 256, 2))
        self.head = nn.Linear(256, classes)
    def forward(self, x):
        x = self.stages(self.stem(x))
        return self.head(x.mean(dim=(2, 3)))   # global average pooling

net = SmallResNet()
print(sum(p.numel() for p in net.parameters()))          # 1,228,970 parameters
x = torch.zeros(1, 3, 32, 32)
print(net(x).shape)                                       # torch.Size([1, 10])

# output-size arithmetic for one layer
H, k, p, s = 32, 3, 1, 2
print((H + 2 * p - k) // s + 1)                           # 16

count = lambda m: sum(p.numel() for p in m.parameters())
print(count(net.stem), [count(b) for b in net.stages], count(net.head))

1228970
torch.Size([1, 10])
16
928 [18560, 57728, 230144, 919040] 2570

Making the identity easy to find

A plain layer computes the identity only if its weights learn it, and with batch norm and ReLU in the way that is a hard target. A residual block reaches the identity by making F small. Initialise the last layer of each branch near zero (in practice set the scale \gamma of the branch’s last batch norm to 0, as Goyal et al. 2017 do) and every block starts as the identity: a deep ResNet begins as a shallow network and grows into its depth as its branches learn.

Why the gradient survives

Take the pre-activation block of He et al.'s second paper (2016b), in which F is BN, ReLU and convolution twice and nothing follows the addition (Figure 3.9, right). Write the block equation for i = l, \dots, L-1 and add the equations; the intermediate terms cancel in pairs:

Every later representation is an earlier one plus a sum of branch outputs. Differentiate with the chain rule, writing gradients as row vectors so that they multiply Jacobians from the left:

The identity term delivers the top gradient to block l unchanged, however deep the network. For the total to vanish, the second term would have to cancel the identity exactly, for every example. A plain network \mathbf{h}_{i+1} = G_i(\mathbf{h}_i) has instead a product of Jacobians, \partial \mathbf{h}_L / \partial \mathbf{h}_l = \mathbf{J}_{L-1} \cdots \mathbf{J}_l (Module 02, Section 3), which shrinks or grows geometrically when its factors are a little smaller or larger than 1. Block by block, the residual Jacobian is \mathbf{I} + \partial F_i/\partial \mathbf{h}_i; multiplying out the L - l factors gives a sum over all 2^{L-l} subsets of branches, so the gradient travels along every path, and the shortest path crosses no branch at all (Figure 3.10).

The original block’s ReLU after the addition multiplies each block’s Jacobian by that ReLU’s 0/1 derivative, so a unit switched off at any block closes its own highway (Exercise 6 works this out). That gate is why He et al. moved to the pre-activation form.

Lab 3 measures this on 8 × 8 digits with networks of 16-channel 3 × 3 layers. At initialisation and without normalisation, the gradient reaching the stem of a plain network is 1.4 \times 10^{-5} at 9 layers and 9 \times 10^{-10} at 19, and at 55 it underflows to 0 in float32; the residual network’s stays between 0.02 and 0.09 at every depth. The plain network loses a factor of about 0.4 per layer, close to the 1/\sqrt{6} = 0.41 predicted for PyTorch’s default initialisation, whose weight variance is a sixth of He’s (Module 02, Section 6). Batch norm does not cure the plain network: with it, the 55-layer stem gradient is about 190, exploding instead, against 0.8 with shortcuts. Trained for 12 epochs with batch norm, the plain 55-layer network’s training loss is still about 2.1, barely below chance, while the residual one reaches about 0.3; at 19 layers the plain network already trails, about 0.4 against 0.04 with shortcuts. The deep residual network is held back by the learning rate, not the architecture: at a peak rate of 0.03 or 0.02 instead of 0.05 its loss falls to 0.04–0.08 over three seeds, while the plain networks stay far behind at every rate tried.

The bottleneck block

For ResNet-50, 101 and 152 the branch is 1 × 1, 3 × 3, 1 × 1: a 256-channel input is reduced to 64 channels, filtered by a 3 × 3 at 64 channels and expanded back to 256 (Section 6’s bottleneck).

Results

He et al. trained ResNets up to 152 layers on ImageNet, and an ensemble of them won the ILSVRC-2015 classification task with 3.57% top-5 error. On CIFAR-10, ResNet-110, with 1.7 million parameters, reached 6.43% test error (ResNet-20, 0.27 million: 8.75%). A 1,202-layer network reached a similar training error to the 110-layer one but tested worse, 7.93%: that is overfitting, 19.4 million parameters on 50,000 training images, not degradation. Depth had stopped being an obstacle.

DenseNet: concatenate instead of add

DenseNet (Huang et al. 2017) connects every layer of a block to every earlier one: layer \ell receives the concatenation of all earlier outputs along the channel axis and adds k new channels, the growth rate. In DenseNet-121, with k = 32 and 64 channels entering the first block, that block’s sixth layer reads 64 + 5 \cdot 32 = 224 channels and the block outputs 64 + 6 \cdot 32 = 256. DenseNet-121 has about 8 million parameters and strong accuracy per parameter; the price is activation memory, since every concatenated map is kept for the backward pass. Concatenation (DenseNet, and U-Net in Section 12) keeps features separate and grows the width; addition (ResNet) keeps the width fixed and merges them.

The same form, \mathbf{x} + F(\mathbf{x}), wraps every attention and feed-forward sublayer of the transformer (Module 06, Section 5), and the derivation above is why transformers can be stacked dozens of layers deep.

Once ResNet had made depth trainable, the work moved to cost: how much accuracy a network delivers per multiply-add, and how much of a reported gain is due to the architecture at all.

MobileNet

MobileNet (Howard et al. 2017) uses depthwise-separable blocks throughout (Section 6) and adds two knobs for trading accuracy against cost. The width multiplier \alpha scales every channel count by \alpha; the pointwise layers’ cost is proportional to C_{\text{in}} C_{\text{out}}, so it falls by about \alpha^2. The resolution multiplier \rho scales the input, and with it every map’s height and width, so the cost falls by \rho^2. The baseline MobileNet-224 has 4.2 million parameters and costs 569 million multiply-adds at 70.6% ImageNet top-1. It runs on a phone.

MobileNetV2: the inverted residual

MobileNetV2 (Sandler et al. 2018) adds shortcuts, arranged the other way round from ResNet’s bottleneck. Its inverted residual block expands a narrow input with a 1 × 1 convolution (by a factor of 6), filters at the wide width with a depthwise 3 × 3, and projects back to the narrow width with a 1 × 1 convolution that has no ReLU after it; the shortcut joins the narrow ends. The wide middle is affordable because it is depthwise. The projection is left linear because a ReLU on a narrow representation zeroes part of it, and what it zeroes there is lost: the paper calls this a linear bottleneck.

EfficientNet: compound scaling

A network can be enlarged in three ways: deeper, wider or at higher resolution. EfficientNet (Tan and Le 2019) scales all three together by a fixed ratio. A convolution costs k^2 C_{\text{in}} C_{\text{out}} H W MACs. Multiplying the width by w multiplies both channel counts, and the cost by w^2; multiplying the resolution by r multiplies H and W, and the cost by r^2; multiplying the depth by d multiplies the number of layers, and the cost by d. So the FLOPs scale as d\,w^2 r^2. Compound scaling sets

and the constraint \alpha\beta^2\gamma^2 \approx 2 makes each unit of \phi double the FLOPs. A grid search on the small baseline B0 at \phi = 1 chose \alpha = 1.2, \beta = 1.1, \gamma = 1.15; B1 to B7 scale B0 with larger \phi. EfficientNet-B0 has 5.3 million parameters and costs 0.39 billion FLOPs. The paper does not define the term, but the same table lists ResNet-152 at 11 billion, the ResNet paper’s multiply-add count, so these FLOPs are multiply-adds. B0 reaches 76.3% top-1 in Table 2 of the ICML paper. A later arXiv revision reports 77.1%, so quote the version you cite.

ConvNeXt: separating the recipe from the architecture

ConvNeXt (Liu et al. 2022) set out to measure how much of the vision transformer’s advantage lay in its training recipe rather than its architecture. The authors took ResNet-50 and first trained it with the transformer era’s recipe: AdamW, 300 epochs, mixup, CutMix, RandAugment, random erasing, stochastic depth and label smoothing. Then they changed the architecture one step at a time, measuring ImageNet-1k top-1 after each step. Table 10, in Appendix C of the arXiv version, lists every step (Figure 2 plots the same values); rounded to 0.1:

The last value is 81.97 \pm 0.06. Two steps lose accuracy, and the step after each pays it back: the depthwise convolution cuts compute, which the widening then spends; moving the depthwise layer up makes room for its larger kernel. The finished ConvNeXt-T (Table 1) reaches 82.1% with 28.6 million parameters (Table 9; Table 1 prints 29M) and 4.5 G multiply-adds, against 81.3% for the Swin-T transformer at 28 million and 4.5 G.

The lesson for reading papers: compare architectures only under the same training recipe. An improvement over a baseline trained with an older recipe may be mostly recipe.

Where the vision transformer fits

The vision transformer (Dosovitskiy et al. 2021) cuts the image into 16 × 16 patches, embeds each patch as a token with what amounts to a 16 × 16 convolution of stride 16, and runs a transformer over the tokens, with no other convolution. With less built in than a CNN it trails comparable ResNets when trained on ImageNet-1k alone, and matches or beats them after pretraining on far larger datasets: Lab 2’s inductive-bias lesson at scale, which Module 06, Section 8 takes up in full.

Where CNNs remain the default

As of 2026, convolutional networks remain a default choice where labelled data are scarce, where inputs are large (high-resolution images, 3D volumes), where latency or power is constrained, and where the output is dense, one value per pixel or voxel. Hybrids that combine convolutions with attention are common.

An engineering image dataset is usually small: a few hundred labelled micrographs, a few thousand inspection photographs. This section gives the practical recipe for training a CNN on such data, with measurements that show when each ingredient helps and when it does not. Optimisers, learning-rate schedules and the overfit-one-batch test are Module 02’s; for a CNN, train with random crops, flips where they are valid and a cosine schedule.

Augmentation

Augmentation applies random transformations to each training image every time it is used. It is the most effective regulariser for images, with one rule: every transform must leave the label true. A vertical flip of a 6 is a 9, and a mirrored 2 is not a digit at all.

• Geometric: random crops (pad and crop back, or crop a random region and resize it), flips, small rotations and scalings, and elastic deformations, which the U-Net paper singles out as the key augmentation for microscopy with few annotated images. Rotate only where the label is rotation-invariant.
• Photometric: brightness, contrast, colour jitter, noise and blur.
• For segmentation the mask receives the same geometric transform as the image, resampled with nearest-neighbour interpolation so that it stays a set of labels; photometric transforms touch the image only.

Mixing methods blend two training examples. Mixup (Zhang et al. 2018) forms

with one-hot labels \mathbf{y} and a around 0.2, which puts most \lambda near 0 or 1. CutMix (Yun et al. 2019) pastes a random rectangle of \mathbf{x}_j into \mathbf{x}_i and mixes the labels in proportion to the area pasted. Both make the targets soft.

Augmentation must match deployment

A transform can preserve every label and still hurt. Augmentation tells the network which variation to ignore; if the test data never contain that variation, the network spends capacity on it for nothing. Try-this items 2 and 3 of Lab 2 measure both sides on digits; runs made when this module was prepared gave these numbers:

• On the centred 8 × 8 digits, with 20 training images per class, random shifts of ±1 pixel and rotations of ±10° lower validation accuracy from about 0.90 to about 0.83. A one-pixel shift is an eighth of the image, the centred test set contains no such shifts, and the 1,898-parameter network cannot absorb the extra variation.
• On the 16 × 16 canvas, where digits really do appear at different positions, re-placing each training digit at a fresh random position every epoch raises test accuracy from 0.36 to 0.66 at 5 images per class, and from 0.76 to 0.90 at 20.

Augment with the variation the deployment data contain, and check the choice on a validation set drawn like those data.

Normalising the input

Subtract the training set’s per-channel mean and divide by its per-channel standard deviation, so that the first layer sees inputs of order 1. With a pretrained backbone, use its statistics, not your dataset’s: for torchvision’s ImageNet models, RGB values scaled to [0, 1], then mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224, 0.225). The channel order, the value range and the resize convention are part of the same contract.

Batch normalisation in CNNs

In a CNN, batch norm computes its statistics per channel over the batch and both spatial axes, (B, H, W); conv, BN, ReLU was ResNet’s recipe and remains common. Module 02, Section 10 owns the mechanics. Below about 16 examples per batch the statistics become unreliable, and group normalisation (Wu and He 2018), which normalises each example over groups of channels, or layer norm replaces it. What counts is the number of independent examples, not the number of values: two 64 × 64 images give 8,192 values per channel, but from two examples. When fine-tuning with small batches, keep a pretrained backbone’s batch-norm layers in eval mode. requires_grad=False stops their weights from changing, but not their running means and variances, which update from your batches whenever the layer is in train mode.

Transfer learning

A backbone pretrained on a large, generic dataset has learned edge, texture and shape detectors that transfer. Replace its final layer with one for your labels and fine-tune: first the new layer alone at a normal learning rate, then the whole network at about a tenth of it. The options, from the fewest trained parameters to the most (Figure 3.11 draws them with the accuracies of Lab 4):

1. Linear probe: freeze the whole backbone and train only a new linear head on its features.
2. Keep early layers: copy the first blocks, frozen, and train the rest from random weights.
3. Fine-tuning: start every layer from its pretrained weights and train them all, with discriminative learning rates (smaller for earlier layers) as a refinement.
4. Training from scratch.

The choice depends on two things: how many labelled images there are, and how close the source data are to the target. A probe needs the least data but works only if the backbone’s last features suit the new task. Yosinski et al. (2014) found early layers general and later layers specific to the source task, so a distant or narrow source should contribute only its early layers.

Pretraining helps less when the target set is large. He, Girshick and Dollár (2019) trained detectors on COCO from scratch and matched ImageNet-pretrained ones given longer schedules; Raghu et al. (2019) found small accuracy gains from ImageNet pretraining on large medical-imaging datasets, with faster convergence. With a few hundred labelled images, pretraining remains the default starting point, including for scientific images that look nothing like the pretraining set.

A small-data checklist

• Start from a pretrained backbone, or at least from pretrained early layers.
• Use strong augmentation that preserves the label and matches deployment.
• Keep the head small.
• Stop early on a validation set split by specimen, not by image (Module 01, Section 10).
• Report results over several seeds and folds.

An inspection system must say where, and how many: every pore on a weld radiograph. Object detection outputs a set of triples (box, class, score), of any size, empty included. A box is written by its corners (x_1, y_1, x_2, y_2) or by its centre and size (x, y, w, h). The backbone is the CNN of the previous sections; what is new is box geometry, turning dense predictions into a list, and scoring the list.

Intersection over union

Two boxes are compared by their intersection over union,

The intersection of two axis-aligned boxes is a box whose lower corner is the larger of the two lower corners and whose upper corner is the smaller of the two upper corners; its width is \max\big(0, \min(x_2^A, x_2^B) - \max(x_1^A, x_1^B)\big), and likewise its height. PASCAL VOC counts a detection as a true positive if its IoU with a not-yet-matched ground-truth box of the same class exceeds 0.5; COCO repeats the evaluation at the ten thresholds 0.50, 0.55, \dots, 0.95 and averages, which rewards tight boxes.

Anchors and the box parameterisation

A network cannot regress an unknown number of boxes directly. Faster R-CNN (Ren et al. 2015) places k anchors, reference boxes of several scales and aspect ratios, at every feature-map position: 3 scales × 3 ratios, k = 9 (Figure 3.12, left). For each anchor the network predicts class scores and four offsets:

Decoding inverts this: x = x_a + t_x w_a, w = w_a e^{t_w}. Dividing by the anchor’s size makes the shifts scale-free, and the logarithm keeps every decoded width positive and makes size errors relative. A 1000 × 600 image at stride 16 gives about 60 × 40 = 2,400 positions, and so about 20,000 anchors, the figure Ren et al. give.

The loss

A classification term over the sampled anchors plus a box term over those matched to objects:

with targets t^* computed as above and \operatorname{smooth}_{L_1}(e) = 0.5e^2 for |e| < 1, |e| - 0.5 otherwise: quadratic near zero, linear for outliers. Later detectors use IoU-based box losses, which optimise what evaluation measures.

Two stages or one

A two-stage detector proposes, then classifies. In Faster R-CNN a region proposal network scores every anchor as object or background and keeps a few hundred to about two thousand proposals. RoI pooling max-pools each proposal’s region of the feature map into a fixed 7 × 7 grid, and a second head classifies the region and refines its box. Mask R-CNN (He et al. 2017) replaced it by RoI align, which samples bilinearly instead of rounding the region’s edges. Two stages are accurate but slower. Figure 3.13 sets the two pipelines, and the set-prediction detector described below, side by side.

A one-stage detector predicts class and box at every cell or anchor in one pass. YOLO (Redmon et al. 2016) divides the image into a 7 × 7 grid; each cell predicts 2 boxes of 5 numbers (four coordinates and a confidence) and 20 class probabilities, an output of 7 \times 7 \times (2 \cdot 5 + 20) = 7 \times 7 \times 30. SSD (Liu et al. 2016) places anchors on several feature maps. Of about 20,000 anchors nearly all are easy background, and their summed cross-entropy swamps the few objects. RetinaNet (Lin et al. 2017) introduced the focal loss; with p_t the predicted probability of the true class,

cross-entropy scaled by (1 - p_t)^\gamma, with \gamma = 2 and \alpha = 0.25 for the object class in the paper.

Non-maximum suppression

A detector fires on many anchors around each object. Non-maximum suppression (NMS) keeps one (Figure 3.12, right): sort the boxes by score; keep the highest; delete every remaining box of the same class whose IoU with it exceeds a threshold (0.5 is common); repeat with the best survivor.

NMS fails in crowds: if two true boxes overlap by more than the threshold, the lower-scoring one is deleted. Soft-NMS (Bodla et al. 2017) instead lowers overlapping boxes’ scores by a factor that shrinks as the IoU grows.

Average precision

Sort a class’s detections by score; at each rank, precision is the fraction of detections so far that are true positives and recall the fraction of ground-truth objects found. The interpolated precision at recall r is the highest precision at any recall \ge r, and average precision (AP) is the area under that curve. mAP averages AP over classes, and COCO’s AP also over the ten IoU thresholds.

Set prediction

DETR (Carion et al. 2020) puts a transformer (Module 06) on a CNN backbone and outputs a fixed number of (box, class) predictions, most of them “no object”. Inside the loss the Hungarian algorithm matches predictions one-to-one to the ground truth at the lowest total cost, so each object is claimed by exactly one prediction and duplicates are trained away: no anchors and no NMS. Mask R-CNN’s mask branch extends detection to instance segmentation (Section 12).

Semantic segmentation labels every pixel with a class; instance segmentation also separates objects of the same class, so two touching cells get two labels (Mask R-CNN adds a mask head to each detected box). For semantic segmentation the network outputs a map of logits \mathbf{Z} \in \R^{C \times H \times W}, one score per class per pixel, and the basic loss is the cross-entropy of Module 01 averaged over pixels:

The fully convolutional network

Long et al. (2015) observed that a classifier’s dense layers are convolutions: a dense layer from C inputs to C' outputs applied to a 1 \times 1 map is a 1 \times 1 convolution with the same C' \times C weights, and a dense layer reading a 7 \times 7 \times 512 map is a 7 \times 7 convolution. Rewritten this way, the classifier runs at every position of a larger input, a fully convolutional network (FCN). Its score map is coarse: after five poolings the stride is 32, so a 512 × 512 image gives 16 × 16 scores. The FCN upsamples them to full size and fuses in predictions from finer maps (stride 16 and 8). Every segmentation network since downsamples to gather context, then upsamples to recover position.

Upsampling and the transposed convolution

The decoder upsamples either by interpolation followed by a convolution, or by a transposed convolution, which scatters each input value through the kernel: input i adds K_t \, x_i to output position i s + t - p for each tap t. It is the gradient of a strided convolution with respect to its input (Section 2), run forwards as a layer with learned weights. Its output size inverts the convolution’s:

where output_padding picks one of the input sizes that a strided convolution maps to the same output size.

Output position o receives one contribution for every tap t with t \equiv o + p \pmod s. If s divides k, every residue class contains k/s taps and every interior output receives the same number of contributions. If it does not, some positions receive \lceil k/s \rceil and others \lfloor k/s \rfloor, and in two dimensions the product of two such patterns is a checkerboard (Figure 3.14). Training can hide it, but the artefact is easy to produce and hard to remove (Odena et al. 2016). Kernel sizes divisible by the stride (k = 2 or 4 with s = 2), or resizing followed by a convolution, avoid it.

U-Net

The U-Net (Ronneberger et al. 2015) is a symmetric encoder–decoder. The contracting path repeats two 3 × 3 convolutions with ReLU and a 2 × 2 max pooling, doubling the channels from 64 to 1,024. The expanding path repeats a 2 × 2 up-convolution that halves the channels, a concatenation with the encoder map of the same resolution, and two 3 × 3 convolutions. A final 1 × 1 convolution maps the 64 channels to class scores (Figure 3.15). The bottleneck sees wide context at a sixteenth of the resolution; the skips carry the detail it has lost straight across, so fine boundaries survive.

Lab 5 measures what the skips buy on synthetic images: segment the circles, not the rectangles of the same brightness (Dice and IoU, defined below, pooled over all validation pixels). A tuned global threshold reaches Dice 0.64 (IoU 0.47), since brightness cannot tell shapes apart. In full mode the U-Net reaches Dice 0.965 (IoU 0.933), the same network without skips 0.952 (0.908). The larger difference is at the boundaries: pixel accuracy within 2 pixels of the true edge is 0.950 with skips and 0.920 without, and the lab’s prediction plots show the no-skip model’s errors lying along the object outlines.

Dice, IoU and the soft Dice loss

With P the predicted foreground and G the true one, the Dice coefficient and the IoU (Jaccard index) are

They are tied. From the second, \mathrm{TP} + \mathrm{FP} + \mathrm{FN} = \mathrm{TP}/J, so 2\mathrm{TP} + \mathrm{FP} + \mathrm{FN} = \mathrm{TP}(1 + J)/J and

D \ge J, and since D increases with J both rank models identically. Quote one and say which.

To train on Dice directly, replace the sets by probabilities p_i \in [0, 1] and binary targets g_i. With I = \sum_i p_i g_i, P = \sum_i p_i and G = \sum_i g_i, the soft Dice loss is

(Milletari et al. 2016 square the terms of the denominator.) By the quotient rule, with \partial I/\partial p_i = g_i and \partial P/\partial p_i = 1,

positive on foreground pixels and negative on background ones. Every gradient depends on the global sums, so the loss is normalised by object size. Dice plus cross-entropy is the common combination.

Background usually dominates. A 20 × 20 object in a 256 × 256 image is 400/65{,}536 = 0.61\% of the pixels, so predicting all background scores 99.4% pixel accuracy and Dice 0. Report Dice or IoU per class instead. Empty masks raise the opposite problem: with no foreground in prediction or truth, Dice is 0/0. The \epsilon term defines it as 1, computing Dice over the batch avoids it; either way, state how images without objects were scored.

U-Net is the standard architecture for biomedical segmentation. Its output is a label mask, the input a meshing step expects: an isosurface algorithm turns it into a surface whose volume and area can be measured, as Section 13 does with its error budget.

Sensor streams, the observation series of a digital twin and spectra are 1D signals; CT scans and confocal microscopy stacks are 3D volumes. The convolution carries over to both unchanged; what changes is causality in 1D and cost in 3D.

1D convolution and causality

With dilation d, a 1D convolution computes y_t = \sum_{u=0}^{k-1} w_u\, x_{t - u d}, written so that the window ends at t. The receptive-field arithmetic of Section 3 is unchanged: a stride-1 stack has r = 1 + \sum_l (k_l - 1) d_l. What is new is direction. A forecaster must not see the future, so a causal convolution pads (k - 1)d zeros on the left only: the output keeps the input’s length, and the output at time t depends only on inputs up to t. Symmetric padding would let each output read (k - 1)d/2 future samples, a leak that flatters a forecaster offline.

Dilation gives long histories without the loss of time resolution stride would cause. WaveNet (van den Oord et al. 2016) stacked causal convolutions of kernel 2 with dilations doubling at every layer; Figure 3.16 draws the first three layers of such a stack.

Such residual stacks, temporal convolutional networks (Bai et al. 2018), are strong baselines against recurrent networks for sequence tasks. Module 04, Exercise 15 replaces an LSTM forecaster with a dilated causal stack and compares the two.

3D convolution and its cost

A 3D layer has k^3 C_{\text{in}} C_{\text{out}} + C_{\text{out}} parameters and costs k^3 C_{\text{in}} C_{\text{out}} D_{\text{out}} H_{\text{out}} W_{\text{out}} MACs: one more factor of k than 2D. The 3D U-Net (Çiçek et al. 2016) is Section 12’s U-Net with 3 × 3 × 3 convolutions, trained from sparsely annotated slices.

import torch.nn as nn
import torch.nn.functional as F

class Down3D(nn.Module):
    def __init__(self, cin, cout):
        super().__init__()
        self.conv = nn.Sequential(
            nn.Conv3d(cin, cout, 3, padding=1), nn.GroupNorm(8, cout), nn.ReLU(),
            nn.Conv3d(cout, cout, 3, padding=1), nn.GroupNorm(8, cout), nn.ReLU())

    def forward(self, x):
        skip = self.conv(x)                  # kept for the decoder's skip connection
        return F.max_pool3d(skip, 2), skip

print(sum(p.numel() for p in Down3D(1, 32).parameters()))

28704

Group norm, because a volume batch is usually one or two stacks, where batch norm’s statistics (Module 02, Section 10) are meaningless; group norm normalises each example over groups of channels and all voxels, independently of the batch.

Memory decides the patch size

In 3D the binding constraint is the activations kept for the backward pass.

So volumes are trained on patches with a batch of one or two. Sliding-window inference tiles the volume with overlapping patches and blends the overlaps, by averaging or with Gaussian weights that trust a patch’s centre more than its padded edges, so that no seams appear. The U-Net paper’s overlap-tile strategy is the 2D version.

Anisotropic voxels and cheaper 3D

Confocal stacks are often sampled at 0.5 × 0.5 µm in plane and 2 µm between slices: a 3 × 3 × 3 kernel spans 1.5 µm across but 6 µm deep. Remedies: resample to isotropic voxels, use anisotropic kernels (1 × 3 × 3 in early layers), or segment slice by slice in 2D. Whichever is chosen, the spacing must travel with the data.

The (2+1)D factorisation replaces a 3 × 3 × 3 convolution by a 1 × 3 × 3 spatial one and a 3 × 1 × 1 axial one: 9C^2 + 3C^2 = 12C^2 weights instead of 27C^2 at width C. Depthwise 3D convolutions follow Section 6.

From mask to surface

A U-Net trained on annotated microscopy or CT produces a probability volume and, thresholded, a mask. Volume is voxel count times voxel volume. The surface is an isosurface extracted by marching cubes (Lorensen and Cline 1987): visit each cube of 8 neighbouring voxel centres; classify its corners as inside or outside (2^8 = 256 cases, 15 up to rotation and complement); on each edge whose corners disagree, place a vertex by linear interpolation of the probability at level 0.5; and join the vertices into triangles from a lookup table. Scaled by the spacing, the mesh gives area and enclosed volume.

Counting exposed voxel faces is biased at every resolution. For a convex object the faces looking along +x add up to its projected area on the yz plane, so a sphere shows 6\pi r^2 of faces against 4\pi r^2, a factor of 1.5 at any voxel size. In 2D a disc’s pixel-edge perimeter is 8r against 2\pi r, a factor of 4/\pi = 1.27. Figure 3.17 sets the staircase beside the isosurface in both dimensions.

An error budget

Module 01, Section 8’s split into bias and variance organises a measurement. Bias comes from discretisation (the staircase), the threshold level and systematic segmentation errors; variance from noise and from where the object falls on the voxel grid.