Stochastic optimisation and regularisation

Stochastic optimisation replaces an expensive full gradient with a sampled estimate. That replacement changes variation, cost and the meaning of progress; it does not change a training score into a population guarantee. This lesson derives conditional gradient unbiasedness and batch covariance, analyses a noisy quadratic, defines momentum and Adam updates precisely, and separates penalties, decay, early stopping and clipping. Each experiment records the finite objective, sampled units, update convention and evaluation budget.

Retrieval check: Module 19 supplies smooth descent and quadratic curvature; Module 23 supplies covariance; Module 25 supplies MAP penalty scales. Module 26’s split rules support evaluation. The optional spectral view revisits Module 13. All labs use the pinned NumPy preparation, with standard-library arithmetic for the fault examples and no GPU.

For a probability law P of examples Z and a loss ℓ(w;Z), population risk is R(w)=E_P[ℓ(w;Z)] when this expectation is defined. For a fixed dataset z₁,…,z_N, empirical risk is F(w)=(1/N)Σℓ(w;z_i). The first averages over a population, the second over recorded data. Once the dataset is fixed, F is a deterministic function even when an algorithm samples indices to estimate its gradient. Distinguish data randomness, batch randomness conditional on data, and any other algorithmic randomness before making an unbiasedness statement.

For differentiable finite losses, G(w)=∇F(w)=(1/N)Σg_i(w), with g_i=∇ℓ_i. No interchange theorem is needed for this finite sum. For population risk, writing ∇Eℓ=E∇ℓ needs additional conditions, for example differentiability in a neighbourhood with an integrable envelope controlling the derivative components. A finite-data minibatch derivation cannot silently establish that exchange for every unbounded population model. The course’s explicit quadratic examples keep gradient and moment existence transparent.

Let I be uniform on {1,…,N}, sampled freshly conditional on current w and the fixed data. Then E[g_I(w)|w,data]=(1/N)Σg_i(w)=G(w). A batch average of b independent uniform indices has the same conditional expectation. The current iterate can depend on previous batches; conditioning on it still leaves a fresh uniform draw valid. A deterministic batch selected because its gradient looks favourable is a different estimator and need not have that expectation. Uniform sampling is the essential finite-index calculation, not an informal slogan that every small batch is representative.

If indices are sampled nonuniformly with fixed positive probabilities p_i, unweighted g_I has expectation Σp_i g_i, the gradient of a differently weighted objective. Importance weighting g_I/(Np_I) restores the uniform-objective mean: summing p_i times that vector gives (1/N)Σg_i. Very small p_i can make the weighted gradients large and increase variance; unbiasedness alone says nothing about efficiency. State whether weights are fixed or data/history-dependent, and use the corresponding conditional probabilities if claiming the conditional identity.

For a regularised objective F(w)+λ||w||²/2, its exact penalty gradient is λw. A sampled data gradient plus this deterministic term is unbiased for that regularised objective conditional on w. Adding λw to each individual data gradient before averaging also gives the same term; multiplying it by the number of batch entries changes the objective. Sum versus mean data losses require the matching coefficients derived in Module 25. Every gradient should carry shape and normalisation checks before optimiser comparison.

Sampling an index from a fixed dataset does not create a new population example. A method can optimise F very accurately while R remains larger, especially after selection or distribution shift. Conversely a noisy estimate of G can still be useful computationally even though its current loss occasionally rises. The optimisation target, estimator expectation and generalisation target are different obligations. Module 30 will connect them through explicit statistical assumptions rather than claiming that lower training loss proves future accuracy.

At fixed w and data, define centred gradients d_i=g_i−G and finite-population covariance C=(1/N)Σd_i d_iᵀ. This denominator N describes a complete finite list, not an unbiased sample covariance estimator of an unknown population. For one uniform index, Cov(g_I)=C. For b independent uniform indices with replacement, the average covariance is C/b by the sum-covariance identity. Its expected squared deviation norm is tr(C)/b. Identical marginal index distributions alone do not justify removing cross terms.

For a uniformly selected size-b subset without replacement, 1≤b≤N and N>1, the average covariance is (C/b)(N−b)/(N−1). To prove it, two different sampled positions have cross-covariance −C/(N−1): Σ_i d_i=0 implies Σ_{i≠j}d_i d_jᵀ=−Σ_i d_i d_iᵀ. Divide by N(N−1). In the batch sum, b diagonal terms contribute C each and b(b−1) ordered cross terms contribute −C/(N−1); division by b² gives the formula. At b=N it becomes zero, matching the exact full gradient.

A general batch average has covariance [ΣCov(g_j)+Σ_{j≠k}Cov(g_j,g_k)]/b². Positive correlation can limit the benefit of larger nominal batches; negative correlation can reduce variation. These terms depend on sampling design and current parameter, not merely on a dataset column called “batch ID.” Duplicating examples or sharing a random augmentation across rows can alter them. Avoid an automatic standard-deviation/√b claim without the cross-term contract, just as copied observations required care in Module 24.

Random reshuffling visits a random permutation without replacement during an epoch. The whole permutation is uniform, but conditional on batches already observed, the next batch is drawn from remaining records rather than the entire original list. Its conditional gradient mean need not equal G(w) at that current iterate. Shuffle-based methods can work well, but this fact means the fresh-IID conditional proof is not their complete convergence analysis. Likewise choosing a hard-example batch based on the current model changes its selection probabilities and potentially its target weighting.

Increasing b reduces replacement variance at a fixed w, but changes update count at a fixed number of per-example gradient evaluations. Hardware parallelism can also make a larger batch faster per evaluated example, while memory and data-loading costs can reverse another comparison. Report both updates and evaluations, and wall-clock conditions if measured. Lab 1 fixes 2400 gradient evaluations, uses fewer full-batch updates and explicitly states that its learning-rate schedule is indexed by update count. Its finite result is not a universal batch-size ranking.

Covariance C itself changes with w for many nonlinear models, so one measured variance at the start cannot describe every training step. For the quadratic records, each gradient is w−a_i, and its centred component is −a_i, making C constant; this special construction enables exact analysis. In a real model, monitor an appropriate estimator only if its extra sampling cost and conditional interpretation are stated. A variance diagnostic is evidence about the supplied batch mechanism, not a proof that population gradients or every future batch behave the same way.

Stochastic gradient descent is w_(t+1)=w_t−η_t g_hat_t, with positive step η_t and a declared gradient estimator. If F has L-Lipschitz gradient and g_hat is conditionally unbiased with finite covariance C_t, the smooth descent inequality gives E[F(w−ηg_hat)|w]≤F(w)−η(1−Lη/2)||G(w)||²+(Lη²/2)tr(C_t). Derive it by substituting the update into Module 19’s quadratic upper bound, averaging the linear term, and using E||g_hat||²=||G||²+tr(C_t). The noise term can offset expected descent near a minimiser.

For a scalar quadratic F(w)=aw²/2 and estimate aw+ξ_t, with fresh mean-zero independent noise variance v and a>0, the update is w_(t+1)=(1−ηa)w_t−ηξ_t. Mean contracts if |1−ηa|<1, equivalently 0<ηa<2. Variance obeys V_(t+1)=(1−ηa)²V_t+η²v. With constant η in that interval, its limiting variance is ηv/[a(2−ηa)]. A constant-rate iterate therefore need not converge to a noiseless point, even though its mean converges to zero.

Decaying steps can reduce later noise contributions, but no arbitrary decreasing schedule guarantees success for every objective. In the noiseless scalar recurrence, if Ση_t is finite and steps are sufficiently small, the product Π(1−aη_t) can stay positive, leaving a nonzero initial component. Schedules commonly examined in stochastic approximation balance continued motion with square-summable noise effects under additional conditions. Here the exact quadratic recurrence and conditional descent bound are proved; broad convergence claims require their own smoothness, curvature, noise, boundedness and sampling assumptions.

The finite-total-step caution has a direct proof. Suppose every positive step satisfies aη_t≤c<1 and their sum is finite. The mean-value theorem applied to −log(1−u) on [0,aη_t] gives −log(1−aη_t)≤aη_t/(1−c). Summing these bounds keeps the negative log of the contraction product finite, so the infinite product is bounded below by a positive number. A nonzero noiseless initial iterate therefore retains a nonzero limiting component. This example does not prove which schedules succeed with random noise; it establishes why merely shrinking steps, even while every individual step is stable, can stop motion before reaching the target. Rate conditions and noise conditions need to be analysed together.

Define one momentum convention explicitly: v_(t+1)=βv_t+g_hat_t, w_(t+1)=w_t−ηv_(t+1), with v₀=0 and 0≤β<1. This buffer uses an unnormalised weighted sum. Under a constant gradient g it approaches g/(1−β), so holding η fixed while changing β changes effective update scale. An exponential-average convention multiplies the new gradient by 1−β and is numerically different unless step scales are adjusted. Momentum can damp or amplify oscillations depending on curvature and step choice; its history-dependent update is not simply a fresh unbiased gradient multiplied by a fixed scalar.

The named Adam method uses m_t=β₁m_(t−1)+(1−β₁)g_t and v_t=β₂v_(t−1)+(1−β₂)g_t², componentwise, with zero initial moments. Define m_hat_t=m_t/(1−β₁^t), v_hat_t=v_t/(1−β₂^t), and w_t=w_(t−1)−η_t m_hat_t/(√v_hat_t+ε), ε>0. These are the update equations in the original Adam paper, Algorithm 1. The v buffer estimates a raw second moment, not a centred variance; the square root and ε are componentwise and ε is outside the root in this convention.

For a stationary gradient distribution, expanding the exponential average shows E[m_t]=(1−β₁^t)E[g] and similarly for its squared-gradient raw moment, explaining zero-initialisation correction. During optimisation, gradient distributions change with iterates, so the corrected buffer is not automatically an unbiased current full gradient. Dividing by a random adaptive denominator also changes expectation. The algorithm definition does not establish unconditional convergence, a universal best optimiser or better generalisation. Preserve the precise formula and state what the finite experiment actually measures.

An explicit penalty changes the objective, while a different optimiser changes how the objective is traversed. Ridge adds λ||w||²/2, λ≥0, whose gradient is λw and Hessian λI. In a quadratic loss it can improve positive curvature and shift the minimiser toward the stated zero centre. It does not merely make numerical updates smaller while retaining the unpenalised target. Gaussian priors and noise scales in Module 25 supply one probabilistic interpretation, with sum/average conventions kept consistent.

For a scalar data loss .5(w−a)², ridge stationarity gives (w−a)+λw=0, so w=a/(1+λ). The coefficient generally stays nonzero. Lasso instead adds λ|w|. Away from zero its derivative is λsign(w), and at zero its convex subgradient set is [−λ,λ]. Optimality means zero belongs to w−a+λ∂|w|. If |a|≤λ, zero is optimal; otherwise w=a−λsign(a). This soft-threshold rule creates exact zeros through nonsmooth geometry, not by an arbitrary floating-point small-value cutoff.

For a differentiable F, a proximal-gradient step for F+λ||w||₁ first forms z=w−η∇F(w), then applies componentwise soft-thresholding with threshold ηλ. Its scalar subproblem minimises .5(u−z)²+ηλ|u|, so the threshold formula above directly derives the operation. Choosing a subgradient sign convention at zero and taking ordinary gradient steps is a different algorithm and may chatter around the kink. No smooth-gradient theorem applies to the whole absolute-value objective without handling its nonsmooth term.

With mean half-SSE, ridge coefficients satisfy (XᵀX/N+λI)w=Xᵀy/N, if penalising those columns and treating any intercept separately. At λ>0 the coefficient matrix is positive definite for finite X, even when X is rank deficient, because vᵀ(XᵀX/N+λI)v=||Xv||²/N+λ||v||²>0 for v≠0. At λ=0, uniqueness requires the usual full-rank condition. A linear solve is the appropriate computation; regularisation can improve conditioning while simultaneously introducing statistical bias.

Feature units change penalty meaning. Rescaling one coordinate of X and inversely rescaling its coefficient preserves predictions, but an isotropic coefficient norm is not invariant under that transformation. A penalty on standardised coefficients corresponds to a differently weighted penalty in raw coordinates. Training-only standardisation can improve numerical curvature, but does not preserve every prior interpretation under an unchanged λ. State whether the intercept is penalised, which coordinate system defines the prior, and how mean versus sum residual conventions are handled.

Lab 2 uses raw features with scales 1 and 100, fits scaling on training only, and inspects the resulting Hessian condition change. It compares a declared ridge list on validation, freezes the choice, then evaluates the test once. In this trace λ=0 wins the validation list. That is a legitimate result, not a reason to edit data or add test-guided candidates until regularisation appears beneficial. A finite validation winner remains a selected rule whose future performance needs its held-out uncertainty interpretation.

Early stopping selects an iterate before full optimisation. It can limit fit to noisy directions, but is a selection rule rather than automatically a fixed explicit penalty. A validation stopping protocol belongs to the training procedure; a final test used to choose the stop becomes selection data. Record how often validation is inspected, the candidate epochs, tie rule and whether the best observed iterate is restored. A random stopping time based on a noisy training gradient does not inherit a fixed-time statistical or optimisation guarantee without further analysis.

For a quadratic F(w)=.5wᵀHw−bᵀw with positive-definite H, full gradient descent from zero has eigenmode coefficient [1−(1−ηh_j)^t]b_j/h_j. This follows by solving the scalar affine recurrence in each eigenbasis direction. Under 0<η≤1/h_max, the filter grows monotonically between zero and one, so small-curvature directions develop more slowly. Ridge with coefficient λ instead has coefficient b_j/(h_j+λ), corresponding to filter h_j/(h_j+λ) relative to the unpenalised optimum. One stopping time generally does not match one λ across all eigenmodes.

This spectral comparison is a qualified mechanism explaining why early stopping can act like shrinkage in simple linear models. It is not an equivalence theorem for every neural network, stochastic path or learning-rate schedule. If ηh_j>1, modes may oscillate and the monotone-filter description no longer applies. If the data objective is singular, null directions and compatible b require separate treatment. State the finite quadratic assumptions before generalising from an attractive training curve.

Plain SGD on F+λ||w||²/2 updates w_new=w−η(g+λw)=(1−ηλ)w−ηg. Thus a multiplicative shrink factor and an L2 penalty gradient coincide under this specific plain-SGD convention. Some libraries parameterise a decay factor independently of learning rate; matching such a convention requires converting the coefficient, not assuming the same numerical hyperparameter. Momentum buffers or adaptive scaling can break this elementary algebra because the penalty gradient enters their history or preconditioner.

In the decoupled Adam-style convention here, compute m_hat and v_hat using the data gradient, then set w_new=(1−ηλ)w−η m_hat/(√v_hat+ε). Coupled L2 instead supplies g+λw to both moment buffers. These distinctions follow the decoupled weight-decay paper; they are update definitions, not a claim that one has universally lower test risk. The decay coefficient and learning-rate schedule jointly determine shrinkage along the trajectory. Label an experiment with its exact recurrence rather than a loose optimiser name.

Loss regularisation, data augmentation, dropout and early stopping may all affect training or generalisation, but they are not interchangeable objectives. Augmentation changes the examples/loss distribution, dropout adds a declared masking mechanism, a penalty changes F directly, and stopping chooses a point on a path. Understanding their mathematics begins by locating where the rule enters the expectation or update. This lesson derives explicit penalty and stopping mechanisms; specialised dropout or augmentation generalisation results require further modelling assumptions.

An ill-conditioned quadratic has curvature ratios that force a common stable step to be small for flat directions while avoiding explosive steep-direction updates. Module 19’s full-gradient eigenmode condition still informs a stochastic experiment, although sampled gradients add noise and different per-example curvature. A step stable for one average Hessian is not an unconditional guarantee for arbitrary random component updates. Inspect feature scale, objective normalisation, gradient norms and the actual sampling mechanism before treating erratic traces as unavoidable optimiser randomness.

For F(w)=10w²/2, the deterministic multiplier is 1−10η. Step .1 sends any w directly to zero; step .25 has multiplier −1.5 and grows in magnitude while alternating sign. This is a mathematical instability independent of the seed. A noisy method can also oscillate in a stable interval, so both magnitude trend and the model’s curvature must be assessed. Changing λ increases curvature and can change the stable step requirement, even while a penalty improves the condition ratio of a multi-dimensional problem.

Gradient clipping maps g to g min(1,c/||g||) for threshold c>0, with zero left zero. It bounds individual update inputs and preserves each nonzero vector’s direction, but usually changes the expectation and can bias an originally unbiased gradient estimator. Componentwise clipping is another rule with different geometry. Clipping can help control numerical excursions, yet cannot be cited as proof that the transformed updates optimise the original expectation without additional analysis.

A small noisy gradient is not a reliable certificate that the full gradient is small. In the four-record quadratic, one sampled gradient vanishes whenever w equals its selected a_i, although full G(w)=w can be far from zero. Likewise a small parameter step may result from a tiny learning rate, a large adaptive denominator or clipping, rather than near-optimality. Evaluate an appropriate full-gradient or statistically justified diagnostic when needed, and distinguish convergence of an algorithmic path from validation-based model selection.

Compare methods at a specified cost budget and fixed data protocol. Equal epochs, equal updates, equal example evaluations and equal wall time are different budgets. Record the schedule’s indexing, batch replacement rule, regularisation convention and precision. Seed-to-seed variation estimates algorithmic randomness conditional on a fixed dataset; additional independent datasets or test units are needed for population claims. Paired seeds or minibatch streams can make comparisons less noisy if they preserve intended marginal algorithms, but cross-run copying does not add independent replications.

Report training objective separately from unpenalised data loss and held-out metrics. A stronger penalty can raise training data error while lowering total regularised objective at its own optimum; comparing those totals across different λ also compares different functions. Validation chooses the protocol, and a final independent test evaluates the frozen choice. Lab 3 uses explicitly illustrative score arrays to demonstrate correct versus test-guided stopping; those inputs are not measured model benchmark results. Lab 2 supplies actual fitted-model outputs under the declared split.

The explorer shows multiple seeded scalar SGD paths and the analytical expected path for its explicit additive-noise quadratic. It labels the local recurrence stability condition and regularisation curvature. A reproducible random trace illustrates variation, while the proved variance and descent formulas supply the mathematical statements. Neither a low final loss nor one smooth-looking curve establishes universal convergence or generalisation. Carry the objective, dependence, numerical scale and evaluation contract together into the numerical-computation lesson next.

Lab 1 · Exact batches and equal evaluation counts

Download lab1_full_and_minibatch_updates.py

"""Fixed finite objective: exact gradient moments and equal evaluation budgets."""
import itertools
import numpy as np


if __name__ == "__main__":
    records = np.array([[-2.,0.],[0.,2.],[2.,0.],[0.,-2.]])
    w = np.array([3.,-2.])
    gradients = w-records
    print("Objective F(w)=mean(.5*||w-a_i||^2); four fixed records; minimiser=(0,0)")
    print(f"Full gradient={gradients.mean(axis=0).tolist()}; single-index gradient covariance={np.cov(gradients.T,bias=True).tolist()}")
    for replacement in (True,False):
        batches = list(itertools.product(range(4),repeat=2)) if replacement else list(itertools.combinations(range(4),2))
        means = np.array([gradients[list(batch)].mean(axis=0) for batch in batches])
        print(f"b=2, replacement={replacement}: exact average={means.mean(axis=0).tolist()}, coordinate variance={means.var(axis=0).round(6).tolist()}")
    budget = 2400
    for batch in (1,2,4):
        rng = np.random.default_rng(28028+batch)
        weights = np.array([3.,-2.])
        updates = budget//batch
        for t in range(updates):
            indices = np.arange(4) if batch==4 else rng.integers(0,4,batch)
            g = (weights-records[indices]).mean(axis=0)
            eta = .4/(1+t/50)
            weights -= eta*g
        gap = .5*(weights@weights)
        print(f"batch={batch}: evaluations={updates*batch}, updates={updates}, final w={weights.round(6).tolist()}, objective gap={gap:.9f}")
    print("The schedule is indexed by updates; equal evaluation budgets do not imply identical schedules or wall-clock costs.")

Objective F(w)=mean(.5*||w-a_i||^2); four fixed records; minimiser=(0,0)
Full gradient=[3.0, -2.0]; single-index gradient covariance=[[2.0, 0.0], [0.0, 2.0]]
b=2, replacement=True: exact average=[3.0, -2.0], coordinate variance=[1.0, 1.0]
b=2, replacement=False: exact average=[3.0, -2.0], coordinate variance=[0.666667, 0.666667]
batch=1: evaluations=2400, updates=2400, final w=[-0.141294, -0.033818], objective gap=0.010553878
batch=2: evaluations=2400, updates=1200, final w=[-0.154031, 0.047368], objective gap=0.012984558
batch=4: evaluations=2400, updates=600, final w=[0.0, -0.0], objective gap=0.000000000
The schedule is indexed by updates; equal evaluation budgets do not imply identical schedules or wall-clock costs.

Derive all three covariance cases before running. Compare the full/single/two-sample traces at equal evaluation count, explicitly retaining the differing update-indexed schedule.

Lab 2 · Training-only scaling and regularisation

Download lab2_regularisation_and_scaling.py

"""Training-only scaling and stated ridge/lasso coefficient conventions."""
import numpy as np


if __name__ == "__main__":
    rng = np.random.default_rng(28029)
    latent = rng.normal(size=(240,2))
    features = latent*np.array([1.,100.])
    response = 2*latent[:,0]-.5*latent[:,1]+rng.normal(0,.5,240)
    train, validation, test = np.arange(144), np.arange(144,192), np.arange(192,240)
    center, scale = features[train].mean(axis=0), features[train].std(axis=0)
    scaled = (features-center)/scale
    response_center = response[train].mean()
    print("NumPy1.26.4; seed28029; fixed144/48/48 split; transformations fitted on training only")
    print(f"raw Hessian condition={np.linalg.cond(features[train].T@features[train]):.6f}; scaled condition={np.linalg.cond(scaled[train].T@scaled[train]):.6f}")
    candidates = [0., .01, .1, 1.]
    fitted = []
    for lam in candidates:
        a = scaled[train]
        # Mean half-SSE + lambda/2||w||^2; intercept response_center is not penalised.
        w = np.linalg.solve(a.T@a/len(train)+lam*np.eye(2),a.T@(response[train]-response_center)/len(train))
        val_mse = np.mean((response_center+scaled[validation]@w-response[validation])**2)
        fitted.append((val_mse,lam,w))
        print(f"lambda={lam:.2f}, standardised coefficients={w.round(6).tolist()}, validation MSE={val_mse:.6f}")
    _, selected, weights = min(fitted,key=lambda row:row[0])
    test_mse = np.mean((response_center+scaled[test]@weights-response[test])**2)
    print(f"Frozen validation-selected lambda={selected:.2f}; final test MSE={test_mse:.6f}")
    a, lam = .8, 1.
    ridge = a/(1+lam)
    lasso = np.sign(a)*max(abs(a)-lam,0.)
    print(f"Scalar .5(w-a)^2: a={a}, lambda={lam}; ridge lambda/2*w^2 solution={ridge:.6f}; lasso lambda*|w| solution={lasso:.6f}")
    print("Standardised-coordinate penalties define different prior geometry from penalties on raw units; scaling is not permission to reuse the same probabilistic interpretation.")

NumPy1.26.4; seed28029; fixed144/48/48 split; transformations fitted on training only
raw Hessian condition=11263.011726; scaled condition=1.021254
lambda=0.00, standardised coefficients=[1.983694, -0.475552], validation MSE=0.269910
lambda=0.01, standardised coefficients=[1.964101, -0.471048], validation MSE=0.272948
lambda=0.10, standardised coefficients=[1.803755, -0.43404], validation MSE=0.324786
lambda=1.00, standardised coefficients=[0.99307, -0.242984], validation MSE=1.325412
Frozen validation-selected lambda=0.00; final test MSE=0.286922
Scalar .5(w-a)^2: a=0.8, lambda=1.0; ridge lambda/2*w^2 solution=0.400000; lasso lambda*|w| solution=0.000000
Standardised-coordinate penalties define different prior geometry from penalties on raw units; scaling is not permission to reuse the same probabilistic interpretation.

State the mean half-SSE convention and unpenalised intercept. Explain the validation-selected zero penalty and the difference between standardised and raw prior geometry.

Lab 3 · Rate correlation decay and selection faults

Download lab3_rates_correlation_and_decay.py

"""Deliberate rate, correlation, nonlinear preconditioning and selection faults."""
import math


if __name__ == "__main__":
    curvature, start = 10.,1.
    for eta in (.1,.25):
        w = start
        for _ in range(10):
            w -= eta*curvature*w
        print(f"F=.5*10*w^2, eta={eta:.2f}: factor={1-eta*curvature:.2f}, w after10={w:.6f}, loss={.5*curvature*w*w:.6f}")
    gradients = [-2.,0.,2.,0.]
    mean = sum(gradients)/4
    variance = sum((g-mean)**2 for g in gradients)/4
    print(f"Two IID gradient indices: mean variance={variance/2:.6f}; one index copied twice: variance={variance:.6f}")
    w, g, lam, eta, eps = 2., .1, .5, .01, 1e-8
    coupled_gradient = g+lam*w
    # Adam first step, zero-initialised moments with both bias corrections.
    coupled = w-eta*coupled_gradient/(abs(coupled_gradient)+eps)
    decoupled = (1-eta*lam)*w-eta*g/(abs(g)+eps)
    sgd_coupled = w-eta*(g+lam*w)
    sgd_decay = (1-eta*lam)*w-eta*g
    assert math.isclose(sgd_coupled,sgd_decay,abs_tol=1e-15)
    print(f"SGD L2-gradient={sgd_coupled:.9f}, decoupled decay={sgd_decay:.9f} (equal under this convention)")
    print(f"Bias-corrected Adam first step: coupled L2={coupled:.9f}, decoupled decay={decoupled:.9f} (different)")
    grad_values = [-2.,1.]
    clipped = [max(-1.,min(1.,x)) for x in grad_values]
    print(f"Unclipped mean={sum(grad_values)/2:.6f}; clipped mean={sum(clipped)/2:.6f} (clipping can change expectation)")
    # Illustrative input scores, not measured model runs.
    print("Illustrative score arrays demonstrate the selection rule, not a model benchmark.")
    validation = [.30,.26,.25,.27,.29]
    test = [.31,.29,.28,.24,.26]
    chosen = min(range(len(validation)),key=validation.__getitem__)
    invalid = min(range(len(test)),key=test.__getitem__)
    print(f"Predeclared validation stopping chooses epoch{chosen+1}; corresponding one-time test={test[chosen]:.2f}")
    print(f"Test-based stopping chooses epoch{invalid+1}, reported={test[invalid]:.2f}; this is selection, not an untouched final test.")

F=.5*10*w^2, eta=0.10: factor=0.00, w after10=0.000000, loss=0.000000
F=.5*10*w^2, eta=0.25: factor=-1.50, w after10=57.665039, loss=16626.283650
Two IID gradient indices: mean variance=1.000000; one index copied twice: variance=2.000000
SGD L2-gradient=1.989000000, decoupled decay=1.989000000 (equal under this convention)
Bias-corrected Adam first step: coupled L2=1.990000000, decoupled decay=1.980000001 (different)
Unclipped mean=-0.500000; clipped mean=0.000000 (clipping can change expectation)
Illustrative score arrays demonstrate the selection rule, not a model benchmark.
Predeclared validation stopping chooses epoch3; corresponding one-time test=0.28
Test-based stopping chooses epoch4, reported=0.24; this is selection, not an untouched final test.

Repair instability and copied-gradient variance, derive the first adaptive update, and diagnose clipping bias. The stopping arrays are labelled illustrative inputs rather than measured benchmark scores.

Exercises 1–12 are required. Optional Exercises 13–14 add 35 minutes beyond the twelve-hour schedule.

Use Mathematics for Machine Learning for optimisation and model objectives. Check Adam Algorithm 1 and the decoupled weight-decay paper for their exact update definitions. The finite-batch covariance, noisy quadratic and penalty derivations above are proved here without borrowing universal performance claims.

Derive unbiased sampling, batch covariance, expected descent, a quadratic noise floor, ridge/lasso rules and decay differences. Explain budgets, coordinates, clipping and split-safe stopping in one coherent report.

Exit task: contrast an IID two-index batch, a distinct-index subset and a copied gradient. Repair a large-curvature step and an Adam L2/decay equivalence claim, then name the final evaluation protocol.

Ready to move on: you can audit a stochastic training update mathematically. Numerical computation next investigates rounding, conditioning and stable implementations. See the course overview.