Information theory, entropy, and probabilistic objectives

Why does probability fitting use logarithmic loss, and what does perplexity actually measure? This module builds finite discrete entropy, conditional entropy, cross-entropy, KL divergence and mutual information from explicit probability models. It proves the relevant inequalities, connects expected loss to decodable codes and likelihood, and finishes with sequence factorisation and a carefully limited continuous preview. Every reported information quantity names its log base, alphabet and averaging unit.

Retrieval check: use Module 16 for the logarithm inequality, Module 23 for expectations and joint laws, and Module 25 for likelihood. The optional density preview uses Module 17. All labs are standalone standard-library scripts on a CPU; no external corpus or model download is needed.

For an outcome with probability p(x)>0, its surprisal is −log_b p(x), where b>1 is the declared log base. Smaller probability means larger surprisal; a certain outcome has zero surprisal. Base two measures bits and base e measures nats. For independent outcomes, probabilities multiply and surprisals add. More generally a joint probability factors into conditional probabilities, so conditional surprisals add without assuming independent outcomes. The additivity of log turns a probability product into a useful additive information or loss quantity.

An impossible outcome has infinite surprisal under the model. In an expectation under p, outcomes with p(x)=0 have zero weight; define 0 log 0=0 by the limit t log t→0 as t approaches zero from above. This convention does not let us replace a positive-weight zero predicted probability by a harmless zero term. When p(x)>0 but another model q(x)=0, its expected log-loss contribution is infinite. Support handling therefore belongs in both mathematical definitions and executable implementations.

Changing log base rescales information. A nats value divided by log 2 becomes bits, because log₂ u=log u/log 2. A numerical loss of one has different meaning in those two units. Probability, code length and log-loss should not share an unlabeled column. When exponentiating a loss later, the exponential base must match the logarithm base; Lab 3 exposes the familiar exp(bits) mistake.

A prefix code has no codeword equal to the beginning of another. Reading to the next leaf of its binary tree gives an unambiguous next symbol, then restarts at the root. Assigning each symbol a very short word without preserving unique decoding is not a compression achievement. For example words 0,1,01 are ambiguous because the bit string 01 can mean the third symbol or the first followed by the second. A code’s average length needs a decodability contract, not just an attractive weighted average.

For a finite binary prefix code with lengths ℓ_i, Kraft’s inequality is Σ2^(−ℓ_i)≤1. One proof associates each binary word with its dyadic subinterval of [0,1): a length-ℓ word identifies an interval of length 2^(−ℓ). Prefix-free words identify disjoint intervals, so their total length is at most one. This finite proof will give the expected-length entropy lower bound in Section 3. The converse existence theorem for valid integer lengths is a named coding result, with further reading linked at the end.

An ideal surprisal such as −log₂(.3) is generally not an integer codeword length. It is an information quantity; rounding and a code construction are separate steps. Block coding or arithmetic-coding approaches can approach ideal average rates under stated model and decoding conditions, but we do not infer an exact real-number-length code from a fractional logarithm. Likewise entropy does not count the entire storage cost of a file: model description, headers, finite-block effects and implementation overhead also matter.

For a nondegenerate finite source with at least two positive-probability symbols, rounded lengths ℓ_i=ceil(−log₂ p_i) are positive integers and satisfy Σ2^(−ℓ_i)≤Σp_i=1. The stated Kraft converse supplies a prefix code with those lengths. Since each rounded length is less than its ideal length plus one, its expected length is less than H₂(p)+1; the lower bound derived later places it at least H₂(p). This brackets average coding cost rather than assigning fractional codewords. Zero-probability symbols may be excluded from the declared source support for this construction, but a prediction system that must handle previously unseen symbols needs a separate support convention. Encoding the sequence length or a termination marker can also add cost if it is not already known to the decoder. The existence theorem is named here; proving its tree construction is a useful follow-on coding exercise, while the finite Kraft necessity and expected-length lower bound are proved in this lesson.

For a finite alphabet with probability vector p, entropy H_b(p)=−Σp_i log_b p_i is expected surprisal under that same law. It is a property of the distribution, not the surprise of one realised outcome. Each positive term is nonnegative because probabilities are at most one; entropy is zero exactly for a deterministic variable. For an alphabet of m symbols the maximum is log_b m at the uniform law, proved through KL divergence in Section 4. The bound uses declared alphabet size, while a distribution with smaller support cannot attain the full uniform maximum.

For a joint law p(x,y), joint entropy averages −log p(x,y). Conditional entropy H(X|Y) averages H(X|Y=y) with weights p(y), considering only y with positive mass. It is not the entropy of one selected conditional group, nor an unweighted average over group labels. On each positive joint cell, p(x,y)=p(y)p(x|y). Taking negative logs, weighting and summing yields H(X,Y)=H(Y)+H(X|Y), the entropy chain rule. The reverse order also holds by the same argument.

For independent finite variables the conditional law equals the marginal on positive cases, giving H(X,Y)=H(X)+H(Y). Without independence, the missing amount is mutual information, developed in Section 5. Joint entropy is not usually the sum of marginal entropies. Since discrete conditional entropy is nonnegative, H(X,Y)≥H(Y), and a deterministic function Y=f(X) has H(Y|X)=0. The chain rule then implies H(Y)≤H(X): deterministic processing cannot create new discrete uncertainty about its output.

Conditioning reduces entropy on average in finite discrete models: H(X|Y)≤H(X), with equality precisely under independence. This is proved by the mutual-information KL identity below. It does not assert that every individual group entropy is smaller than H(X). For instance Y chooses with probability .9 a deterministic-zero X group, and with probability .1 a fair-bit X group. Marginal P(X=1)=.05 gives entropy about .286397, while the fair group’s conditional entropy is 1. The weighted conditional entropy .1 is nevertheless smaller. The average and selected-case statements have different quantifiers.

For a sequence, repeatedly apply the chain rule: H(X₁,…,X_T)=ΣH(X_t|X₁,…,X_(t−1)). No token independence is needed. A context model can reduce conditional uncertainty by representing dependence; a unigram model instead uses marginal probabilities. A causal ordering of prediction contexts is a probability factorisation, not automatically a scientific causal claim about interventions. Every conditional term must be evaluated on contexts with positive relevant probability, just as in Module 21’s conditional laws.

The finite-alphabet setting keeps all entropies finite and makes rearrangement of sums safe. Countably infinite alphabets can have infinite entropy, so subtraction expressions such as infinity minus infinity need extra care even when a relative-information quantity can be defined directly. The labs and proofs here use finite alphabets. State this scope before carrying chain differences into an infinite vocabulary or continuous density model; the same notation can conceal different existence requirements.

For a source p and prediction q on the same ordered finite alphabet, cross-entropy H_b(p,q)=−Σp_i log_b q_i averages predicted surprisal under the actual source. If q_i=0 where p_i>0, it is infinite. A zero p_i contributes zero even if q_i is zero there. Checking matching alphabet semantics is essential: swapping labels in one vector changes the quantity even though each vector still sums to one. Normalisation, nonnegativity and finite numerical input are model checks, not optional formatting.

Cross-entropy differs from H(q). The former weights predictions by p, whereas the latter uses q as both source and prediction. A model can have very low own entropy by confidently choosing one outcome and still have enormous cross-entropy under a different source. Confidence is therefore not predictive correctness. Similarly a uniform q on m symbols gives cross-entropy log_b m for every p, even though the source entropy changes with p. This provides a useful fixed baseline with an explicit alphabet size.

For IID categorical observations x₁,…,x_n from a specified q family, negative log-likelihood is −Σlog q(x_i). If empirical frequencies are p_hat_i=c_i/n, the mean loss equals −Σp_hat_i log q_i=H(p_hat,q). Thus empirical cross-entropy is the average categorical negative log-likelihood. Its expectation under IID source p is H(p,q) for a fixed q, provided the support makes loss finite. A q fitted to those same observations is data-dependent; its training loss is not automatically an unbiased estimate of its future population loss.

For unconstrained categorical MLE, q_i=p_hat_i maximises the multinomial likelihood, including zero counts as boundary values. Its apparent training optimum can assign zero probability to unseen categories and produce infinite held-out loss if those categories later occur. Additive smoothing such as q_i=(c_i+a)/(n+ma), a>0, gives positive support on a fixed alphabet. This can be interpreted through an appropriate Dirichlet model, beyond the two-category beta update, or simply stated as a specified smoothing rule. It changes the estimator and should be chosen without using held-out labels.

The code-length connection needs precision. If a complete dyadic model q_i=2^(−ℓ_i) describes integer code lengths of a binary prefix code, its expected source length is exactly H₂(p,q). A generic q does not itself specify integer codewords; ceiling lengths give a coding construction with rounding overhead. Conversely for any finite binary prefix code, let Z=Σ2^(−ℓ_i)≤1 and r_i=2^(−ℓ_i)/Z. Then ℓ_i=−log₂ r_i−log₂ Z, so expected length is H₂(p,r)−log₂ Z. Section 4’s KL bound makes this at least H₂(p), proving the entropy lower bound with its decoding assumption.

The coding bound is average under the stated source, not a claim that every message uses at least its surprisal or that no individual message compresses exceptionally well. A mismatch p versus q adds average log loss; model overhead and finite-block coding cost remain separate. In prediction, a finite held-out token list estimates its own empirical mean log loss; it does not prove the source’s population entropy or the model’s quality along every dimension. The tiny language-model lab intentionally reports a smoothed frequency model worse than uniform on its fixed test tokens, so fitted training frequencies do not acquire an automatic held-out advantage.

For finite probability vectors p,q, define D_b(p||q)=Σ_{p_i>0}p_i log_b(p_i/q_i), with value infinity if any required q_i is zero. A zero p_i contributes zero. When support is compatible, subtract and regroup finite sums to get H_b(p,q)=H_b(p)+D_b(p||q). This identity identifies excess expected log loss and keeps the direction visible: p is the source whose outcomes receive weight. It is not a symmetric distance formula.

Prove nonnegativity using the natural-log inequality −log u≥1−u for u>0. Its difference g(u)=−log u−1+u has derivative 1−1/u, decreases below one, increases above one, and equals zero only at one. For q positive on the support S of p, apply it to u=q_i/p_i, multiply by p_i and sum: D(p||q)≥Σ_S(p_i−q_i)=1−Σ_S q_i≥0. Other bases divide by positive log b. If a required q_i is zero, infinite divergence is also nonnegative.

Equality requires q_i/p_i=1 on every positive p_i and no q mass outside that support. Since p sums to one, equality on support already uses all q mass, so globally p=q. This completes the zero-handling part often lost in a proof that silently assumes every probability is positive. Taking q uniform on m declared symbols gives D(p||q)=log_b m−H_b(p), hence H_b(p)≤log_b m with equality only for that uniform p. This also proves cross-entropy is minimised at the true source among all finite-alphabet predictions.

KL also does not satisfy all metric axioms. It is nonnegative and vanishes at equal laws, but lacks symmetry and in general the triangle inequality. For example Bernoulli rates .1,.5,.9 have D₂(.1||.9)≈2.53594 bits, while D₂(.1||.5)+D₂(.5||.9)≈1.26797 bits, so a triangle inequality fails. A square-root or symmetrised alternative requires its own definition and properties; calling an arbitrary symmetrisation a metric without proof is not justified.

The direction matters in fitting. Minimising D(p||q_θ) over a model family is equivalent to minimising source cross-entropy because H(p) is fixed. Reversing to D(q_θ||p) changes both the weighting and objective, and can be unavailable when p is an unknown data distribution. In a restricted family the minimum may be positive; KL nonnegativity does not guarantee that the family contains p or that an optimiser finds its best member. Model fit, numerical optimisation and the loss’s mathematical lower bound are separate claims.

Numerically near-equal distributions can produce a tiny negative calculated divergence from rounding even though the theorem says it is nonnegative. Validate probabilities and support first, compute stable logs and sum carefully, and interpret a scale-appropriate rounding discrepancy. Do not silently clamp a large negative value that reveals invalid inputs or a wrong formula. The lab uses finite vectors and exact zero conventions; a floating-point tolerance is disclosed in the normalisation check. All-zero weights cannot be normalised into a probability law.

The explorer explicitly normalises nonnegative category weights into p and q, then reports entropy, both cross-entropies and both KL directions in the selected log unit. A positive source weight against a zero prediction gives a visible infinity message. Matching q to p makes divergence zero; a zero-weight category in both is harmless. The bars depict probabilities, not codeword lengths, and the displayed ideal lengths remain informational quantities unless a decoding scheme is specified.

Replacing a zero prediction by a small epsilon changes the model and its loss. It can be a disclosed smoothing rule followed by renormalisation, but it cannot be presented as the original distribution’s exact divergence. Keep mathematical infinity and numerical log-domain stability separate.

Mutual information is I(X;Y)=D(p_XY||p_Xp_Y) for a finite joint law. Any positive joint cell has positive marginals, so its denominator is positive. Expand the log ratio into joint and marginal pieces to obtain I=H(X)+H(Y)−H(X,Y)=H(X)−H(X|Y)=H(Y)−H(Y|X). Nonnegativity follows from the proved KL inequality; equality holds precisely when the joint factors into independent marginals. Mutual information is symmetric even though general KL is directional, because this particular joint-versus-product expression treats X and Y symmetrically.

The entropy identities prove H(X|Y)≤H(X) on average, and I≤min(H(X),H(Y)) because discrete conditional entropies are nonnegative. A deterministic relation Y=f(X) gives I(X;Y)=H(Y), not necessarily H(X): a many-to-one function can discard information. Invertible relabelling preserves information. The quantities describe statistical dependence under a law; they do not identify causal direction. A common source can induce large mutual information without one measured variable causing the other.

Conditional mutual information is the p(z)-weighted KL between p(x,y|z) and p(x|z)p(y|z), summed over positive z. Each term is nonnegative. Entropy expansions give the chain identity I(X;Y,Z)=I(X;Y)+I(X;Z|Y). In a finite Markov relation X→Y→Z, meaning X and Z are conditionally independent given Y, I(X;Z|Y)=0. Thus I(X;Y,Z)=I(X;Y). Expanding in the other order gives I(X;Y,Z)=I(X;Z)+I(X;Y|Z)≥I(X;Z). We have proved the finite data-processing inequality I(X;Z)≤I(X;Y) with its conditional-independence assumption.

The theorem includes deterministic Z=f(Y), and more generally a stochastic channel whose conditional law depends on Y rather than extra access to X. It does not say adding a new sensor with independent information about X cannot help; that violates the stated processing chain. Nor does it say every output entropy must decrease: a stochastic processor can add fresh noise. Distinguish information about X from randomness in Z, and state the input whose information is being compared.

Mutual information is also different from linear covariance. Zero covariance can coexist with nonlinear dependence as Module 23 showed. For a finite version, let X take −1,0,1 equally and Y=X². Then Cov(X,Y)=0 by symmetry, while Y is a nonconstant deterministic function of X, so I(X;Y)=H(Y)>0. Estimating information from sparse data involves its own statistical bias and support issues; empirical contingency tables are not automatically accurate population laws. Discretising continuous data changes the observable variables and the measured information.

For a finite sequence x₁,…,x_T under model q, the probability chain rule gives q(x₁:T)=∏q(x_t|x_<t), with a start-context convention and positive prefixes where conditionals are used. Total negative log-likelihood is the sum of conditional token losses. Independence is unnecessary; an independent unigram model is a restrictive special case. For variable-length generative sequences a termination probability must also be defined for a normalised law over all finite strings. Omitting end tokens can instead define a conditional or partial evaluation score, which should be described that way.

Mean nats per scored token is L=−(1/T)Σlog q(x_t|x_<t); perplexity is exp L. In bits it is 2^L_bits. Equivalently it is the reciprocal of the geometric mean assigned token probability. It need not be an integer and is not literally the number of candidates the model considered. With a uniform model over m tokens, per-token cross-entropy log m yields perplexity m, providing a useful special-case intuition rather than a universal interpretation.

When aggregating unequal-length sequences, token-level corpus mean loss divides total scored loss by total scored tokens. Averaging sequence means instead weights every sequence equally and answers a different question. Include or exclude padding, prompt tokens, start/end tokens and masked positions according to a specified protocol. A model evaluation cannot quietly use a different token denominator from the advertised one. Sequence dependence also means token losses need not supply independent samples for a standard error; use suitable independent sequence or subject units if claiming statistical uncertainty.

Perplexity measures predictive log-loss under the chosen evaluation distribution. It does not by itself measure truthfulness, usefulness, fairness, reasoning success or every compression overhead. Low average loss can coexist with rare serious failures, and changing the test distribution changes the target. Held-out evaluation and selection rules from Module 26 still apply. A value of infinity from an unsupported required token is a model-support problem, while numerical overflow from exponentiating a huge finite loss is a representation problem; reporting log loss remains meaningful in the latter case.

Optional continuous preview: for a density f, differential entropy h(X)=−∫f(x)log f(x)dx when the integral is well defined. It is not discrete entropy on exact real points, whose probability is zero. Uniform(0,a) has density 1/a and differential entropy log a. At a=.25 this is negative in nats because density height 4 exceeds one. Changing units by Y=cX, c≠0, yields h(Y)=h(X)+log|c| when the transformation and integrability conditions hold. Thus nonnegativity and unit invariance of finite discrete entropy cannot be copied to differential entropy.

In a fine equal-width discretisation with bin width Δ and sufficient regularity, a useful approximation is H(discretised X)≈h(X)−log Δ; it records the resolution cost. This is a qualified approximation, not an identity for arbitrary mixed distributions or infinite-entropy densities. Continuous KL compares two densities with the same reference measure and has additional support/integrability conditions; a coordinate Jacobian cancels in its ratio when both laws transform consistently. Full continuous information theory is follow-on material rather than an assumption hidden in the finite proofs above.

Lab 1 · Entropy directional KL and information

Download lab1_entropy_kl_and_information.py

"""Finite information measures in bits, with explicit support conventions."""
import math


def validate(p):
    if not p or any(not math.isfinite(x) or x < 0 for x in p) or not math.isclose(math.fsum(p), 1., abs_tol=1e-12, rel_tol=0.):
        raise ValueError("probabilities must be finite, nonnegative, and sum to one")


def entropy(p):
    validate(p)
    return -math.fsum(x*math.log2(x) for x in p if x > 0)


def cross_entropy(p, q):
    validate(p)
    validate(q)
    if len(p) != len(q):
        raise ValueError("same ordered alphabet required")
    if any(x > 0 and y == 0 for x, y in zip(p, q)):
        return math.inf
    return -math.fsum(x*math.log2(y) for x, y in zip(p, q) if x > 0)


def kl(p, q):
    return cross_entropy(p, q)-entropy(p)


if __name__ == "__main__":
    p, q = [.5, .3, .2], [.25, .5, .25]
    print(f"p={p}; q={q}; same ordered alphabet; units=bits")
    print(f"H(p)={entropy(p):.6f}; H(p,q)={cross_entropy(p,q):.6f}; KL(p||q)={kl(p,q):.6f}; KL(q||p)={kl(q,p):.6f}")
    assert abs(cross_entropy(p,q)-entropy(p)-kl(p,q)) < 1e-12
    print(f"KL([1,0]||[.5,.5])={kl([1.,0.],[.5,.5]):.6f}; reverse={kl([.5,.5],[1.,0.])}")
    joint = [.45, .05, .05, .45]
    independent = [.25]*4
    mi = kl(joint, independent)
    print(f"Fair binary input, independent flip probability.1: MI={mi:.6f}; H(Y|X)={entropy([.1,.9]):.6f}")
    print(f"H(X,Y)={entropy(joint):.6f}; H(X)+H(Y)-H(X,Y)={2-entropy(joint):.6f}")
    assert abs(mi-(1-entropy([.1,.9]))) < 1e-12
    code_p, lengths = [.5,.25,.25], [1,2,2]
    expected = math.fsum(x*l for x,l in zip(code_p,lengths))
    print(f"Prefix code0/10/11: expected length={expected:.6f}; entropy={entropy(code_p):.6f}")

p=[0.5, 0.3, 0.2]; q=[0.25, 0.5, 0.25]; same ordered alphabet; units=bits
H(p)=1.485475; H(p,q)=1.700000; KL(p||q)=0.214525; KL(q||p)=0.198965
KL([1,0]||[.5,.5])=1.000000; reverse=inf
Fair binary input, independent flip probability.1: MI=0.531004; H(Y|X)=0.468996
H(X,Y)=1.468996; H(X)+H(Y)-H(X,Y)=0.531004
Prefix code0/10/11: expected length=1.500000; entropy=1.500000

Predict the support-infinite direction and prove the entropy/cross-entropy identities. Explain the prefix-code equality and the noisy bit’s conditional entropy.

Lab 2 · A tiny held-out frequency model

Download lab2_frequency_language_model.py

"""A tiny categorical unigram model evaluated on a fixed held-out token list."""
import collections
import math


if __name__ == "__main__":
    alphabet = ("a", "b", "c")
    training = list("aaaaaabbbc")
    test = list("abcac")
    counts = collections.Counter(training)
    # Alphabet and additive smoothing are specified before examining test tokens.
    alpha = 1.
    models = {"uniform": {t:1/3 for t in alphabet},
              "training-only smoothed unigram": {t:(counts[t]+alpha)/(len(training)+alpha*len(alphabet)) for t in alphabet}}
    print(f"Fixed alphabet={alphabet}; training counts={dict(counts)}; test tokens={test}; smoothing alpha=1")
    for name, model in models.items():
        losses = [-math.log(model[t]) for t in test]
        nll = math.fsum(losses)
        average = nll/len(test)
        bits = average/math.log(2)
        print(f"{name}: probabilities={[round(model[t],6) for t in alphabet]}")
        print(f"  held-out total NLL={nll:.6f}, mean nats/token={average:.6f}, bits/token={bits:.6f}, perplexity={math.exp(average):.6f}")
    print("This fixed unigram approximation ignores order and context. Its poor held-out result cannot be repaired by fitting test counts.")
    sequence_probability = .5*.8*.6
    print(f"Conditional sequence example: .5*.8*.6={sequence_probability:.6f}; chain NLL={-math.log(sequence_probability):.6f}")
    print("The chain factorisation uses conditional probabilities; it does not require independent tokens.")

Fixed alphabet=('a', 'b', 'c'); training counts={'a': 6, 'b': 3, 'c': 1}; test tokens=['a', 'b', 'c', 'a', 'c']; smoothing alpha=1
uniform: probabilities=[0.333333, 0.333333, 0.333333]
  held-out total NLL=5.493061, mean nats/token=1.098612, bits/token=1.584963, perplexity=3.000000
training-only smoothed unigram: probabilities=[0.538462, 0.307692, 0.153846]
  held-out total NLL=6.160338, mean nats/token=1.232068, bits/token=1.777498, perplexity=3.428310
This fixed unigram approximation ignores order and context. Its poor held-out result cannot be repaired by fitting test counts.
Conditional sequence example: .5*.8*.6=0.240000; chain NLL=1.427116
The chain factorisation uses conditional probabilities; it does not require independent tokens.

Keep alphabet and smoothing fixed before test evaluation. Derive the unigram probabilities and explain why training frequencies need not beat uniform on this held-out sequence.

Lab 3 · Units support and scoring repairs

Download lab3_units_support_and_tokenisation.py

"""Challenge units, invalid PMFs, zero support, and per-token comparisons."""
import math
def validate(p):
    if not p or any(not math.isfinite(x) or x < 0 for x in p) or not math.isclose(math.fsum(p), 1., abs_tol=1e-12, rel_tol=0.):
        raise ValueError("probabilities must be finite, nonnegative, and sum to one")


def cross_entropy(p, q):
    validate(p)
    validate(q)
    if len(p) != len(q):
        raise ValueError("same ordered alphabet required")
    if any(x > 0 and y == 0 for x, y in zip(p, q)):
        return math.inf
    return -math.fsum(x*math.log2(y) for x, y in zip(p, q) if x > 0)


if __name__ == "__main__":
    for bad in ([.8,.4], [-.1,1.1], [math.nan,1.], [0.,0.]):
        try:
            validate(bad)
        except ValueError:
            print(f"Rejected invalid probability list: {bad}")
    bits = 1.
    print(f"One bit loss: wrong exp(bits)={math.exp(bits):.6f}; correct 2**bits={2**bits:.6f}")
    print(f"Nats conversion={bits*math.log(2):.6f}; exp(nats)={math.exp(bits*math.log(2)):.6f}")
    print(f"Required support missing: cross-entropy([.5,.5],[1,0])={cross_entropy([.5,.5],[1.,0.])}")
    # Artificial joint models assign the same probability to the same raw string.
    probability = 1/16
    nll = -math.log(probability)
    for tokens in (2,4):
        ppl = math.exp(nll/tokens)
        print(f"Same raw-string probability1/16, {tokens} scored tokens: mean nats={nll/tokens:.6f}, perplexity={ppl:.6f}")
    print("A smaller token perplexity here reflects a different unit count, not a better raw-string probability.")
    p = .25
    print(f"Density preview: Uniform(0,.25) density=4, differential entropy nats={math.log(p):.6f}; exact-point probability=0")
    print("Differential entropy can be negative and changes with coordinate units; discrete entropy's nonnegativity does not transfer.")

Rejected invalid probability list: [0.8, 0.4]
Rejected invalid probability list: [-0.1, 1.1]
Rejected invalid probability list: [nan, 1.0]
Rejected invalid probability list: [0.0, 0.0]
One bit loss: wrong exp(bits)=2.718282; correct 2**bits=2.000000
Nats conversion=0.693147; exp(nats)=2.000000
Required support missing: cross-entropy([.5,.5],[1,0])=inf
Same raw-string probability1/16, 2 scored tokens: mean nats=1.386294, perplexity=4.000000
Same raw-string probability1/16, 4 scored tokens: mean nats=0.693147, perplexity=2.000000
A smaller token perplexity here reflects a different unit count, not a better raw-string probability.
Density preview: Uniform(0,.25) density=4, differential entropy nats=-1.386294; exact-point probability=0
Differential entropy can be negative and changes with coordinate units; discrete entropy's nonnegativity does not transfer.

Reject invalid probability lists, repair the bit/nat exponent mismatch and explain identical raw-string probability with different perplexities. The density preview requires Module 17.

Exercises 1–12 are required. Optional Exercises 13–14 add 35 minutes beyond the ten-hour schedule; the continuous alternative requires Module 17.

Use Stanford’s information-measures notes, variable-length coding notes and continuous-information lecture. The finite proofs and original examples above state zero-support assumptions explicitly; continuous theory is an optional follow-on.

Compute the finite measures, prove KL positivity and chain identities, derive likelihood loss and a prefix-code lower bound, and give a qualified perplexity report. Explain dependence without confusing information and causation.

Exit task: calculate the three-symbol mismatch, identify the infinite support direction, and repair a bit/nat or token-denominator comparison. Trace a dependent sequence’s probability into additive conditional losses.

Ready to move on: you can connect probabilistic models to log-loss objectives. Stochastic optimisation next studies how sampled gradients fit such objectives and how regularisation changes training. See the course overview.