Topic Overview
What are expectation, variance, and covariance? They are the three fundamental summaries of any random variable. Expectation tells you where the center is. Variance tells you how spread out it is. Covariance tells you how two variables move together.
Why do they exist? Because a full probability distribution contains too much information. If I give you the exact probability of every possible pixel configuration in an image, you cannot reason with it. You need summaries: the average brightness (expectation), the contrast or noise level (variance), and how the red channel relates to the blue channel (covariance).
What problem do they solve? They allow us to optimize, stabilize, and reason about systems in the presence of noise. You cannot minimize a loss function that changes every millisecond due to random data batches; you minimize its expected value. You cannot train a deep network if the magnitudes of the activations keep exploding or vanishing; you fix their variance.
Why should engineers care? Machine learning is just probability plus optimization. Expectation is the objective we optimize (Expected Risk). Variance is the instability we fight against (Batch Normalization, Gradient Noise). Covariance is the data structure we model (Multivariate Gaussians in VAEs). If you do not understand these three, you do not understand what your model is trying to do, why it is failing, or how to fix it.
Prerequisites
- Random Variables and Distributions: You must understand what a random variable \(X\) is and what a probability distribution \(p(x)\) or \(P(X=x)\) represents. Expectation and variance are operations performed on these distributions.
- Basic Linear Algebra: Needed for covariance matrices, which extend scalar variance into multi-dimensional space using outer products.
Historical Motivation
In the 17th century, as probability theory emerged, scientists faced a practical engineering problem: how do you make decisions in uncertain situations? A single game of chance is unpredictable. But what about a thousand games?
Christiaan Huygens and others developed the concept of Expectation to price fair games. If a game pays out different amounts with different probabilities, the “fair price” to enter the game is the expected value.
But expectation wasn’t enough. In the 19th century, as physics and statistics matured, researchers realized that two systems could have the exact same expectation but wildly different behaviors. A tight grouping of rifle shots and a scattered grouping both have the same center (expectation), but one is clearly superior. They needed a metric for “spread,” leading to the invention of Variance.
As scientists began studying complex systems—like the relationship between weather and crop yields—they needed to measure how two different uncertainties interacted. Covariance was invented to capture this co-movement, allowing actuaries to hedge risks and physicists to model multivariate systems. In ML, we use these exact concepts to optimize loss (Expectation), stabilize activations (Variance), and model latent spaces (Covariance).
Intuitive Explanation
Expectation (The Center of Gravity): Imagine the probability distribution is a physical object cut out of wood. The expectation is the exact point where you could balance it on the tip of a pencil. It is the weighted average, where more likely outcomes pull the balance point toward them.
Variance (The Bounce): Imagine you are bouncing on a trampoline. The expectation is where you land on average (the center). The variance is how high you bounce. A low-variance trampoline barely lifts you; you stay glued to the center. A high-variance trampoline launches you wildly all over the place. Variance measures the average squared distance from the center.
Covariance (The Seesaw): Imagine two people on a seesaw. If person X goes up and person Y goes down consistently, they have a negative covariance. If they both go up at the same time (somehow), they have a positive covariance. If X goes up and Y just randomly spins around, they have zero covariance. Covariance measures the tendency of two variables to move together.
Visual Understanding
1. Expectation: Draw a 2D bell curve. Draw a vertical line down the center. That line is the expectation \(\mathbb{E}[X]\). The area to the left equals the area to the right.
2. Variance: Draw two bell curves with the same center line. One is tall and narrow; the other is short and wide. The wide one has high variance. Visually, variance is the “wingspan” of the distribution.
3. Covariance: Draw a 2D scatter plot of points \((x, y)\). If the points form a diagonal line going up-right, they have positive covariance. If they form a diagonal line going down-right, negative covariance. If they look like a circular cloud, zero covariance.
4. Covariance Matrix: Visualize an ellipse in 2D space. The center of the ellipse is the mean vector. The axes of the ellipse point in the directions of the eigenvectors of the covariance matrix. The lengths of the axes represent the variance in those directions (eigenvalues).
Mathematical Foundations
Expectation
The expected value (mean) of a random variable \(X\). * Discrete: \(\mathbb{E}[X] = \sum_x x \cdot P(X=x)\) * Continuous: \(\mathbb{E}[X] = \int x \cdot p(x) \, dx\)
Why multiply by probability? We weight rare events lightly and common events heavily. It is the mathematically precise definition of the “balance point.”
Linearity of Expectation: \(\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]\). This holds always, even if \(X\) and \(Y\) are dependent. It is the most powerful property in ML, allowing us to decompose complex expectations into simpler ones.
Variance
How much \(X\) deviates from its mean, squared. \[\text{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]\]
Why the square? If we just used \((X - \mathbb{E}[X])\), the positive and negative deviations would cancel out to zero. Squaring forces all deviations to be positive, penalizing large outliers heavily.
The Computational Shortcut: \[\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2\] Derivation: 1. Expand the square: \(\mathbb{E}[X^2 - 2X\mathbb{E}[X] + (\mathbb{E}[X])^2]\) 2. Apply linearity: \(\mathbb{E}[X^2] - 2\mathbb{E}[X]\mathbb{E}[X] + \mathbb{E}[(\mathbb{E}[X])^2]\) 3. Note that \(\mathbb{E}[X]\) is a constant, so \(\mathbb{E}[\mathbb{E}[X]] = \mathbb{E}[X]\). 4. Result: \(\mathbb{E}[X^2] - 2(\mathbb{E}[X])^2 + (\mathbb{E}[X])^2 = \mathbb{E}[X^2] - (\mathbb{E}[X])^2\). Why this matters: In code, you can compute variance in one pass by tracking the mean of \(X\) and the mean of \(X^2\), without needing to store the whole dataset.
Covariance
How \(X\) and \(Y\) co-vary. \[\text{Cov}(X, Y) = \mathbb{E}[(X - \mathbb{E}[X])(Y - \mathbb{E}[Y])]\] The intuition: If \(X\) is above its mean and \(Y\) is above its mean, the product is positive. If they consistently do this together, the expectation of the product is positive.
Shortcut: \(\text{Cov}(X, Y) = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y]\).
Covariance Matrix
For a random vector \(\mathbf{x} \in \mathbb{R}^d\), we need to track variances of all \(d\) dimensions and covariances between all pairs. We pack them into a matrix: \[\boldsymbol{\Sigma} = \mathbb{E}[(\mathbf{x} - \boldsymbol{\mu})(\mathbf{x} - \boldsymbol{\mu})^\top]\] * Diagonals \(\Sigma_{ii} = \text{Var}(x_i)\) * Off-diagonals \(\Sigma_{ij} = \text{Cov}(x_i, x_j)\) * It is always symmetric and positive semi-definite.
Correlation
Covariance depends on units (e.g., measuring in meters vs millimeters changes the number). Correlation normalizes it to \([-1, 1]\): \[\rho(X, Y) = \frac{\text{Cov}(X,Y)}{\sqrt{\text{Var}(X)\text{Var}(Y)}}\]
Worked Numerical Example
Let \(X\) and \(Y\) be discrete random variables with the following joint outcomes: * \((1, 2)\) with probability \(0.5\) * \((3, 6)\) with probability \(0.5\)
1. Expectation: * \(\mathbb{E}[X] = (1 \cdot 0.5) + (3 \cdot 0.5) = 0.5 + 1.5 = 2\) * \(\mathbb{E}[Y] = (2 \cdot 0.5) + (6 \cdot 0.5) = 1.0 + 3.0 = 4\)
2. Variance (Using Shortcut): * \(\mathbb{E}[X^2] = (1^2 \cdot 0.5) + (3^2 \cdot 0.5) = 0.5 + 4.5 = 5\) * \(\text{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2 = 5 - 2^2 = 5 - 4 = 1\) * \(\mathbb{E}[Y^2] = (2^2 \cdot 0.5) + (6^2 \cdot 0.5) = 2 + 18 = 20\) * \(\text{Var}(Y) = 20 - 4^2 = 20 - 16 = 4\)
3. Covariance (Using Shortcut): * \(\mathbb{E}[XY] = ((1 \cdot 2) \cdot 0.5) + ((3 \cdot 6) \cdot 0.5) = (2 \cdot 0.5) + (18 \cdot 0.5) = 1 + 9 = 10\) * \(\text{Cov}(X,Y) = \mathbb{E}[XY] - \mathbb{E}[X]\mathbb{E}[Y] = 10 - (2 \cdot 4) = 10 - 8 = 2\)
4. Correlation: * \(\rho(X,Y) = \frac{2}{\sqrt{1 \cdot 4}} = \frac{2}{2} = 1\) * Interpretation: Perfect positive linear correlation. Notice that \(Y = 2X\) for all outcomes.
Computational Interpretation
What is the computer actually doing?
In ML, we rarely have the true probability distribution \(p(x)\). We have a dataset (a batch). We compute empirical statistics—the Monte Carlo approximation of the expectation.
- Empirical Expectation (Mean):
sum(x) / N. \(O(N)\) time. - Empirical Variance: We don’t use \(\sum (x - \text{mean})^2 / N\) because it
requires two passes over the data (one to find mean, one to sum
squares). Instead, we use the shortcut:
(sum(x^2)/N) - (sum(x)/N)^2. \(O(N)\) time in one pass.
Memory & Scaling: * Mean of a batch of shape
[B, D] requires storing a vector of shape [D].
\(O(D)\) memory. * Covariance matrix
requires outer products: \((\mathbf{x}-\boldsymbol{\mu})(\mathbf{x}-\boldsymbol{\mu})^\top\).
For a vector of dimension \(D\), the
covariance matrix is \(D \times D\). If
\(D=1000\), the matrix has 1,000,000
elements. Computing and inverting full covariance matrices is \(O(D^3)\), a severe bottleneck. This is why
ML models use diagonal or spherical covariance approximations.
Implementation Perspective
NumPy (From Scratch & Built-in)
import numpy as np
# Batch of 4 samples, 3 features
X = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9],
[10, 11, 12]])
# --- Expectation (Mean over batch, axis=0) ---
mean = np.mean(X, axis=0) # Shape: (3,)
# --- Variance ---
# BIASED variance (divide by N). Used in BatchNorm.
var_biased = np.var(X, axis=0, ddof=0)
# UNBIASED variance (divide by N-1). Used in statistics.
var_unbiased = np.var(X, axis=0, ddof=1)
# Manual biased variance using shortcut
var_manual = np.mean(X**2, axis=0) - mean**2
# --- Covariance Matrix ---
# Row var = False means columns are features. ddof=0 for biased.
cov_matrix = np.cov(X, rowvar=False, ddof=0) # Shape: (3, 3)PyTorch (Production ML)
import torch
# Batch of 4 samples, 3 features
x = torch.randn(4, 3)
# --- Expectation ---
mean = torch.mean(x, dim=0) # Shape: (3,)
# --- Variance ---
# CRITICAL: PyTorch defaults to unbiased (ddof=1).
# BatchNorm requires biased (correction=0)!
var_biased = torch.var(x, dim=0, correction=0)
# --- Batch Normalization Logic ---
eps = 1e-5
x_norm = (x - mean) / torch.sqrt(var_biased + eps) # Shape: (4, 3)
# --- Monte Carlo Expectation (RL Policy Gradient) ---
# log_probs: shape [batch_size], rewards: shape [batch_size]
log_probs = torch.log(torch.tensor([0.1, 0.4, 0.2, 0.3]))
rewards = torch.tensor([1.0, 0.0, 1.0, 0.0])
# Expectation of R * log(pi)
loss = -(log_probs * rewards).mean() Where It Appears In AI
| AI Component | Exact Usage of Expectation, Variance, Covariance |
|---|---|
| BatchNorm / LayerNorm | Normalizes activations to \(\mathbb{E}[x]=0, \text{Var}(x)=1\) to stabilize gradient flow. |
| Loss Functions | Expected Risk \(\mathcal{L}(\theta) = \mathbb{E}_{(x,y)}[\ell(f_\theta(x), y)]\). SGD estimates this expectation. |
| VAE | Encoder outputs mean \(\mu\) and variance \(\sigma^2\). KL divergence loss is a function of these expectations and variances. |
| RL (Policy Gradient) | Objective is expectation of reward: \(J = \mathbb{E}_\pi[R]\). High variance of this estimator is the central problem of RL. |
| Attention (Transformers) | \(QK^T\) is essentially an uncentered covariance matrix between query and key sequences, scaled by variance (\(\sqrt{d_k}\)). |
| Diffusion Models | Forward process is defined by shifting expectation (\(\sqrt{1-\beta_t}x\)) and adding variance (\(\beta_t\)). |
| Contrastive Learning | Maximizes covariance between positive pairs while minimizing variance across negative pairs. |
Deep Dive Into Real Models
1. Transformers (Scaled Dot-Product Attention) The attention logits are computed as \(A = \frac{QK^\top}{\sqrt{d_k}}\). Why divide by \(\sqrt{d_k}\)? Assume entries of \(Q\) and \(K\) are i.i.d. with zero mean and unit variance. The dot product is a sum of \(d_k\) products. By the properties of variance, \(\text{Var}(q_i k_i) = \text{Var}(q_i)\text{Var}(k_i) = 1\). The variance of the sum is \(d_k\). Thus, as dimension \(d_k\) grows, the variance of the dot product explodes, pushing softmax into regions with vanishing gradients. Dividing by \(\sqrt{d_k}\) forces the variance back to 1.
2. Batch Normalization (MLPs/CNNs) During training, BatchNorm computes the empirical expectation and variance across the spatial and batch dimensions: \(\mu_B = \mathbb{E}_{B, H, W}[x]\) and \(\sigma_B^2 = \text{Var}_{B, H, W}[x]\). It normalizes the layer to zero mean and unit variance. During inference, it does not use the batch statistics; it uses a running average of the expectations computed during training. If you mess up the variance calculation (biased vs unbiased), inference will be completely broken.
3. Reinforcement Learning (REINFORCE) We want to maximize the expected return: \(J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]\). We cannot compute this integral analytically. Instead, we sample trajectories (episodes) and compute the Monte Carlo estimate: \(\nabla J \approx \frac{1}{N} \sum R(\tau_i) \nabla \log \pi(\tau_i)\). This is an unbiased estimator of the expectation, but it suffers from massive variance. A single lucky episode can skew the expectation. Techniques like Advantage Actor-Critic (A2C) are essentially variance-reduction techniques for this expectation.
Failure Modes
1. The Biased vs. Unbiased Variance Trap *
Cause: In statistics, we estimate population variance by
dividing by \(N-1\) (Bessel’s
correction). In ML, BatchNorm normalizes the current batch, so
it must divide by \(N\) (the exact
variance of those specific points). * Symptoms: PyTorch’s
torch.var defaults to unbiased (correction=1).
If you use this to implement a custom BatchNorm, your training will
diverge because the normalization magnitude is systematically wrong, and
the running stats will be biased. * Solution: Always use
torch.var(x, correction=0) when computing batch statistics
for normalization.
2. Covariance Singularity * Cause: If
features in your data are linearly dependent, the covariance matrix
\(\boldsymbol{\Sigma}\) is singular
(determinant is 0). * Symptoms: Any algorithm that needs to
invert \(\boldsymbol{\Sigma}\) (like
Mahalanobis distance, Gaussian Mixture Models, or Quadratic Discriminant
Analysis) will throw a LinAlgError: Singular matrix. *
Solution: Add a small diagonal jitter \(\boldsymbol{\Sigma} + \epsilon \mathbf{I}\)
before inversion, or use PCA to remove redundant dimensions.
3. High Variance in RL Gradients * Cause: The REINFORCE gradient estimator \(\hat{g} = R(\tau)\nabla \log \pi\) has a variance proportional to the variance of the returns \(R\). If rewards swing between +100 and -100, the gradients will violently oscillate. * Symptoms: The policy fails to converge, or collapses to a suboptimal deterministic policy. * Solution: Subtract a baseline \(b\) (like a value function estimate): \(\hat{g} = (R(\tau) - b)\nabla \log \pi\). This doesn’t change the expectation (if \(b\) is independent of the action) but drastically reduces the variance.
Engineering Insights
Welford’s Online Algorithm: How does BatchNorm track running means and variances without storing all previous batches? It uses Welford’s algorithm. For each new sample \(x_n\), the mean is updated: \(\mu_n = \mu_{n-1} + (x_n - \mu_{n-1})/n\). The sum of squares is updated similarly. This allows computing exact variance in \(O(1)\) memory per feature, streaming infinitely.
Precision in Mixed Precision Training: When
computing variance in FP16 (half precision), the shortcut formula \(\mathbb{E}[X^2] - \mathbb{E}[X]^2\) is
dangerous. If \(X\) is large, \(X^2\) might overflow to inf in
FP16. If \(X\) has a very tight
distribution, subtracting two large, similar numbers results in
catastrophic cancellation (yielding 0 or negative variance). PyTorch’s
BatchNorm performs these accumulations in FP32 under the hood, even in
AMP.
Diagonal Covariance Assumption: In VAEs, we output a mean vector \(\mu\) and a variance vector \(\sigma^2\). We are explicitly assuming the latent space covariance matrix is diagonal (features are independent). We do this because outputting a full \(D \times D\) covariance matrix is \(O(D^2)\) parameters, and computing its log-determinant for the KL divergence is \(O(D^3)\). The diagonal assumption reduces this to \(O(D)\), at the cost of modeling feature correlations.
Interview Questions
Beginner: Q: What is the difference between variance and covariance? A: Variance measures how much a single random variable deviates from its own mean. Covariance measures how two different random variables deviate from their respective means together. Variance is always positive; covariance can be positive, negative, or zero.
Intermediate: Q: Why does the Scaled Dot-Product Attention in Transformers divide by \(\sqrt{d_k}\) instead of \(d_k\)? A: Because variance is additive. Assuming independent elements, the variance of the dot product scales linearly with \(d_k\). The standard deviation (which determines the spread of the logits) scales with \(\sqrt{d_k}\). Dividing by \(\sqrt{d_k}\) normalizes the standard deviation to 1, keeping the softmax out of saturation regions.
Advanced: Q: Explain why subtracting a baseline in Reinforcement Learning does not bias the policy gradient. A: The policy gradient is \(\nabla J = \mathbb{E}[R\nabla \log \pi]\). If we subtract a baseline \(b\) that depends only on the state (not the action), we compute \(\mathbb{E}[(R-b)\nabla \log \pi] = \mathbb{E}[R\nabla \log \pi] - \mathbb{E}[b \nabla \log \pi]\). Because \(\mathbb{E}[\nabla \log \pi] = \nabla \mathbb{E}[1] = \nabla 1 = 0\), the second term vanishes. The expectation remains unchanged, but the term \((R-b)\) is typically smaller, reducing the variance of the Monte Carlo estimate.
Research-level: Q: How does the Fisher Information Matrix relate to Covariance? A: The Fisher Information Matrix is the covariance of the score function (gradient of the log-likelihood). \(\mathbf{F} = \mathbb{E}[(\nabla_\theta \log p(x|\theta))(\nabla_\theta \log p(x|\theta))^\top]\). It measures how much the gradient of the log-likelihood varies—essentially, the “curvature” or variance of the gradient. It is used in Natural Gradient Descent to normalize steps according to distribution variance, not parameter variance.
Research Perspective
Current Limitations: Covariance only captures linear relationships. If \(Y = X^2\) and \(X\) is symmetric around 0, their covariance is exactly 0, even though they are perfectly dependent. In deep networks, features are highly non-linearly dependent, rendering standard covariance matrices incomplete descriptors of the internal representations.
Open Problems: How do we measure non-linear dependencies in high-dimensional latent spaces efficiently? How can we enforce disentanglement (zero covariance) without destroying information capacity?
Modern Improvements: Researchers are moving beyond simple covariance for measuring representation similarity. Centered Kernel Alignment (CKA) measures similarities between representations that capture both linear and non-linear relationships. In generative models, Flow Matching replaces the rigid variance-preserving constraints of standard diffusion with optimal transport paths, implicitly handling the variance schedule more elegantly.
Connections
- Prerequisites: Random Variables, Linear Algebra.
- Enables: Loss Functions (Expected Risk), Batch Normalization, Policy Gradient.
- Used by: VAEs (KL Divergence), Attention (Variance scaling), Diffusion (Noise schedules).
- Extended by: Information Theory (Entropy is expectation of log-prob), Principal Component Analysis (Eigendecomposition of covariance).
- Related concepts: Law of Large Numbers (Expectations converge), Central Limit Theorem (Sums converge to Gaussian).
Final Mental Model
Think of these three concepts as the “Camera Controls” for viewing a noisy world.
- Expectation is the Focus. It finds the stable signal hidden in the noise.
- Variance is the Exposure. It tells you how much noise and blur there is. High variance means a noisy, grainy image.
- Covariance is Stereo Vision. It tells you how the movement in your left eye correlates with the movement in your right eye, allowing you to perceive depth and structure instead of just flat noise.
Comments
Post a Comment