Topic Overview
What is a random variable? It is a mathematical translator that takes chaotic, unpredictable real-world outcomes and maps them onto the clean number line. A distribution is the rulebook that tells us how likely each of those numbers is to occur.
Why does it exist? Because determinism is a lie. In engineering, if you measure the voltage of a sensor twice, you get two different numbers. If you show an image to a human, they might label it differently on different days. Existing mathematical tools required exact inputs to yield exact outputs. We needed a framework to compute with uncertainty—to do algebra with the unknown.
What problem does it solve? It allows us to quantify, propagate, and reason about uncertainty. Without it, a model can only say, “The input is X, so the output is Y.” With it, a model can say, “The input is X, and I am 95% confident the output falls between A and B.”
Why should engineers care? Modern AI is not a calculator; it is a probabilistic reasoning engine. Neural networks do not just predict a number; they predict a distribution over numbers. Classification is modeling a Categorical distribution. Regression is modeling a Gaussian distribution. Generative AI is entirely about learning complex, high-dimensional distributions. If you do not understand random variables, you do not understand what your model is actually outputting.
Prerequisites
- Scalars: Understanding individual real numbers. This is needed because a random variable’s fundamental job is to output a scalar (or a vector of scalars) so we can do math on uncertain events.
Historical Motivation
In the 17th century, gamblers wanted to know how to bet intelligently. Deterministic math failed them; a die roll has no exact equation. Blaise Pascal and Pierre de Fermat invented probability theory to solve this.
By the 19th and 20th centuries, physicists like James Clerk Maxwell and Ludwig Boltzmann hit a wall: they couldn’t track the exact position and velocity of trillions of gas molecules. They didn’t need exact paths; they needed the distribution of speeds to derive temperature and pressure.
In ML, we face the exact same problem as the physicists. We don’t have a perfect, deterministic equation mapping pixels to the word “cat.” The data generating process is inherently noisy and stochastic. We adopted the language of random variables and distributions because it is the only mathematical framework capable of describing a world where the same input can yield different outputs.
Intuitive Explanation
Imagine a weather station. The physical world produces chaotic outcomes: sunshine, rain, snow. A Random Variable is the thermometer outside that assigns a number to that outcome: 25 for sun, 10 for rain, -5 for snow. The variable itself is random because we don’t know which outcome nature will choose before it happens.
The Distribution is the historical climate data. It tells you: “Over the last 100 years, 25 happened 40 times, 10 happened 50 times, and -5 happened 10 times.”
It is crucial to separate the variable (the mechanism that reports the number) from the distribution (the rules of likelihood). The variable is the slot machine; the distribution is the rigged gears inside that make cherries rare and lemons common.
Visual Understanding
1. The Bernoulli Distribution (The Coin): Visualize a single bar chart with two bars. A bar at \(x=0\) reaching up to height \((1-p)\), and a bar at \(x=1\) reaching up to height \(p\). If \(p=0.5\), the bars are equal. It is a binary switch.
2. The Categorical Distribution (The Roulette Wheel): Visualize a pie chart divided into \(K\) slices. The size of slice \(k\) is \(p_k\). Or, visualize a bar chart with \(K\) bars of varying heights. The sum of all bar heights is exactly 1.
3. The Gaussian Distribution (The Bell Curve): Visualize a smooth, symmetric hill. The peak of the hill is at the mean \(\mu\). The width of the hill’s base is determined by the standard deviation \(\sigma\). A small \(\sigma\) is a steep, narrow spike; a large \(\sigma\) is a flat, wide mound. The total area under the curve is exactly 1.
Mathematical Foundations
A Random Variable \(X\) is a function mapping outcomes \(\omega\) from a sample space \(\Omega\) to real numbers \(\mathbb{R}\).
We categorize them by the type of number they output: * Discrete: Outputs integers (0, 1, 2…). Described by a Probability Mass Function (PMF): \(P(X=x)\). * Continuous: Outputs any real number. Described by a Probability Density Function (PDF): \(p(x)\). (Note: \(p(x)\) is not a probability; it is a density. The probability is the area under the curve between two points).
3.1.1 Bernoulli Distribution
The simplest discrete distribution. It models a single yes/no trial. * \(X \in \{0, 1\}\) * Parameter: \(p \in [0, 1]\) (probability of success).
PMF: \(P(X=x) = p^x(1-p)^{1-x}\) * Why this equation? If \(x=1\), \(p^1(1-p)^0 = p\). If \(x=0\), \(p^0(1-p)^1 = 1-p\). It is a compact if/else statement.
Mean (Expected Value): \(\mathbb{E}[X] = 1 \cdot p + 0 \cdot (1-p) = p\). (The average outcome is just the probability). Variance: \(\text{Var}(X) = p(1-p)\). (Maximum uncertainty at \(p=0.5\); zero uncertainty at \(p=0\) or \(p=1\)).
3.1.2 Categorical Distribution
A generalization of Bernoulli to \(K\) mutually exclusive outcomes (like rolling a \(K\)-sided die). * \(X \in \{1, 2, \dots, K\}\) * Parameters: \(\mathbf{p} = [p_1, p_2, \dots, p_K]\), where \(\sum_{k=1}^K p_k = 1\).
PMF: \(P(X=k) = p_k\).
3.1.3 Gaussian (Normal) Distribution
The most important continuous distribution. It models noise, errors, and natural variation. * \(X \in \mathbb{R}\) * Parameters: Mean \(\mu \in \mathbb{R}\) (location of peak), Variance \(\sigma^2 > 0\) (spread).
PDF: \(p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)\) * \(\frac{1}{\sqrt{2\pi\sigma^2}}\): The normalization constant. It ensures the area under the curve is exactly 1. * \(\exp(\dots)\): The exponential decay. It makes points far from \(\mu\) incredibly unlikely. * \(-\frac{(x-\mu)^2}{2\sigma^2}\): The penalty term. \((x-\mu)^2\) measures distance from the mean. Dividing by \(\sigma^2\) scales the penalty by the variance. High variance = less penalty for being far away.
Multivariate Gaussian: For a vector \(\mathbf{x} \in \mathbb{R}^d\). \[p(\mathbf{x}) = \frac{1}{(2\pi)^{d/2}|\boldsymbol{\Sigma}|^{1/2}} \exp\!\left(-\frac{1}{2}(\mathbf{x}-\boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x}-\boldsymbol{\mu})\right)\] * \(\boldsymbol{\mu}\): Mean vector (shape \([d]\)). * \(\boldsymbol{\Sigma}\): Covariance matrix (shape \([d, d]\)). Defines variance in each dimension and correlation between dimensions. * \(|\boldsymbol{\Sigma}|\): Determinant of the covariance. Scales the volume of the distribution in \(d\)-dimensions. * \((\mathbf{x}-\boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x}-\boldsymbol{\mu})\): The Mahalanobis distance. A generalized squared distance that accounts for correlation.
Worked Numerical Example
1. Bernoulli: Let \(p = 0.8\). * \(P(X=1) = 0.8^1(1-0.8)^0 = 0.8\). * \(P(X=0) = 0.8^0(1-0.8)^1 = 0.2\). * Mean \(= 0.8\). Variance \(= 0.8 \times 0.2 = 0.16\).
2. Categorical: 3 classes. \(\mathbf{p} = [0.2, 0.5, 0.3]\). * \(P(X=1) = 0.2\), \(P(X=2) = 0.5\), \(P(X=3) = 0.3\).
3. Gaussian: Let \(\mu = 0, \sigma^2 = 1\). What is the density at \(x=0\) and \(x=1\)? * At \(x=0\): \(p(0) = \frac{1}{\sqrt{2\pi(1)}} \exp\!\left(-\frac{(0-0)^2}{2(1)}\right) = \frac{1}{\sqrt{2\pi}} \cdot 1 \approx 0.3989\). * At \(x=1\): \(p(1) = \frac{1}{\sqrt{2\pi}} \exp\!\left(-\frac{1}{2}\right) \approx 0.3989 \cdot 0.6065 \approx 0.2419\). (Notice \(p(0) > 1\) is impossible for probability, but perfectly valid for a probability density!)
Computational Interpretation
What is the computer actually doing?
Sampling: Generating random numbers from a distribution. Computers are deterministic, so they use Pseudo-Random Number Generators (PRNGs) that approximate randomness. * Uniform to Gaussian: To sample from \(\mathcal{N}(\mu, \sigma^2)\), the computer first samples a uniform number, applies the Box-Muller transform (or similar algorithm) to shape it into a bell curve, then scales by \(\sigma\) and shifts by \(\mu\). * Categorical Sampling: The computer creates a cumulative sum array \([p_1, p_1+p_2, p_1+p_2+p_3...]\), draws a uniform \([0,1]\) number, and performs a binary search to find which bin it falls into. \(O(\log K)\) time.
Evaluating Densities: Plugging \(x\) into the PDF equation. This requires computing exponentials and, for multivariate Gaussians, matrix determinants and inversions. * Multivariate Gaussian Cost: Inverting \(\boldsymbol{\Sigma}\) takes \(O(d^3)\) computation. This is a massive bottleneck for high-dimensional data, forcing ML engineers to use diagonal or spherical covariance matrices where \(\boldsymbol{\Sigma}\) is just a vector of variances, reducing inversion to element-wise reciprocals (\(O(d)\)).
Implementation Perspective
NumPy (From Scratch Logic)
import numpy as np
# Bernoulli Sampling
p = 0.8
# Draw uniform, threshold at p
bernoulli_sample = (np.random.rand() < p).astype(int)
# Categorical Sampling
probs = np.array([0.2, 0.5, 0.3])
# Cumulative sum: [0.2, 0.7, 1.0], find bin
categorical_sample = np.argmax(np.random.rand() < np.cumsum(probs))
# Gaussian Sampling (using built-in, which uses Box-Muller)
mu, sigma = 0.0, 1.0
gaussian_sample = np.random.normal(mu, sigma)PyTorch (Production ML)
import torch
import torch.nn.functional as F
# --- Bernoulli (Dropout) ---
x = torch.ones(5)
p = 0.5
# Creates a mask of 0s and 1s
mask = torch.bernoulli(torch.full_like(x, p))
# Inverted dropout: scale by 1/p so expected value remains unchanged during training
x_dropped = x * mask / p
# --- Categorical (LLM Next-Token) ---
logits = torch.randn(1, 100) # Raw scores for 100 tokens
temperature = 0.7
# Convert scores to probabilities
probs = F.softmax(logits / temperature, dim=-1)
# Sample one token index
next_token = torch.multinomial(probs, num_samples=1)
# --- Gaussian (VAE Reparameterization Trick) ---
mu = torch.randn(1, 32)
log_var = torch.randn(1, 32) # Network outputs log variance for numerical stability
std = torch.exp(0.5 * log_var) # sigma = exp(0.5 * log(sigma^2))
eps = torch.randn_like(std) # Sample from N(0, I)
# This specific formulation allows gradients to flow back through mu and log_var!
z = mu + eps * std # z ~ N(mu, std^2)Where It Appears In AI
| AI Component | Exact Usage of Random Variables & Distributions |
|---|---|
| Linear Layer | Weight initialization: Parameters drawn from \(\mathcal{N}(0, \sigma^2)\) (He/Xavier) to prevent vanishing/exploding gradients. |
| Dropout | Bernoulli random variable applied to activations. Each neuron is a coin flip. |
| Transformer Output | Categorical distribution over the vocabulary at every timestep. |
| VAE Latent Space | Multivariate Gaussian \(\mathcal{N}(\mathbf{0}, \mathbf{I})\) prior; encoder outputs parameters \(\mu, \sigma\) of approximate posterior. |
| Diffusion Models | Forward process adds Gaussian noise \(\mathcal{N}(0, \beta_t \mathbf{I})\); reverse process learns to predict that noise. |
| BatchNorm | Normalizes batch features to approximate a standard Gaussian \(\mathcal{N}(0, 1)\). |
| Reinforcement Learning | Policy networks output Categorical distributions (discrete actions) or Gaussian distributions (continuous actions). |
Deep Dive Into Real Models
1. LLMs (Transformers) An LLM does not output a word; it outputs the parameters of a Categorical distribution. For a vocabulary of size \(V\), the final linear layer outputs \(V\) logits. The softmax function converts these into \(\mathbf{p} = [p_1, \dots, p_V]\). During training, we maximize the log-probability of the correct token. During inference, we sample from this distribution using techniques like Top-K or Nucleus Sampling, which zero out probabilities of unlikely tokens before sampling, preventing the model from spouting gibberish.
2. VAEs (Variational Autoencoders) A deterministic autoencoder maps an image to a single point in latent space. A VAE maps an image to a Gaussian distribution in latent space. The encoder outputs a mean vector \(\mu\) and a variance vector \(\sigma^2\). To generate an image, we sample \(z \sim \mathcal{N}(\mu, \sigma^2)\) and pass it to the decoder. The reparameterization trick (\(z = \mu + \sigma \cdot \epsilon\), where \(\epsilon \sim \mathcal{N}(0,1)\)) is the only reason backpropagation can train this model; it separates the parameters (\(\mu, \sigma\)) from the randomness (\(\epsilon\)).
3. Diffusion Models Diffusion is a hierarchy of Gaussians. The forward process takes a clean image \(x_0\) and gradually adds Gaussian noise over \(T\) steps: \(q(x_t | x_{t-1}) = \mathcal{N}(\sqrt{1-\beta_t}x_{t-1}, \beta_t \mathbf{I})\). Eventually, \(x_T\) is pure isotropic Gaussian noise. The U-Net is trained to reverse this: given a noisy image, predict the noise \(\epsilon\) that was added. The variance \(\beta_t\) is scheduled to ensure smooth degradation.
Failure Modes
1. Confusing Probability Density with Probability *
Cause: Forgetting that continuous distributions yield \(p(x) = 0\) for any exact point. \(p(x)\) is a density (height), not a
probability (area). * Symptoms: Wondering how \(p(x)\) can be greater than 1 (e.g., \(p(0)\) for \(\mathcal{N}(0, 0.1)\) is ~1.26). Writing
code that checks if pdf > 1. * Solution: Always
integrate continuous distributions over an interval to get probability.
Use log-probabilities for calculations.
2. The Reparameterization Gradient Break *
Cause: Sampling is a non-differentiable operation. If you write
z = torch.normal(mu, sigma), PyTorch cannot route gradients
back to mu and sigma because the random
sampling operation breaks the computation graph. * Symptoms:
VAE latent layers have no gradients; encoder weights don’t update. *
Solution: Always use the reparameterization trick:
z = mu + sigma * torch.randn_like(sigma). Gradients flow
safely through the mu and sigma additions.
3. Numerical Underflow in Logits * Cause:
Softmax computes \(e^{z_i} / \sum
e^{z_j}\). If logits are large (e.g., 1000), \(e^{1000}\) is inf. If they are
very negative, \(e^{-1000}\) underflows
to 0. * Symptoms: NaN losses during
training. * Solution: PyTorch’s CrossEntropyLoss
combines LogSoftmax and NLLLoss. It subtracts max(logits)
before applying exp internally to ensure numerical
stability. Never pass softmax outputs into a log-loss function manually;
use the combined function.
Engineering Insights
Log-Space is King: Computing the joint probability of 100 events means multiplying 100 numbers \(< 1\). The result is \(10^{-100}\), which underflows to zero in float32. Engineers compute everything in log-space: \(\log(p_1 \cdot p_2) = \log p_1 + \log p_2\). Addition does not underflow. Neural networks output log-probabilities (logits) natively for this reason.
Temperature Scaling: In categorical sampling, dividing logits by a temperature \(T\) before softmax controls the entropy of the distribution. * \(T \to 0\): The distribution becomes a Dirac delta (greedy argmax). Deterministic. * \(T = 1\): Original distribution. * \(T \to \infty\): Uniform distribution. Maximum randomness. This is an engineering dial to trade off between determinism (factual accuracy) and diversity (creativity) in LLMs.
Diagonal Covariance: In a 1024-dimensional space, a full covariance matrix \(\boldsymbol{\Sigma}\) has \(\approx 500,000\) parameters and requires \(O(1024^3)\) operations to invert. In ML, we almost exclusively use diagonal covariances (a vector of 1024 variances). It assumes dimensions are independent, reducing the parameter count to 1024 and inversion to element-wise division. The cost is ignoring correlations, which VAEs and Diffusion models accept as a necessary tradeoff.
Interview Questions
Beginner: Q: What is the difference between a probability mass function and a probability density function? A: A PMF maps discrete outcomes to probabilities (values between 0 and 1 that sum to 1). A PDF maps continuous outcomes to densities (values \(\ge 0\) that integrate to 1). A PDF can be greater than 1; a PMF cannot.
Intermediate: Q: Why does the VAE
reparameterization trick work? Why not just backpropagate through the
sample() function? A: Sampling is a stochastic,
non-differentiable process. The gradient of a random sample with respect
to the distribution parameters is undefined. The reparameterization
trick moves the randomness outside the computation graph. Instead of
sampling \(z \sim \mathcal{N}(\mu,
\sigma^2)\), we sample noise \(\epsilon
\sim \mathcal{N}(0,1)\) and compute \(z
= \mu + \sigma \cdot \epsilon\). The function \(f(\mu, \sigma, \epsilon) = \mu + \sigma \cdot
\epsilon\) is fully deterministic and differentiable with respect
to \(\mu\) and \(\sigma\).
Advanced: Q: What is the Maximum Entropy principle, and why does it make the Gaussian distribution the default assumption for noise in ML? A: The Maximum Entropy principle states that, given a set of constraints (like a known mean and variance), you should choose the distribution with the highest entropy (most uncertainty). It makes the fewest assumptions beyond the constraints. The Gaussian is the maximum entropy distribution for a continuous variable with fixed mean and variance. Therefore, if we only know the mean and variance of our noise, assuming Gaussian is the mathematically safest, least presumptuous choice.
Research-level: Q: How does the Gumbel-Softmax trick allow gradients to flow through discrete Categorical sampling? A: Discrete sampling (argmax of \(p_k + \text{Gumbel noise}\)) is non-differentiable. Gumbel-Softmax replaces the hard argmax with a continuous softmax with a temperature parameter. As \(T \to 0\), the output becomes a one-hot vector (discrete), but for \(T > 0\), it is a continuous, differentiable approximation. This allows backpropagation through what is essentially a discrete sampling step, enabling training in areas like discrete VAEs.
Research Perspective
Current Limitations: Modeling complex, high-dimensional distributions (like images) with simple parametric distributions (like Gaussians) is impossible. VAEs and Normalizing Flows warp simple distributions into complex ones, but often suffer from blurry generation (VAEs) or severe architectural constraints (Flows).
Open Problems: How can we efficiently represent and sample from multi-modal distributions in high-dimensional spaces? Diffusion models currently solve this by iterating denoising steps, but this is computationally expensive.
Modern Improvements: Flow Matching (or Rectified Flow) is replacing standard Diffusion. Instead of mapping data to noise via a stochastic Gaussian process, Flow Matching constructs a deterministic vector field (an ODE) that transports the data distribution straight to a Gaussian. It yields straighter paths during sampling, requiring fewer steps than the curved paths of traditional stochastic differential equation (SDE) diffusion models.
Connections
- Prerequisites: Scalars.
- Enables: Loss Functions (Cross-Entropy is log of Categorical; MSE is log of Gaussian), Generative AI.
- Used by: VAEs (Gaussian/Categorical latent spaces), Diffusion (Gaussian noise), LLMs (Categorical outputs), RL (Policy distributions).
- Extended by: Information Theory (Entropy, KL Divergence), Probabilistic Graphical Models.
- Related concepts: Monte Carlo Sampling, Central Limit Theorem.
Final Mental Model
Think of a Random Variable as a programmable slot machine, and the Distribution as the microchip controlling its reels.
- A Bernoulli microchip is a coin: only two symbols, you set the weight on one side.
- A Categorical microchip is a weighted roulette wheel: you set the size of each slice.
- A Gaussian microchip is a dartboard: you set where the bullseye is (\(\mu\)) and how scattered the throws are (\(\sigma\)).
When you build an AI model, you aren’t just outputting numbers; you are designing microchips. The neural network’s final layer is just a factory that manufactures the parameters (\(p, \mathbf{p}, \mu, \sigma\)) for these slot machines based on the input data.
Comments
Post a Comment