Why Correlation is Lying to You (Mutual Information)

Topic Overview

What is Mutual Information?

Mutual Information (MI) is the exact, mathematically rigorous measurement of how much knowing one variable reduces your uncertainty about another. If I tell you it is raining, your uncertainty about whether people are carrying umbrellas drops to near zero. That drop in uncertainty is Mutual Information.

Why does it exist?

In engineering, we constantly need to measure relationships between variables. Historically, we used correlation (Pearson). But correlation only detects linear relationships. If \(Y = X^2\), correlation says they are unrelated (zero). Mutual Information exists because we needed a metric that detects any relationship—linear, non-linear, categorical, or chaotic—based purely on probability distributions.

What problem does it solve?

It solves the problem of quantifying “dependency” in complex systems. It tells you exactly how many bits of information are shared between two signals. In modern AI, it solves the problem of forcing neural networks to learn useful features without labels, serving as the mathematical backbone of self-supervised learning.

Why should engineers care?

If you train a vision model using SimCLR or MoCo, you are optimizing Mutual Information. If you build a tabular model and use mutual information for feature selection, you are directly using this math. If you don’t understand MI, you cannot understand why contrastive learning works, nor can you debug representations that collapse into uselessness.

Prerequisites

To build Mutual Information from scratch, we need two building blocks:

1. Entropy (\(H(X)\))

Why it is needed: Entropy is the baseline uncertainty of a system. Mutual Information is defined as the reduction in this entropy. Without knowing how uncertain you were to begin with, you cannot measure how much less uncertain you became.
Where it appears: It is the \(H(X)\) in the equation \(I(X;Y) = H(X) - H(X|Y)\).

2. Conditional Probability (\(p(x|y)\))

Why it is needed: To measure how much \(Y\) tells us about \(X\), we must update our beliefs about \(X\) after observing \(Y\). This updated state is exactly conditional probability, and its uncertainty is Conditional Entropy \(H(X|Y)\).
Where it appears: It dictates the shape of the joint distribution \(p(x,y)\), which is the core engine of the MI calculation.

Historical Motivation

In 1948, Claude Shannon invented Information Theory to solve a specific engineering problem: how to transmit messages over a noisy channel without loss.

The problem was: How do we mathematically define the “capacity” of a wire? If you send a 1, and noise flips it to a 0 half the time, the output is dependent on the input, but corrupted. Shannon needed a way to measure the maximum amount of useful information that slips through the noise.

He couldn’t use correlation because the encoding might be non-linear (e.g., frequency modulation). He derived Mutual Information to measure the shared information between the transmitted signal (\(X\)) and the received signal (\(Y\)). It became the absolute theoretical speed limit of all communication systems—from copper wires to fiber optics to 5G.

Intuitive Explanation

Imagine two streams of data: * Stream X: The daily temperature in a city. * Stream Y: The daily ice cream sales in that city.

If I give you a locked box containing all the temperature data, how much can you guess about the ice cream sales? You might have some baseline guess (average sales), but high uncertainty.

Now, I unlock the box and let you read the temperatures. * If you see “95°F”, your uncertainty about ice cream sales plummets. You know sales are high. * If you see “20°F”, your uncertainty also plummets. You know sales are near zero.

Mutual Information is the total amount of guessing power you gained by unlocking that box.

What if Stream Y was “The number of stars in the universe”? Unlocking the temperature box gives you zero guessing power about the stars. The uncertainty does not drop. Therefore, \(I(X; Y) = 0\). They are independent.

Visual Understanding

Draw a Venn Diagram with two overlapping circles.

Left Circle (\(H(X)\)): The total uncertainty of variable X. The area represents the bits needed to encode X.
Right Circle (\(H(Y)\)): The total uncertainty of variable Y.
The Overlapping Region (\(I(X;Y)\)): This is Mutual Information. It is the information that is perfectly redundant—if you know it from X, you don’t need to learn it from Y.
Left Crescent (\(H(X|Y)\)): The uncertainty remaining in X after you know Y. Conditional Entropy.

Visualizing the Math: If the circles do not overlap at all, \(I(X;Y) = 0\). Independence. If the left circle is entirely inside the right circle, \(I(X;Y) = H(X)\). Knowing Y tells you absolutely everything about X.

Mathematical Foundations

Let’s formalize the Venn diagram.

Definition 1: The Reduction in Uncertainty \[I(X; Y) = H(X) - H(X|Y)\] * \(H(X)\): Bits needed to encode X blind. * \(H(X|Y)\): Bits needed to encode X when you already have Y. * Result: The difference is the bits you saved.

Definition 2: Symmetry Because the shared information is the same regardless of which side you view it from: \[I(X; Y) = H(X) - H(X|Y) = H(Y) - H(Y|X)\]

Definition 3: The Information-Theoretic Form (The most profound) Using the chain rule of entropy, we can expand \(H(X,Y) = H(X) + H(X|Y)\). Rearranging Definition 1: \[I(X; Y) = H(X) + H(Y) - H(X, Y)\]

Now, apply the definition of Entropy: \[I(X; Y) = \sum_x p(x)\log\frac{1}{p(x)} + \sum_y p(y)\log\frac{1}{p(y)} - \sum_{x,y} p(x,y)\log\frac{1}{p(x,y)}\]

Combine the logarithms: \[I(X; Y) = \sum_{x,y} p(x,y) \log \left( \frac{p(x,y)}{p(x)p(y)} \right)\]

The Deep Meaning of the Ratio: Look at the fraction \(\frac{p(x,y)}{p(x)p(y)}\). * \(p(x)p(y)\) is the joint distribution if X and Y were independent. * \(p(x,y)\) is the actual joint distribution. * Therefore, Mutual Information is exactly the KL Divergence between the true joint distribution and the assumed independent distribution. \[I(X; Y) = D_{KL}[p(x,y) \| p(x)p(y)]\]

This proves: MI is zero if and only if X and Y are truly independent. If they are even slightly dependent, the distributions differ, and MI \(> 0\).

Worked Numerical Example

Let’s calculate MI for a simple medical test. * Disease (\(X\)): 10% of population has it (\(D\)), 90% do not (\(\neg D\)). * Test (\(Y\)): Positive (\(+\)) or Negative (\(-\)).

Suppose the test is pretty good: * True Positive Rate: \(P(+|D) = 0.8\) * False Positive Rate: \(P(+|\neg D) = 0.1\)

Step 1: Calculate Joint Probabilities \(P(X, Y)\) * \(P(D, +) = P(D) \cdot P(+|D) = 0.10 \cdot 0.8 = 0.08\) * \(P(D, -) = 0.10 \cdot 0.2 = 0.02\) * \(P(\neg D, +) = 0.90 \cdot 0.1 = 0.09\) * \(P(\neg D, -) = 0.90 \cdot 0.9 = 0.81\) (Check: they sum to 1.0)

Step 2: Calculate Marginals \(P(X)\) and \(P(Y)\) * \(P(D) = 0.10\), \(P(\neg D) = 0.90\) * \(P(+) = 0.08 + 0.09 = 0.17\) * \(P(-) = 0.02 + 0.81 = 0.83\)

Step 3: Calculate MI using \(I(X;Y) = \sum \sum p(x,y) \log_2 \frac{p(x,y)}{p(x)p(y)}\)

We only calculate where \(p(x,y) > 0\): 1. Sick & Positive: \(0.08 \cdot \log_2(\frac{0.08}{0.10 \cdot 0.17}) = 0.08 \cdot \log_2(\frac{0.08}{0.017}) \approx 0.08 \cdot \log_2(4.706) \approx 0.08 \cdot 2.233 = \mathbf{0.179}\) 2. Sick & Negative: \(0.02 \cdot \log_2(\frac{0.02}{0.10 \cdot 0.83}) = 0.02 \cdot \log_2(\frac{0.02}{0.083}) \approx 0.02 \cdot \log_2(0.241) \approx 0.02 \cdot (-2.055) = \mathbf{-0.041}\) 3. Healthy & Positive: \(0.09 \cdot \log_2(\frac{0.09}{0.90 \cdot 0.17}) = 0.09 \cdot \log_2(\frac{0.09}{0.153}) \approx 0.09 \cdot \log_2(0.588) \approx 0.09 \cdot (-0.767) = \mathbf{-0.069}\) 4. Healthy & Negative: \(0.81 \cdot \log_2(\frac{0.81}{0.90 \cdot 0.83}) = 0.81 \cdot \log_2(\frac{0.81}{0.747}) \approx 0.81 \cdot \log_2(1.084) \approx 0.81 \cdot 0.119 = \mathbf{0.096}\)

Total MI: \(0.179 - 0.041 - 0.069 + 0.096 = \mathbf{0.165}\) bits.

Interpretation: Observing the test result reduces your uncertainty about the disease status by 0.165 bits.

Computational Interpretation

What happens when we try to compute this in a neural network?

The Discrete Bottleneck: In the example above, we had 2 states for X and 2 for Y. We calculated 4 probabilities. This is trivial for a CPU. But what if X is a 256x256 image and Y is a 256x256 augmented image? The state space is \((256 \times 256 \times 255)^2\). It is computationally impossible to build a histogram to find \(p(x,y)\).

The Continuous Trap: Images are continuous. The equation becomes an integral: \[I(X;Y) = \iint p(x,y) \log \frac{p(x,y)}{p(x)p(y)} dx dy\] We do not know the true probability density functions \(p(x)\) or \(p(x,y)\) for high-dimensional images. We only have samples.

The Engineering Reality: Calculating exact MI in deep learning is strictly impossible for high-dimensional data. Therefore, we never compute Mutual Information directly in deep learning. Instead, the entire field of self-supervised learning is built on computing lower bounds on Mutual Information.

Implementation Perspective

Because we cannot compute MI directly, PyTorch does not have a torch.functional.mutual_information for high-dimensional vectors.

Instead, we implement InfoNCE, the most famous lower bound on MI, used in SimCLR and CLIP.

The Math of InfoNCE: If we have one positive pair (an image and its crop) and \(N-1\) negative pairs (random other images), the lower bound is: \[I(X;Y) \ge \log N - \mathcal{L}_{InfoNCE}\]

PyTorch Implementation: InfoNCE

import torch
import torch.nn.functional as F

def info_nce_loss(query, positive_key, negative_keys, temperature=0.1):
    """
    query: [Batch, Dim] - Representation of view 1
    positive_key: [Batch, Dim] - Representation of view 2 (same image)
    negative_keys: [Batch, N-1, Dim] - Representations of other images in batch
    """
    # Calculate positive similarity: [Batch, 1]
    # Cosine similarity scaled by temperature
    pos_sim = F.cosine_similarity(query, positive_key, dim=-1, eps=1e-8) / temperature
    pos_sim = pos_sim.unsqueeze(1) 

    # Calculate negative similarities: [Batch, N-1]
    # query is [B, D], negative_keys is [B, N-1, D]. We use einsum for efficiency.
    neg_sim = torch.einsum('bd,bnd->bn', query, negative_keys) / temperature

    # Concatenate to get all logits: [Batch, N]
    # Index 0 is the positive, 1 to N-1 are negatives
    logits = torch.cat([pos_sim, neg_sim], dim=1)

    # Labels: The positive is always at index 0
    labels = torch.zeros(query.size(0), dtype=torch.long, device=query.device)

    # Cross-entropy loss maximizes the positive logit
    loss = F.cross_entropy(logits, labels)
    return loss

Why this works: By pushing the similarity of the positive pair high and the negatives low, we are implicitly maximizing the Mutual Information between the two views of the image.

Where It Appears In AI

AI Component	Exact Usage
Contrastive Learning (SimCLR, MoCo)	InfoNCE loss acts as a differentiable lower bound on MI between two augmented views of the same image.
Deep InfoMax (DIM)	Maximizes MI between global image features (output of a CNN) and local image patches, forcing the network to understand spatial relationships.
**Tabular Feature Selection	Used via `sklearn.feature_selection.mutual_info_classif`. Calculates discrete MI between categorical/continuous features and the target label to drop useless columns.
Information Bottleneck Theory	The theory that deep learning is fundamentally a process of maximizing \(I(Z; Y)\) (representation retains label info) while minimizing \(I(Z; X)\) (representation forgets input noise).
MINE (Mutual Information Neural Estimation)	A specific neural network architecture trained to estimate the KL divergence form of MI for complex distributions in RL and generative models.

Deep Dive Into Real Models

SimCLR (Contrastive Learning)

SimCLR takes a batch of \(N\) images. It creates two random cropped/augmented versions of each image, resulting in \(2N\) images. It passes them through a massive ResNet to get \(2N\) embedding vectors.

The Goal: Maximize the MI between the two views of the same image.
The Mechanism: For image \(i\), view 1 is the “query”. View 2 is the “positive key”. The other \(2N-2\) images are “negative keys”.
The Loss: InfoNCE.
Tensor Shapes:
- Queries: [2N, 128] (Embedding dim 128)
- Keys: [2N, 128]
- Similarity Matrix: [2N, 2N] (computed via matrix multiplication).
- The diagonal (offset by batch size) contains the positive pairs. The rest are negatives.
Why MI? If the model maps two completely different images to the same vector, the negative similarities rise, InfoNCE loss increases, and the gradient punishes the model. MI acts as the ultimate arbitrer of “sameness.”

Information Bottleneck in LLMs

Consider a Transformer layer mapping input tokens \(X\) to a hidden representation \(Z\), which predicts the next token \(Y\). Theoretical work suggests the network inherently solves: \[\max I(Z; Y) - \beta I(Z; X)\] * \(I(Z; Y)\): Ensure the hidden state remembers the meaning needed for the next word. * \(I(Z; X)\): Destroy the exact pixel/token-level details of the input (e.g., exact wording) to generalize better (e.g., understand the concept).

Failure Modes

1. Representation Collapse (Maximizing MI without Negatives)

Cause: If you try to explicitly maximize \(I(X;Y)\) between two views using a neural network, and you don’t use negative samples (like in InfoNCE), the network can cheat.
Symptom: The encoder maps every single image to the exact same constant vector (e.g., [1, 1, 1, ...]).
Why it happens: If the output is constant, \(H(Z) = 0\). But \(H(Z|X) = 0\) (because it’s a deterministic function). Therefore \(I(X;Z) = H(Z) - H(Z|X) = 0 - 0 = 0\). Wait, no. Actually, if it maps to a constant, the joint distribution \(p(x,z)\) collapses. Without negative constraints, the network takes the path of least resistance and collapses.
Solution: This exact failure mode is why Contrastive Learning (InfoNCE) was invented. The negative samples provide the necessary pressure to prevent collapse.

2. The “Curse of Dimensionality” in Estimation

Cause: Trying to estimate MI using k-nearest neighbor methods (like Kraskov estimation) on high-dimensional CNN features.
Symptom: The estimated MI value plateaus or becomes highly noisy regardless of actual relationship.
Solution: Abandon direct estimation. Use neural lower bounds (InfoNCE, MINE) or switch to alternative metrics (like Barlow Twins, which avoids MI entirely by using invariance and covariance redundancy reduction).

Engineering Insights

Tradeoff: Batch Size vs. MI Bound Quality In InfoNCE, the lower bound is \(\log N - \mathcal{L}\). * If your batch size \(N\) is 32, your theoretical maximum MI bound is \(\log_2(32) = 5\) bits. * If your batch size is 4096, your maximum bound is 12 bits. * Insight: Small models struggle with contrastive learning not just because of capacity, but because small batch sizes provide a terribly weak MI bound. This is why MoCo uses a massive memory queue of past negatives—to artificially inflate \(N\) without needing a massive GPU.

GPU Memory vs. Computation Computing the full similarity matrix for InfoNCE is \(O(N^2 \cdot D)\). For \(N=4096\) and \(D=128\), this requires allocating a \(4096 \times 4096\) matrix in FP32 (64 MB). For massive batches, this causes Out-Of-Memory (OOM) errors, requiring gradient checkpointing or ring-buffer tricks.

Interview Questions

Beginner

Q: What does it mean if the Mutual Information between two variables is exactly 0? A: It means the two variables are strictly independent. Knowing the state of variable Y provides absolutely zero reduction in uncertainty about variable X. (Note: \(I(X;Y)=0\) implies independence, but \(I(X;Y)\) close to 0 does not necessarily mean they are independent; they could have complex non-linear dependencies that require more data to resolve).

Intermediate

Q: Why is Mutual Information preferred over Pearson Correlation for feature selection in non-linear ML models? A: Pearson correlation only measures linear relationships (\(Y = mX + c\)). If the relationship is parabolic (\(Y = X^2\)) or circular, Pearson will report 0, suggesting the feature is useless. Mutual Information measures any statistical dependency, linear or non-linear, because it relies on probability distributions, not just variance and covariance.

Advanced

Q: Why do we use InfoNCE instead of calculating exact Mutual Information in Contrastive Learning? A: Exact MI requires knowing the true joint probability density function \(p(x,y)\) and marginals \(p(x), p(y)\). For high-dimensional continuous data like images, estimating these densities is mathematically intractable (curse of dimensionality). InfoNCE provides a differentiable, computationally tractable lower bound on MI that only requires comparing similarities among a finite batch of samples.

Research

Q: What is the Information Bottleneck principle, and how does it relate to deep learning generalization? A: The IB principle states that a good model should compress the input \(X\) into a representation \(Z\) that is maximally informative about the target \(Y\), while discarding all information in \(X\) that is irrelevant to \(Y\) (minimizing \(I(X;Z)\)). Theoretically, deep neural networks are argued to go through two phases: fitting phase (maximizing \(I(Z;Y)\)) and compression phase (minimizing \(I(X;Z)\)), which explains why overparameterized networks don’t always overfit.

Research Perspective

Current Limitations: The biggest open wound in MI for deep learning is estimation. InfoNCE is a very loose lower bound. It requires massive batch sizes (thousands of negative samples) to approach the true MI value. This makes training incredibly memory-intensive.

Modern Directions: 1. Moving Away from MI: Because MI bounds are so noisy and expensive, modern self-supervised learning (like BYOL, SimSiam, Barlow Twins) has largely abandoned explicit MI maximization. Instead, they use architecture tricks (stop-gradients, asymmetry) or covariance matrices to achieve the effect of high MI (good representations) without computing the bound. 2. FLOPs-based bounds: Research into bounds that scale linearly \(O(N)\) rather than quadratically \(O(N^2)\) with batch size, to democratize contrastive learning for smaller labs.

Connections

Prerequisites: * Entropy (Baseline uncertainty) * Conditional Probability (Updating beliefs) * KL Divergence (The mathematical mechanism of MI)

Enables: * Contrastive Learning (SimCLR, MoCo, CLIP) * Information Bottleneck theory * Representation Learning

Used by: * Self-Supervised Learning frameworks * Feature selection pipelines (Scikit-learn) * Causal Inference frameworks

Extended by: * Conditional Mutual Information \(I(X;Y|Z)\) (Used in causal discovery) * InfoNCE / NT-Xent Loss (Practical implementations)

Related concepts: * Pearson/Spearman Correlation (Linear alternatives) * Cross-Correlation (Signal processing alternative) * Redundancy Reduction (Goal of MI minimization)

Final Mental Model

To permanently remember Mutual Information, visualize the “Information Venn Diagram with a Lock.”

Imagine two locked safes, X and Y. * Entropy is the number of combinations you have to try to open safe X blind. * Mutual Information is the number of combinations you don’t have to try if someone hands you the contents of safe Y. * If the safes contain completely random, unrelated garbage, handing you safe Y does nothing. MI = 0. * If safe Y contains a piece of paper that literally says “The combination to X is 45-12-89”, then handing you safe Y eliminates all uncertainty. MI is maximum.

In deep learning, we don’t have the combination. We just have a neural network trying to arrange its weights such that the overlap in the Venn diagram between the input and the target is as large as possible.

Neural Nexus

Search This Blog