Skip to main content

Why Every Loss Function is Actually a Probability

Why Every Loss Function is Actually a Probability

Topic Overview

Every time you train a neural network, you are using Maximum Likelihood Estimation (MLE) or Maximum A Posteriori (MAP) inference. You might not know it, but the loss functions you use every day—Cross-Entropy, Mean Squared Error—are not just arbitrary mathematical punishments applied to your model. They are exact, mathematically derived consequences of MLE.

MLE and MAP are the foundational frameworks for parameter estimation in statistics and machine learning. They answer the most fundamental question in AI: “Given the data I just saw, what should my model’s parameters be?”

Why does it exist? It exists to provide a principled, mathematical bridge between a probabilistic model of the world and the specific numbers (weights) that model should use. Without it, choosing weights would be a matter of guessing.

What problem does it solve? It turns the vague goal of “make the model accurate” into a rigorous optimization problem: “find the parameters that make the observed data most probable.”

Why should engineers care? Because if you don’t understand MLE and MAP, you don’t truly understand why your loss function is CrossEntropyLoss and not something else. You won’t understand why Weight Decay (L2 regularization) prevents overfitting, and you will be helpless when trying to design custom loss functions for novel architectures like VAEs or Diffusion Models.

Prerequisites

Concept Why It Is Needed Where It Appears
Probability Understanding \(P(y \mid x)\) is required to define the likelihood function. The core of the likelihood definition.
Conditional Probability MLE conditions the parameters on the data. MAP uses Bayes’ Theorem to add priors. The transition from MLE to MAP.
Logarithms Logarithms turn products into sums, preventing underflow and simplifying derivatives. The log-likelihood formulation.
Calculus (Derivatives) Finding the maximum of a function requires taking its derivative and setting it to zero. The optimization step (\(\arg\max\)).

Historical Motivation

In the early 20th century, statisticians like R.A. Fisher faced a crisis. They had probabilistic models (like the Gaussian distribution), but they needed a rigorous, universal method to estimate the parameters of those models (like the mean and variance) from limited, noisy data.

Existing methods, like the Method of Moments, worked but lacked a deep theoretical justification for why they were the “best” estimates. Fisher proposed MLE: the best estimate for a parameter is the one that makes the observed data most likely to have occurred.

Engineering necessity: In deep learning, we don’t just want a model that “fits” the data; we want the model that maximizes the probability of generating the exact dataset we have. MLE provides the mathematical guarantee that minimizing certain loss functions is mathematically identical to maximizing the probability of the truth.

Intuitive Explanation

Imagine you find a biased coin on the ground. You flip it 10 times, and it lands on Heads 7 times. What is the probability that this coin lands on Heads?

  • Is it 0.5? No, the data suggests otherwise.
  • Is it 1.0? No, it landed on tails 3 times.
  • Is it 0.7? Yes.

Why 0.7? Because if the true probability of Heads was 0.7, then seeing 7 Heads out of 10 flips is the most probable outcome. This is MLE: you pick the parameter (\(\theta = 0.7\)) that makes the observed data (7 Heads) look as likely as possible.

Now, imagine MAP (Maximum A Posteriori). Before you flipped the coin, you found it in a regular wallet. Coins in wallets are almost always fair (50/50). You have a prior belief. If you flip it 10 times and get 7 Heads, do you suddenly believe the coin is exactly 70% biased? Probably not. Your prior belief pulls your estimate back toward 0.5. You might conclude the true bias is 0.65.

MLE trusts only the data. MAP trusts the data, but pulls the estimate toward your prior beliefs.

Visual Understanding

Draw this: A 2D plot. The x-axis is the parameter value \(\theta\) (e.g., the probability of Heads, from 0 to 1). The y-axis is the “Probability of the Data” (Likelihood).

There is a bell curve on this plot. The peak of this bell curve is at \(\theta = 0.7\). - MLE is a vertical line dropping straight down from this peak. It says, “I only care about the peak.”

Now, draw another curve on the same plot, centered at \(\theta = 0.5\). This is your Prior. It represents your belief that fair coins are common.

If we multiply the Likelihood curve by the Prior curve, we get a new curve: the Posterior. Because the Prior is pulling the distribution to the left, the peak of the Posterior will not be at 0.7. It will be somewhere between 0.5 and 0.7—say, 0.65. - MAP is a vertical line dropping from the peak of the Posterior. It says, “I care about the peak, but I let the prior pull me back.”

Mathematical Foundations

1. The Likelihood Function

For a dataset \(\mathcal{D}\) with \(N\) independent samples, the likelihood of the parameters \(\theta\) given the data is the joint probability of observing all \(y^{(i)}\) given all \(\mathbf{x}^{(i)}\):

\[\mathcal{L}(\theta; \mathcal{D}) = \prod_{i=1}^N p(y^{(i)} \mid \mathbf{x}^{(i)}; \theta)\]

Explain every symbol: - \(\mathcal{L}(\theta; \mathcal{D})\): The likelihood. Note that this is not \(P(\theta \mid \mathcal{D})\). It is a function of \(\theta\), treating the data as fixed. - \(\prod\): Product. Because samples are independent, the joint probability is the product of individual probabilities. - \(p(y^{(i)} \mid \mathbf{x}^{(i)}; \theta)\): The probability of output \(y\) given input \(\mathbf{x}\) under parameters \(\theta\). The semicolon separates the parameters from the variables.

Why does this equation exist? To formalize “how likely was this data?”

2. The Log-Likelihood

Multiplying thousands of probabilities (numbers between 0 and 1) results in floating-point underflow (it becomes 0.0). We take the natural logarithm to turn products into sums:

\[\ell(\theta) = \log \mathcal{L}(\theta) = \sum_{i=1}^N \log p(y^{(i)} \mid \mathbf{x}^{(i)}; \theta)\]

Because \(\log\) is a monotonically increasing function, maximizing \(\mathcal{L}\) is identical to maximizing \(\ell\).

3. MLE Definition

\[\hat{\theta}_{MLE} = \arg\max_\theta \ell(\theta)\]

We want the parameters \(\theta\) that maximize the log-likelihood. In deep learning, we minimize negative log-likelihood (NLL), because optimizers are built to minimize loss.

4. MAP Definition

Using Bayes’ Theorem: \(P(\theta \mid \mathcal{D}) = \frac{P(\mathcal{D} \mid \theta)P(\theta)}{P(\mathcal{D})}\). Taking the log and dropping the denominator (since it doesn’t depend on \(\theta\)):

\[\hat{\theta}_{MAP} = \arg\max_\theta \left[\ell(\theta) + \log p(\theta)\right]\]

Explain every symbol: - \(\ell(\theta)\): The log-likelihood (the data’s voice). - \(\log p(\theta)\): The log-prior (your belief’s voice).

Derivations: How MLE/MAP Create Deep Learning Loss Functions

This is the most critical section for an ML engineer. We will derive the standard DL losses directly from MLE and MAP.

Derivation 1: Cross-Entropy Loss from MLE

Suppose we have a classification model. For a single sample, the true label is a one-hot vector \(\mathbf{y}\), and the model outputs a probability vector \(\hat{\mathbf{y}}\). The probability of the true label under the model is: \[p(y \mid \mathbf{x}; \theta) = \prod_{k=1}^K \hat{y}_k^{y_k}\] (If the true class is 2, then \(y_2=1\), so this resolves to \(\hat{y}_2\)).

Take the log: \[\log p(y \mid \mathbf{x}; \theta) = \sum_{k=1}^K y_k \log \hat{y}_k\] Sum over all \(N\) samples to get the total log-likelihood \(\ell(\theta)\). Since we minimize negative log-likelihood in deep learning, our Loss function \(\mathcal{L}_{CE}\) is: \[\mathcal{L}_{CE} = -\ell(\theta) = -\sum_{i=1}^N \sum_{k=1}^K y_k^{(i)} \log \hat{y}_k^{(i)}\] This is exactly the Cross-Entropy Loss. Cross-Entropy is not an arbitrary formula; it is the mathematical equivalent of saying “maximize the probability of the true labels.”

Derivation 2: MSE Loss from MLE

Suppose we are predicting a continuous value. We assume the true output has Gaussian noise: \(y \sim \mathcal{N}(\hat{y}, \sigma^2)\). The PDF of a Gaussian is: \(p(y \mid \hat{y}) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(y-\hat{y})^2}{2\sigma^2}\right)\). Take the log: \[\log p(y \mid \hat{y}) = -\frac{1}{2}\log(2\pi\sigma^2) - \frac{(y-\hat{y})^2}{2\sigma^2}\] Drop the constant terms (they don’t affect optimization). The negative log-likelihood becomes: \[\mathcal{L}_{MSE} = \sum_{i=1}^N (y^{(i)} - \hat{y}^{(i)})^2\] This is exactly Mean Squared Error. MSE assumes your data has Gaussian noise.

Derivation 3: L2 Regularization (Weight Decay) from MAP

In MAP, we add a prior \(p(\theta)\). Let’s assume a zero-mean Gaussian prior for the weights: \(\theta \sim \mathcal{N}(0, \tau^2)\). This means we believe small weights are more likely than large weights. \[\log p(\theta) = -\frac{1}{2}\log(2\pi\tau^2) - \frac{\|\theta\|^2}{2\tau^2}\] Drop constants. The MAP objective (maximize likelihood + prior) becomes: \[\text{Maximize: } \ell(\theta) - \frac{1}{2\tau^2}\|\theta\|^2\] In deep learning, we minimize negative loss. Let \(\lambda = \frac{1}{\tau^2}\): \[\text{Minimize: } \text{NLL} + \lambda \|\theta\|^2\] This is exactly L2 Regularization (Weight Decay). Weight decay is mathematically equivalent to assuming your weights are drawn from a Gaussian prior.

If you used a Laplace prior instead of a Gaussian, you would derive L1 Regularization (Lasso).

Worked Numerical Example

Let’s compute the MLE and MAP for a simple coin flip scenario manually.

Data: 3 flips, all Heads. \(N=3\), \(k=3\). Parameter to estimate: \(\theta\) (probability of Heads).

MLE Calculation: Likelihood \(L(\theta) = \theta^3\). Log-likelihood \(\ell(\theta) = 3 \log \theta\). Derivative: \(\frac{d\ell}{d\theta} = \frac{3}{\theta}\). Set to 0. Wait, \(\frac{3}{\theta} = 0\) has no solution. We must check the boundaries. The maximum occurs at the edge: \(\hat{\theta}_{MLE} = 1.0\). Insight: MLE overfits to small data. If you only see Heads, it assumes Heads is the only possible outcome.

MAP Calculation: Prior: Beta distribution, equivalent to having seen 2 Heads and 2 Tails previously. \(p(\theta) \propto \theta^2 (1-\theta)^2\). Posterior \(\propto L(\theta) \times p(\theta) = \theta^3 \times \theta^2 (1-\theta)^2 = \theta^5 (1-\theta)^2\). Log-posterior \(= 5 \log \theta + 2 \log(1-\theta)\). Derivative: \(\frac{5}{\theta} - \frac{2}{1-\theta} = 0\). \(5(1-\theta) = 2\theta \implies 5 - 5\theta = 2\theta \implies 5 = 7\theta \implies \hat{\theta}_{MAP} = 5/7 \approx 0.714\).

Insight: The prior (which saw 50% Heads) pulled the estimate away from the extreme 1.0 down to 0.714. This is exactly how regularization prevents overfitting in neural networks.

Computational Interpretation

What the Computer Is Actually Doing

When you call loss.backward() in PyTorch: 1. The framework evaluates the forward pass to get \(\hat{\mathbf{y}}\). 2. It computes the Negative Log-Likelihood (e.g., Cross-Entropy). 3. It computes the derivative of the NLL with respect to the weights (the gradient). 4. The optimizer (e.g., AdamW) steps the weights in the negative direction of the gradient. If using weight decay, it also multiplies the weight by a scalar factor \((1 - \eta \lambda)\), which is the computational equivalent of applying the MAP Gaussian prior gradient.

Complexity

  • Forward pass: \(O(N \times D)\) where \(N\) is batch size, \(D\) is model dimension.
  • Loss evaluation: \(O(N \times C)\) where \(C\) is number of classes (for Cross-Entropy).
  • Bottleneck: The log_softmax operation in Cross-Entropy. PyTorch fuses this into a single highly optimized CUDA kernel (LogSumExp trick) to prevent \(e^x\) from overflowing.

Implementation Perspective

PyTorch: MLE and MAP in Practice

import torch
import torch.nn as nn

# Model: Simple linear layer (weights are the parameters theta)
model = nn.Linear(10, 2)

# MLE: Cross-Entropy Loss is exactly Negative Log-Likelihood for classification
criterion = nn.CrossEntropyLoss()

# MAP: AdamW with weight_decay is exactly MAP with a Gaussian prior
# weight_decay = lambda = 1 / (2 * tau^2) from our derivation
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=1e-2)

# Dummy data
x = torch.randn(32, 10)       # Batch of 32 inputs
y_true = torch.randint(0, 2, (32,)) # True class indices (0 or 1)

# Forward pass (compute likelihood of data under current theta)
logits = model(x)

# Compute Negative Log-Likelihood
loss_mle = criterion(logits, y_true)

# The optimizer.step() will automatically apply the MAP prior gradient (weight_decay)
# when updating the weights.

Every line explained: - nn.CrossEntropyLoss(): This function does not just apply a formula. Under the hood, it computes LogSoftmax followed by NLLLoss (Negative Log Likelihood). It is the exact computational embodiment of MLE for categorical distributions. - weight_decay=1e-2: This single parameter transforms the training from MLE into MAP. During the optimizer step, the gradient is updated as \(w \leftarrow w - \eta(\nabla L + \lambda w)\). The \(\lambda w\) term is the derivative of the Gaussian prior’s log-probability.

Where It Appears In AI

AI Component Exact Usage of MLE / MAP
Classification (CNNs, MLPs) MLE: Cross-Entropy loss is the negative log-likelihood of the categorical distribution.
Regression (Linear Models) MLE: Mean Squared Error is the negative log-likelihood of a Gaussian distribution.
Regularization (All models) MAP: L2 weight decay assumes a Gaussian prior on weights. L1 assumes a Laplace prior.
VAEs (Variational Autoencoders) The ELBO loss is a combination of MLE (reconstruction term) and a MAP prior (KL divergence to a standard Gaussian).
Diffusion Models The training objective is MLE: predicting the noise \(\epsilon\) such that the likelihood of the noisy image is maximized.
Language Models (LLMs) Next-token prediction uses Cross-Entropy loss, which is exact MLE over the sequence.

Deep Dive Into Real Models

Transformers & LLMs

When GPT-4 is trained, the loss function is CrossEntropyLoss. By minimizing this, OpenAI is performing MLE over billions of text tokens. The model is finding the parameters \(\theta\) that maximize the probability of predicting the next token correctly across the entire internet. If they add weight decay, they are performing MAP, ensuring the weights don’t grow unnecessarily large, which would make the model overconfident and brittle.

VAEs (Variational Autoencoders)

A VAE aims to maximize the likelihood of the data \(P(X)\) (MLE). However, the exact likelihood is intractable. The ELBO (Evidence Lower Bound) is used instead: \[\text{ELBO} = \mathbb{E}_{q(z|X)}[\log p(X|z)] - \text{KL}(q(z|X) \parallel p(z))\] - The first term is the MLE term (reconstruction likelihood). - The second term is the MAP term (forcing the latent representation \(z\) to match a prior \(p(z)\), usually a standard Gaussian).

Diffusion Models

Given a clean image \(x_0\) and a noisy image \(x_t\), the model predicts the noise \(\epsilon\). The loss is typically MSE: \[\mathcal{L} = \mathbb{E} \left[ \| \epsilon - \epsilon_\theta(x_t, t) \|^2 \right]\] This is exact MLE, assuming the noise follows a Gaussian distribution. The model is estimating the mean of the noise distribution.

Failure Modes

Failure 1: MLE Overfitting

Cause: MLE relies 100% on the data. If the data is noisy, sparse, or biased, MLE will perfectly fit the noise. Symptoms: The model achieves 100% training accuracy but fails on test data. Solution: Introduce a prior (MAP). Apply Weight Decay (L2) to pull weights toward zero, or use Dropout, which is a form of approximate Bayesian inference.

Failure 2: Numerical Instability in Log-Softmax

Cause: Naively computing \(\log(\text{softmax}(x))\) involves \(\log(\frac{e^{x_i}}{\sum e^{x_j}})\). For large \(x_i\), \(e^{x_i}\) overflows to inf. Symptoms: NaN loss during the first epoch. Solution: PyTorch’s nn.CrossEntropyLoss uses the LogSumExp trick internally. Never implement your own Cross-Entropy using torch.log(torch.softmax(...)). Always use the fused function.

Failure 3: Label Smoothing breaking MLE assumptions

Cause: If you apply hard labels (1.0 and 0.0), MLE pushes the logits to infinity to achieve 0 loss. This causes overconfidence. Symptoms: Model is 99.99% confident on a wrong prediction. Solution: Label Smoothing changes the target distribution from [1, 0] to [0.9, 0.1]. This bounds the maximum likelihood, preventing the weights from exploding and keeping the model calibrated.

Engineering Insights

MLE vs MAP in the Optimizer

There is a subtle difference between adding L2 regularization to the loss function and using Weight Decay in the optimizer. - If you add \(\lambda \|W\|^2\) to the loss function (pure MAP), the gradient includes \(2\lambda W\). Adaptive optimizers like Adam will divide this by the historical squared gradient, distorting the regularization. - If you use weight_decay in AdamW, the optimizer decouples the weight decay, applying it directly as \(W \leftarrow W - \eta \lambda W\) before the gradient step. This is the mathematically correct way to implement MAP in adaptive optimizers.

The Hessian and MLE

The maximum of the likelihood function (the peak) tells you the best parameters. The curvature around that peak (the Hessian matrix) tells you how confident you should be in those parameters. A sharp peak means high confidence; a wide valley means low confidence. This is the basis of the Laplace Approximation in Bayesian Deep Learning.

Interview Questions

Beginner

Q1: What is the difference between probability and likelihood? A1: Probability is a function of the data given fixed parameters: \(P(\text{Data} \mid \theta)\). Likelihood is a function of the parameters given fixed data: \(\mathcal{L}(\theta \mid \text{Data})\). Probability asks “how likely is this outcome?”, while likelihood asks “which parameters make this outcome most likely?”

Q2: Why do we minimize negative log-likelihood instead of just maximizing likelihood? A2: Three reasons: 1) Deep learning optimizers are built to minimize loss. 2) Multiplying many probabilities causes floating-point underflow; logs turn products into sums. 3) Logarithms turn exponential terms (like in Gaussian PDFs) into simple linear terms, making derivatives easy to compute.

Intermediate

Q3: Prove that Cross-Entropy Loss is equivalent to Negative Log-Likelihood for classification. A3: For a one-hot encoded true label \(\mathbf{y}\) and predicted probabilities \(\hat{\mathbf{y}}\), the likelihood of the sample is \(\prod_k \hat{y}_k^{y_k}\). Taking the log gives \(\sum_k y_k \log \hat{y}_k\). Since \(y_k\) is 1 for the true class and 0 otherwise, this is \(\log \hat{y}_{true}\). The negative of this is \(-\log \hat{y}_{true}\), which is exactly the Cross-Entropy loss formula.

Q4: How does L2 Regularization relate to MAP? A4: L2 regularization adds \(\lambda \|W\|^2\) to the loss. In MAP, we maximize the log-likelihood plus the log-prior. If the prior \(p(W)\) is a zero-mean Gaussian, the log-prior is proportional to \(-W^2\). Maximizing this is equivalent to minimizing \(+W^2\). Therefore, L2 regularization is exactly MAP inference with a Gaussian prior, where the regularization strength \(\lambda\) is inversely proportional to the prior’s variance.

Advanced

Q5: If MSE is MLE under a Gaussian noise assumption, what happens if your data has heavy-tailed noise (outliers)? A5: Gaussian MLE (MSE) heavily penalizes large errors quadratically. A single outlier will dominate the loss, pulling the model away from the true trend. The solution is to change the likelihood assumption. Using a Laplace distribution instead of Gaussian yields the Mean Absolute Error (L1 Loss), which is robust to outliers because it penalizes linearly.

Q6: In a Variational Autoencoder, how does the ELBO connect to MLE and MAP? A6: The VAE wants to maximize the data likelihood \(P(X)\) (MLE). Because \(P(X)\) is intractable, it maximizes the ELBO. The ELBO consists of two terms: the reconstruction term \(\mathbb{E}[\log P(X \mid z)]\), which is exact MLE given the latent variable \(z\), and the KL divergence term \(-\text{KL}(q(z \mid X) \parallel p(z))\), which acts as a MAP prior, forcing the approximate posterior to stay close to a standard Gaussian prior.

Research Perspective

Current Limitations

  1. MLE is Prone to Overconfidence: Standard MLE training leads to poorly calibrated models that are overconfident in their predictions, even when wrong.
  2. Mode Collapse in Generative Models: Pure MLE in generative models can lead to mode collapse, where the model generates a limited variety of outputs that maximize likelihood but lack diversity.

Open Problems

  1. Calibrated Uncertainty: How can we perform full Bayesian inference (rather than just MAP point estimates) on 100-billion parameter LLMs without the compute cost increasing by 100x?
  2. Likelihood-Free Inference: For complex simulators where the likelihood \(P(y \mid x, \theta)\) is intractable to compute, how can we estimate parameters? (Addressed by methods like ABC and Flow Matching).

Modern Improvements

  1. AdamW: Decoupled weight decay (MAP) from the adaptive gradient scaling, fixing a fundamental bug in how MAP was applied in standard Adam.
  2. Contrastive Learning (InfoNCE): Instead of strict MLE over pixel data (which is computationally heavy), InfoNCE uses a contrastive lower bound on the mutual information, achieving similar representation learning goals without explicit likelihood calculation.

Recent Research Directions

  1. Diffusion Models and Score Matching: Instead of maximizing likelihood \(P(X)\) directly, score matching minimizes the distance between the model’s gradient (score) and the data’s gradient. It can be shown to be equivalent to a scaled MLE under specific noise schedules.
  2. Surgical Fine-Tuning: Using MLE on only specific subsets of LLM layers, effectively applying a strong prior that the untrained layers remain at their pre-trained values.

Connections

                 PREREQUISITES
                      |
          Probability & Conditional Probability (3.2)
                      |
                      v
               ╔══════════════════╗
               ║      MLE & MAP   ║
               ╚══════════════════╝
              /      |       \
             /       |        \
            v        v         v
        ENABLES   ENABLES   ENABLES
           |         |         |
       Loss Funcs  Optimizers  VAEs
       (XEntropy,  (AdamW,    (ELBO)
        MSE)       Weight Dec)
           |         |
           v         v
      Backpropagation  Regularization
           |
           v
      Deep Learning Training

  USED BY:
  - Every supervised training loop
  - Generative model training (Diffusion, VAE)
  - Hyperparameter tuning (Bayesian Optimization)

  EXTENDED BY:
  - Variational Inference (Approximating the full posterior, not just the peak)
  - Bayesian Deep Learning (MC Dropout, Laplace Approximation)

  RELATED CONCEPTS:
  - Log-Sum-Exp Trick (Numerical stability for MLE)
  - KL Divergence (Measuring distance from prior in MAP)

Final Mental Model

Think of MLE as the Data Dictator and MAP as the Compromise Maker.

MLE looks only at the data and says: “Whatever makes the numbers I just saw most likely is the absolute truth. If the data says 3 Heads in a row, the coin is 100% Heads.” It is powerful, but it overfits and is blind to context.

MAP introduces a prior—a voice of reason. It says: “I hear what the data is saying, but I remember that coins are usually 50/50. I will adjust my parameters to fit the data, but not so much that I violate my common sense.”

Every time you add weight_decay=0.01 to your optimizer, you are activating the Compromise Maker. You are telling the network: “Fit the data (MLE), but keep the weights small (Prior).”

The permanent memory aid:

MLE maximizes the probability of the data given the parameters. MAP maximizes the probability of the parameters given the data (and a prior). Cross-Entropy and MSE are just MLE in disguise. Weight Decay is just MAP in disguise.

Labels:

Comments

Post a Comment

Popular posts from this blog

Why Every AI Model is Actually Bayesian

Why Every AI Model is Actually Bayesian Topic Overview Standard probability asks: “What is the chance of this event happening in the entire universe?” Conditional probability asks: “Given that I already know this to be true, what is the chance of that happening?” Bayes’ Theorem is the engineering tool that flips this condition around. It answers the most critical question in machine learning: “Given the data I just observed, what is the probability that my hypothesis is true?” Why does it exist? Because in the real world, we usually know \(P(\text{Data}|\text{Hypothesis})\) —how likely a certain observation is under a specific model—but what we actually want is \(P(\text{Hypothesis}|\text{Data})\) —how likely our model is to be correct given the observation. Bayes’ Theorem is the mathematical bridge between the data we generate and the truths we want to infer. What problem does it solve? It provides a rigorous, mathematically sound mechani...
Post a Comment
Read more

Why Everything in AI is a Vector ?

Why Everything in AI is a Vector ? Topic Overview In the previous lesson, you learned about the scalar: a single number representing magnitude. But reality is rarely one-dimensional. A single pixel has three color channels (red, green, blue). A word’s meaning has dozens of grammatical and semantic features. A model’s parameters number in the millions. To represent entities that possess multiple attributes simultaneously, we need a mathematical object that holds multiple scalars in a specific order. That object is a vector. A vector \(\mathbf{v} \in \mathbb{R}^n\) is an ordered list of \(n\) scalars. It represents a point, or an arrow from the origin, in \(n\) -dimensional space. Why does it exist? Because we need a mathematical language for things that have more than one degree of freedom. A scalar tells you “how hot.” A vector tells you “where you are” and “which way you’re going.” What problem does it solve? It provides the fundamental d...
Post a Comment
Read more

Why Your PyTorch Code Keeps Breaking (It’s the Scalars)

Why Your PyTorch Code Keeps Breaking (It’s the Scalars) Topic Overview You are about to learn about the most fundamental object in all of mathematics and machine learning: the scalar. A scalar is a single number. That is the entire definition. But do not confuse simplicity with insignificance. Every neural network, no matter how many billions of parameters it contains, is ultimately optimized by reading and writing scalars. The loss value is a scalar. The learning rate is a scalar. The temperature parameter in softmax is a scalar. The scale factor in attention is a scalar. Every gradient clipping decision is based on a scalar. The final prediction of a classification model is a scalar probability. Why does the scalar exist as a named concept? Because not all numbers are created equal. Some numbers exist as components of larger structures—a coordinate in a vector, an entry in a matrix, a pixel in an image. A scalar is a number that stands alone...
Post a Comment
Read more