Topic Overview
In the previous lesson, you learned about the scalar: a single number representing magnitude. But reality is rarely one-dimensional. A single pixel has three color channels (red, green, blue). A word’s meaning has dozens of grammatical and semantic features. A model’s parameters number in the millions. To represent entities that possess multiple attributes simultaneously, we need a mathematical object that holds multiple scalars in a specific order.
That object is a vector.
A vector \(\mathbf{v} \in \mathbb{R}^n\) is an ordered list of \(n\) scalars. It represents a point, or an arrow from the origin, in \(n\)-dimensional space.
Why does it exist? Because we need a mathematical language for things that have more than one degree of freedom. A scalar tells you “how hot.” A vector tells you “where you are” and “which way you’re going.”
What problem does it solve? It provides the fundamental data structure of machine learning. Every input to a neural network, every internal representation, every gradient used for optimization, and every output logit is a vector. Without vectors, we could only process single numbers; we could never process images, sentences, or multi-feature datasets.
Why should engineers care? Because the shape of a
vector—specifically, whether it is a row or a column, and whether it has
1, 768, or 12,288 dimensions—dictates how memory is allocated on the
GPU, how matrix multiplications are scheduled, and whether your model
trains or crashes. Confusing a 1D vector of shape [n] with
a 2D matrix of shape [n, 1] is the gateway bug to countless
hours of debugging dimension mismatches.
Prerequisites
| Concept | Why It Is Needed | Where It Appears |
|---|---|---|
| Scalars (\(x \in \mathbb{R}\)) | A vector is an ordered list of scalars. Every component of a vector is a scalar. | Element-wise operations, bias terms, indexing into a vector |
| Basic Algebra | Vector operations (addition, scaling) are defined component-by-component using scalar algebra. | Computing vector sums, linear combinations |
Historical Motivation
In the mid-19th century, physicists faced a representation crisis. They could easily measure scalar quantities like temperature and mass, but forces, velocities, and electromagnetic fields required both magnitude and direction.
If a wind blows at 10 m/s, that scalar (10) is useless to a sailor unless they also know which way it’s blowing. Early physicists tried using coordinate geometry, writing forces as triplets of numbers \((x, y, z)\). But treating a force as just three independent numbers missed a profound physical reality: the force is a single unified object that happens to be measured against three axes. If you rotate your coordinate system, the individual numbers change, but the physical force remains the same.
Josiah Willard Gibbs and Oliver Heaviside formalized the vector as a distinct mathematical entity—an object with its own algebra and calculus. This allowed them to write \(\mathbf{F} = m\mathbf{a}\) instead of three separate scalar equations.
Engineering necessity in ML: In machine learning, we face the exact same problem, just in thousands of dimensions instead of three. A word like “bank” has hundreds of semantic features (is it a noun? is it financial? is it related to rivers?). We cannot represent this with a scalar. We need an ordered list of numbers—an embedding—where each dimension captures a latent feature. The vector is the mathematical object that lets a machine “hold” the concept of “bank” in its memory.
Intuitive Explanation
Imagine you are describing a house to a real estate agent. You don’t just say “It’s a 3.” You list multiple features: 1. Number of bedrooms: 3 2. Area (sq ft): 1500 3. Price ($): 350,000 4. Age (years): 20
If you write these as an ordered list:
[3, 1500, 350000, 20], you have just created a vector.
Why must it be ordered? If you swap the first and
last numbers, you get [20, 1500, 350000, 3]—a 20-bedroom
house that is 3 years old and costs $350,000. That’s a completely
different house. The position of a number in the list is just
as important as the number itself.
Mental model 1: The Address. A vector is an address in a multi-dimensional city. In a 2D city (like a grid), your address is (Street, Avenue). In a 768-dimensional city (like BERT’s embedding space), your address is a list of 768 coordinates.
Mental model 2: The Arrow. Imagine a compass. The arrow points from the center (the origin, where everything is zero) to your location. The length of the arrow is the magnitude; the way it points is the direction.
Physical interpretation: Velocity is a vector. “Traveling at 60 mph” is a scalar (speed). “Traveling 60 mph North” is a vector (velocity). The vector captures both how fast and which way.
In machine learning, “which way” translates to “what concept.” Words like “dog” and “puppy” point in similar directions in the model’s internal space, while “dog” and “carburetor” point in very different directions. The vector gives the model a geometry of meaning.
Visual Understanding
In \(\mathbb{R}^2\) (2D space): Draw two perpendicular axes: x (horizontal) and y (vertical). The vector \(\mathbf{v} = \begin{bmatrix} 3 \\ 2 \end{bmatrix}\) is an arrow starting at the origin \((0,0)\) and ending at the point \((3,2)\). You walk 3 steps right, 2 steps up.
y
3 |
2 | / v = [3, 2]
| /
1 | /
| /
0 |__/___________ x
0 1 2 3In \(\mathbb{R}^3\) (3D space): Add a z-axis coming out of the page. The vector \(\mathbf{v} = \begin{bmatrix} 2 \\ 1 \\ 3 \end{bmatrix}\) is an arrow in 3D space. You can visualize this with physical objects or 3D software, but human intuition starts to strain.
In \(\mathbb{R}^{768}\) (BERT embedding space): You cannot draw this. You cannot even truly visualize it. Instead, you must shift your visual intuition to clusters and distances. Imagine a vast, dark, 768-dimensional room. Every word in the dictionary is a tiny glowing dot somewhere in that room. Words with similar meanings—“happy,” “joyful,” “glad”—cluster together like a nebula. Words with opposite meanings—“hot,” “cold”—sit on opposite sides of the room.
When you “add a vector,” you take a step in this high-dimensional room. When you “compute a distance,” you measure how far apart two concepts are.
Vector Addition Visually: To add vector \(\mathbf{a}\) and \(\mathbf{b}\), place the tail of \(\mathbf{b}\) at the tip of \(\mathbf{a}\). The resulting vector \(\mathbf{a} + \mathbf{b}\) is the arrow from the origin to the new tip. It forms a parallelogram.
a + b
/ \
/ \
/ b
/ /
a /
/ /
/__________/
0Mathematical Foundations
Formal Definition
A vector \(\mathbf{v} \in \mathbb{R}^n\) is an ordered tuple of \(n\) real numbers:
\[\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}\]
- \(\mathbf{v}\): The vector itself (bold, lowercase by convention).
- \(n\): The dimensionality of the vector.
- \(v_i\): The \(i\)-th component (or element, or coordinate) of the vector. This is a scalar \(\in \mathbb{R}\).
- \(\mathbb{R}^n\): The set of all vectors with \(n\) real components.
Column vs. Row Vectors
By convention in linear algebra and ML, a vector is a column vector. It occupies a vertical space:
\[\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix} \in \mathbb{R}^{n \times 1}\]
A row vector is the transpose of a column vector, denoted \(\mathbf{v}^\top\):
\[\mathbf{v}^\top = \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix} \in \mathbb{R}^{1 \times n}\]
The transpose operation \(\top\) swaps rows and columns. This distinction is purely notational until you multiply vectors together, at which point it becomes a hard engineering requirement.
Vector Operations
1. Vector Addition
Given \(\mathbf{a}, \mathbf{b} \in \mathbb{R}^n\), their sum is computed component-wise:
\[\mathbf{c} = \mathbf{a} + \mathbf{b} \implies c_i = a_i + b_i\]
Both vectors must have the same dimension. You cannot add a vector in \(\mathbb{R}^3\) to a vector in \(\mathbb{R}^5\).
2. Scalar Multiplication
Given a scalar \(\alpha \in \mathbb{R}\) and a vector \(\mathbf{v} \in \mathbb{R}^n\):
\[\mathbf{u} = \alpha \mathbf{v} \implies u_i = \alpha v_i\]
This stretches (if \(|\alpha| > 1\)), compresses (if \(|\alpha| < 1\)), or flips (if \(\alpha < 0\)) the vector without changing its underlying direction (unless \(\alpha = 0\), which collapses it to the zero vector).
3. Linear Combination
Combining addition and scalar multiplication:
\[\mathbf{c} = \alpha \mathbf{a} + \beta \mathbf{b}\]
This is the foundation of all neural network layers. A linear layer computes a linear combination of input vectors (weighted by a matrix).
The Vector Space Axioms (Why this Algebra Works)
For \(\mathbb{R}^n\) to be a valid vector space, it must obey closure under addition and scalar multiplication: - If \(\mathbf{a}, \mathbf{b} \in \mathbb{R}^n\), then \(\mathbf{a} + \mathbf{b} \in \mathbb{R}^n\). - If \(\alpha \in \mathbb{R}\) and \(\mathbf{v} \in \mathbb{R}^n\), then \(\alpha \mathbf{v} \in \mathbb{R}^n\).
This guarantees that no matter how you add or scale these vectors, you never “leave the space.” In ML, this means that if your input is a vector in \(\mathbb{R}^{768}\), the hidden state computed by your linear layer is guaranteed to also be a vector in \(\mathbb{R}^{768}\) (or whatever the output dimension is), never some undefined mathematical object.
Worked Numerical Example
Let us compute a Transformer-style embedding addition: Token Embedding + Positional Encoding.
Assume a tiny model with dimension \(d=3\). The word “Hello” has a token embedding \(\mathbf{e}_{\text{hello}}\), and it appears at position 1, which has a positional encoding \(\mathbf{p}_1\).
\[\mathbf{e}_{\text{hello}} = \begin{bmatrix} 0.5 \\ 0.2 \\ 1.0 \end{bmatrix}, \quad \mathbf{p}_1 = \begin{bmatrix} 0.1 \\ 0.8 \\ 0.3 \end{bmatrix}\]
Step 1: Vector Addition The input to the Transformer layer is the sum:
\[\mathbf{x} = \mathbf{e}_{\text{hello}} + \mathbf{p}_1\]
\[x_1 = 0.5 + 0.1 = 0.6\] \[x_2 = 0.2 + 0.8 = 1.0\] \[x_3 = 1.0 + 0.3 = 1.3\]
\[\mathbf{x} = \begin{bmatrix} 0.6 \\ 1.0 \\ 1.3 \end{bmatrix}\]
Step 2: Scalar Multiplication (LayerNorm scaling) Suppose LayerNorm computes a scale factor \(\gamma = 2.0\) for this vector. We multiply:
\[\mathbf{y} = \gamma \mathbf{x} = 2.0 \times \begin{bmatrix} 0.6 \\ 1.0 \\ 1.3 \end{bmatrix}\]
\[y_1 = 2.0 \times 0.6 = 1.2\] \[y_2 = 2.0 \times 1.0 = 2.0\] \[y_3 = 2.0 \times 1.3 = 2.6\]
\[\mathbf{y} = \begin{bmatrix} 1.2 \\ 2.0 \\ 2.6 \end{bmatrix}\]
What happens if we add a vector from a different space? If \(\mathbf{p}_1\) was mistakenly computed for \(d=2\): \(\mathbf{p}_1 = \begin{bmatrix} 0.1 \\ 0.8 \end{bmatrix}\), the operation \(\mathbf{e}_{\text{hello}} + \mathbf{p}_1\) is undefined. You cannot add a 3D point and a 2D point. In PyTorch, this throws a hard dimension mismatch error.
Computational Interpretation
What the Computer Is Actually Doing
To a computer, a vector is a contiguous block of memory holding \(n\) floating-point numbers, stored sequentially.
For \(\mathbf{v} = [v_1, v_2, v_3]\)
of type float32: - v[0] is stored at memory
address base + 0 * 4 - v[1] is stored at
memory address base + 1 * 4 - v[2] is stored
at memory address base + 2 * 4
The stride is 4 bytes (the size of one
float32). To access element \(i\), the computer calculates
base_address + i * stride. This is \(O(1)\) time complexity.
Computational Complexity
| Operation | Time Complexity | Memory Complexity | Notes |
|---|---|---|---|
| Storage | - | \(O(n)\) | 4 bytes per element for float32 |
| Element access | \(O(1)\) | \(O(1)\) | Direct memory offset |
| Addition \(\mathbf{a} + \mathbf{b}\) | \(O(n)\) | \(O(n)\) | One add per element |
| Scalar multiply \(\alpha \mathbf{v}\) | \(O(n)\) | \(O(n)\) | One multiply per element |
| Dot product \(\mathbf{a} \cdot \mathbf{b}\) | \(O(n)\) | \(O(1)\) | \(n\) multiplies + \(n\) adds, result is scalar |
GPU Execution
When you add two vectors on a GPU, the \(O(n)\) operations are not done in a loop. They are parallelized across hundreds of CUDA cores. If \(n=1024\), 1024 cores can each add one pair of elements simultaneously. The wall-clock time becomes \(O(1)\) (conceptually, though practically limited by memory bandwidth and warp scheduling).
The bottleneck for vectors is almost always memory bandwidth, not compute. Reading \(n\) floats from VRAM takes longer than adding them together.
Implementation Perspective
NumPy Implementation
import numpy as np
# ===== Creating Vectors =====
# A 1D array: a vector of dimension 3
v = np.array([0.5, 0.2, 1.0])
print(v.shape) # (3,) <- 1 dimensional, size 3
print(v.ndim) # 1 <- it is 1D
# CRITICAL DISTINCTION: Row vs Column in NumPy
# NumPy 1D arrays have NO row/column distinction. They are just 1D.
# To make a true column vector (2D matrix with 1 column):
v_col = v.reshape(-1, 1) # or v[:, np.newaxis]
print(v_col.shape) # (3, 1)
# To make a true row vector (2D matrix with 1 row):
v_row = v.reshape(1, -1) # or v[np.newaxis, :]
print(v_row.shape) # (1, 3)
# ===== Vector Operations =====
a = np.array([0.5, 0.2, 1.0])
b = np.array([0.1, 0.8, 0.3])
# Addition: component-wise
c = a + b
# [0.6, 1.0, 1.3]
# Scalar Multiplication
d = 2.0 * a
# [1.0, 0.4, 2.0]
# Element-wise multiplication (Hadamard product) - used in masks/dropout
e = a * b
# [0.05, 0.16, 0.3]
# Dot product (we will cover this in depth later, but crucial to see)
dot = np.dot(a, b)
# (0.5*0.1) + (0.2*0.8) + (1.0*0.3) = 0.05 + 0.16 + 0.3 = 0.51PyTorch Implementation
import torch
# ===== Creating Vectors =====
# A 1D tensor: shape [d]
token_emb = torch.tensor([0.5, 0.2, 1.0])
print(token_emb.shape) # torch.Size([3])
print(token_emb.dim()) # 1
# In PyTorch, a 1D tensor of shape [d] is the standard representation of a vector.
# It is NEITHER a row nor a column vector until forced into 2D.
# ===== Real-World ML Context: Batches =====
# We rarely process one vector at a time. We process a BATCH of vectors.
# This is a 2D tensor (Matrix), but conceptually it is a list of vectors.
# Batch of 32 token embeddings, each of dimension 768
batch_emb = torch.randn(32, 768)
print(batch_emb.shape) # torch.Size([32, 768])
# batch_emb[0] is the first vector, shape [768]
# ===== The Silent Killer: Shape [B, 1] vs [B] =====
# Suppose we have a scalar feature (e.g., a score) for each item in a batch
scores_1d = torch.tensor([0.1, 0.5, 0.9]) # shape [3]
scores_2d = torch.tensor([[0.1], [0.5], [0.9]]) # shape [3, 1]
# Adding a vector to these scores:
bias = torch.tensor([1.0, 2.0, 3.0]) # shape [3]
# This works: [3] + [3] -> [3]
result_1d = scores_1d + bias
# THIS IS A BUG: [3, 1] + [3] broadcasts to [3, 3] !
# PyTorch treats bias as a row vector and broadcasts it across the rows.
result_2d = scores_2d + bias
print(result_2d.shape) # torch.Size([3, 3]) !!
# You expected shape [3, 1], but got a 3x3 matrix.
# This will silently ruin your loss computation.
# How to fix: explicitly reshape bias to [3, 1]
result_correct = scores_2d + bias.unsqueeze(1) # [3, 1] + [3, 1] -> [3, 1]Every line explained: - token_emb.shape
returning torch.Size([3]) means it is a 1D tensor. It has 1
axis (dimension). - batch_emb = torch.randn(32, 768): This
is technically a matrix, but in ML semantics, it is a batch of 32
vectors. The first dimension is always the batch dimension. -
scores_2d + bias: Broadcasting rules in PyTorch/NumPy align
dimensions from the right. bias is shape [3],
which aligns with the last dimension of scores_2d shape
[3, 1]. bias is treated as
[1, 3], and they broadcast to [3, 3]. This is
the most common vector shape bug in deep learning.
Where It Appears In AI
| AI Component | Exact Role of the Vector | Vector Dimensions |
|---|---|---|
| Embedding Layer | Maps discrete tokens (words, IDs) to continuous dense vectors. | \(d_{model}\) (e.g., 768, 4096) |
| Transformer Attention | Query, Key, and Value projections of tokens. | \(d_k, d_v\) |
| Linear/Dense Layer | Bias vector added to the matrix multiplication output. | \(d_{out}\) |
| LayerNorm / BatchNorm | Learnable scale (\(\gamma\)) and shift (\(\beta\)) vectors applied element-wise. | \(d_{model}\) |
| Classifier Head | Final output logit vector before softmax. | num_classes |
| RNN / LSTM | Hidden state vector passed from timestep to timestep. | \(d_{hidden}\) |
| CNN (Global Pooling) | Flattened or pooled feature vector before the final linear layer. | num_channels |
| VAE (Latent Space) | The compressed latent representation \(z\) sampled from the encoder. | \(d_{latent}\) (e.g., 128) |
| Diffusion Models | The noise vector \(\epsilon\) predicted by the U-Net at each timestep. | Same as image flattened dim |
| Contrastive Learning | Image/text embeddings projected into a shared space for similarity comparison. | \(d_{proj}\) (e.g., 256) |
| GNN | Node feature vectors aggregated from graph neighbors. | \(d_{node}\) |
Deep Dive Into Real Models
MLPs (Multi-Layer Perceptrons)
The core of an MLP is the linear layer: \(\mathbf{h} = W\mathbf{x} + \mathbf{b}\). - Input \(\mathbf{x} \in \mathbb{R}^{d_{in}}\): a vector. - Weight \(W \in \mathbb{R}^{d_{out} \times d_{in}}\): a matrix. - Bias \(\mathbf{b} \in \mathbb{R}^{d_{out}}\): a vector. - Output \(\mathbf{h} \in \mathbb{R}^{d_{out}}\): a vector.
The bias vector \(\mathbf{b}\) shifts the output space, allowing the layer to represent functions that don’t pass through the origin. Without \(\mathbf{b}\), if \(\mathbf{x} = \mathbf{0}\), then \(\mathbf{h}\) is always \(\mathbf{0}\), severely limiting what the network can learn.
Shapes in PyTorch:
linear = torch.nn.Linear(in_features=784, out_features=256)
x = torch.randn(784) # input vector
h = linear(x) # output vector
print(h.shape) # torch.Size([256])CNNs
While CNNs process 4D tensors (Batch, Channels, Height, Width), the
final layers always collapse spatial dimensions to produce a vector. A
Global Average Pooling layer takes a feature map of shape
[512, 7, 7] and averages over the spatial dimensions to
produce a single vector of shape [512]. This vector is the
image’s dense representation, passed to a classifier.
Transformers (and LLMs)
Vectors are the lifeblood of Transformers. 1. Embedding: Token ID $ ^{d_{model}}$ 2. Positional Encoding: Position index $ ^{d_{model}}$ 3. Input Representation: \(\mathbf{x} = \mathbf{e} + \mathbf{p}\) (Vector addition!) 4. Attention Projections: \(\mathbf{q} = W_q \mathbf{x}\), \(\mathbf{k} = W_k \mathbf{x}\), \(\mathbf{v} = W_v \mathbf{x}\). Each is a vector. 5. Output Logits: The final hidden vector \(\mathbf{h}_{\text{last}}\) is multiplied by the unembedding matrix to produce a logit vector \(\mathbf{l} \in \mathbb{R}^{V}\) where \(V\) is the vocabulary size (e.g., 50,257 for GPT-2).
In LLaMA-7B, every token flows through 32 layers as a vector of dimension \(d_{model} = 4096\).
Diffusion Models
The U-Net architecture in a diffusion model predicts the noise \(\epsilon_\theta(\mathbf{x}_t, t)\). - The noisy image \(\mathbf{x}_t\) is an enormous vector (e.g., \(3 \times 512 \times 512 = 786,432\) dimensions). - The timestep \(t\) is encoded into a vector via sinusoidal embeddings, then injected into the U-Net layers.
Reinforcement Learning
In RL with continuous state spaces (like robotics), the environment state is represented as a vector: \(\mathbf{s} = [\text{angle}_1, \text{angle}_2, \text{velocity}_1, \text{velocity}_2, \dots]\). The policy network takes this state vector and outputs an action vector \(\mathbf{a} = [\text{torque}_1, \text{torque}_2]\).
Failure Modes
Failure 1: Row vs. Column Confusion (Silent Broadcast Bug)
Cause: Assuming torch.matmul or
@ will figure out row/column alignment automatically.
Symptoms:
# Intended: dot product of two vectors
a = torch.tensor([1.0, 2.0, 3.0]) # shape [3]
b = torch.tensor([4.0, 5.0, 6.0]) # shape [3]
# Outer product instead of dot product!
result = a.unsqueeze(1) @ b.unsqueeze(0)
# shape [3, 1] @ shape [1, 3] = shape [3, 3]
# Expected scalar 32.0, got a 3x3 matrix.Solution: For dot products, use
torch.dot(a, b) or (a * b).sum(). Do not use
@ on 1D tensors unless you understand its specific rules
(it actually does return the dot product for 1D tensors, but if you
unsqueeze to 2D to be “safe”, you create an outer product). Always
verify shapes.
Failure 2: [B, 1]
vs [B] Broadcasting
Cause: A vector of shape [B, 1] (2D)
and [B] (1D) behave completely differently under arithmetic
with other tensors. Symptoms: See the Implementation
Perspective example where [3, 1] + [3] yields
[3, 3]. Solution: Always explicitly
squeeze or unsqueeze to align dimensions. Use
tensor.view(-1, 1) or tensor.unsqueeze(-1) to
force a dimension.
Failure 3: Mixing Embedding Dimensions Across Models
Cause: Trying to pass a vector from a model with
\(d=768\) into a layer expecting \(d=512\). Symptoms:
RuntimeError: mat1 and mat2 shapes cannot be multiplied.
Solution: Always check .shape before
passing tensors between modules. Use a linear projection layer to change
dimensions if necessary.
Failure 4: In-Place Operations on Vectors Requiring Gradients
Cause: Modifying a vector in-place (e.g.,
v[v > 0] = 0) when it is part of the autograd graph.
Symptoms:
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
Solution: Use out-of-place operations or PyTorch
functional versions (e.g., torch.clamp(v, max=0) instead of
masking).
Engineering Insights
Memory Layout: Contiguity
When you slice a vector or transpose it, PyTorch may create a “view”
that is non-contiguous. The data remains in memory in its original
order, but the strides change. -
v.contiguous().is_contiguous() is True. - A
non-contiguous vector is slower to process on GPUs because memory access
patterns are uncoalesced. - If you see errors like
RuntimeError: needs contiguous input, call
.contiguous() before the failing operation.
Precision and Vector Dimensions
A BERT embedding vector has 768 float32 elements. -
Memory per vector: \(768 \times 4 \text{
bytes} = 3 \text{ KB}\). - In bfloat16: \(768 \times 2 \text{ bytes} = 1.5 \text{
KB}\).
When you have a sequence of 512 tokens (a batch of 512 vectors), that’s \(512 \times 3 \text{ KB} \approx 1.5 \text{ MB}\). For an LLM with \(d=12288\) and context length 4096, the input vectors take \(4096 \times 12288 \times 2 \text{ bytes} \approx 96 \text{ MB}\) just for the sequence embeddings. The vector dimension directly dictates VRAM consumption.
Vectorization: Never Loop Over a Vector
In Python, looping over a vector’s elements is catastrophically slow:
# TERRIBLE: Python loop
for i in range(len(v)):
v[i] = v[i] * 2.0
# CORRECT: Vectorized operation
v = v * 2.0Vectorized operations push the loop down into C/CUDA, yielding 100x-1000x speedups. The vector is not just a mathematical concept; it is a computational paradigm. If you are iterating over elements, you are doing it wrong.
Interview Questions
Beginner
Q1: What is the difference between a scalar and a vector? A1: A scalar is a single number (\(x \in \mathbb{R}\)) representing magnitude. A vector is an ordered list of \(n\) numbers (\(\mathbf{v} \in \mathbb{R}^n\)) representing magnitude and direction (or multiple features).
Q2: What does it mean for two embedding vectors to be “similar”? A2: In ML, similarity is typically measured by the dot product or cosine similarity. Geometrically, similar vectors point in roughly the same direction in the high-dimensional embedding space. If \(\mathbf{v}_{\text{dog}}\) and \(\mathbf{v}_{\text{puppy}}\) point in similar directions, their dot product is large and positive.
Intermediate
Q3: How are positional encodings added to token embeddings in Transformers? Why add instead of concatenate? A3: They are added element-wise: \(\mathbf{x} = \mathbf{e} + \mathbf{p}\). If we concatenated (\([\mathbf{e}, \mathbf{p}]\)), the vector dimension would double from \(d\) to \(2d\), doubling the compute of all subsequent matrix multiplications. Adding preserves dimension. It works because the model learns to disentangle the magnitude (token identity) and phase/direction (position) in the high-dimensional space, especially when positional encodings are sinusoidal.
Q4: Explain the difference between a PyTorch tensor of shape
[d] and shape [d, 1].
A4: [d] is a 1D tensor (a pure vector).
[d, 1] is a 2D tensor (a column vector/matrix with 1
column). They behave differently under broadcasting.
[d] + [d, d] will fail, while [d, 1] + [1, d]
broadcasts to [d, d]. 1D tensors broadcast against the last
dimension of other tensors, which is a frequent source of silent shape
bugs.
Advanced
Q5: Why can’t we use a scalar to represent the position of a token in a sequence, instead of a vector? A5: A single scalar (e.g., \(t=1, 2, 3...\)) provides no rich relational information. It only tells the model “A comes before B.” A positional encoding vector \(\mathbf{p} \in \mathbb{R}^d\) allows the model to project the position into the same high-dimensional space as the token, enabling the attention mechanism to compute complex geometric relationships (like relative distance) via dot products. Sinusoidal positional vectors specifically allow the model to linearly combine them to represent relative offsets.
Q6: In the context of Vector Space Models, what is the curse of dimensionality, and how does it affect embedding vectors? A6: As dimensionality \(n\) increases, the volume of the space grows exponentially. In very high dimensions, randomly distributed vectors become approximately orthogonal (their dot product approaches 0). This means distances become less meaningful, and models require exponentially more data to learn the true geometry of the space. Techniques like dimensionality reduction (PCA, t-SNE) and contrastive learning are used to ensure the vector space retains meaningful structure.
Research-Level
Q7: Discuss “Superposition” in the context of LLM embedding vectors. How can a model represent more features than it has vector dimensions? A7: Superposition (as explored by Anthropic) is the phenomenon where a neural network represents more than \(d\) features in a \(\mathbb{R}^d\) vector space by allowing features to be slightly non-orthogonal. If features were strictly orthogonal, a 768-dim vector could only represent 768 features. By accepting some interference (polysemantic neurons where one dimension encodes multiple unrelated concepts), the model can represent thousands of features. This is only possible because the vectors are continuous; the model can pack concepts in “between” other concepts, trading off orthogonality for capacity.
Research Perspective
Current Limitations
- Interpretability: A vector in \(\mathbb{R}^{768}\) is mathematically precise but semantically opaque. We know “king” is a vector, but we cannot easily map each dimension to a human-readable feature.
- Static Embeddings vs. Context: Traditional word2vec assigns one vector per word. The word “bank” has the same vector whether it’s a river bank or a financial bank. Modern LLMs solve this via contextual embeddings (the vector changes based on the surrounding sentence), but the underlying representation is still a single vector.
Open Problems
- Polysemanticity: As mentioned in superposition, a single dimension in a vector often encodes multiple unrelated concepts. How do we disentangle these to create monosemantic vectors?
- Optimal Dimensionality: Is there a mathematical limit to how many features can be packed into a \(d\)-dimensional vector before learning fails? How should we choose \(d\) for a new architecture?
Modern Improvements
- Sparse Vectors: Instead of dense vectors (where every dimension is non-zero), research into sparse attention and retrieval-augmented generation uses vectors where most dimensions are zero, making them interpretable and efficient.
- Matryoshka Representation Learning (MRL): Training embedding vectors such that the first \(k\) dimensions are a valid, lower-dimensional embedding. This allows truncating the vector to save memory without retraining. (e.g., a 2048-dim vector can be truncated to 256-dim on the fly).
Recent Research Directions
- Geometric Deep Learning: Attempting to unify ML architectures by viewing all data as vectors on graphs and manifolds, respecting the underlying geometric structure (symmetries) of the space.
- Vector Symbolic Architectures (VSA) / Hyperdimensional Computing: Using extremely large (e.g., \(d=10,000\)), quasi-orthogonal binary/bipolar vectors to represent symbols. This mimics brain-like robustness and allows logical reasoning via vector binding and unbinding operations.
Connections
PREREQUISITES
|
Scalars
(1.1)
|
v
╔═══════════╗
║ VECTORS ║
╚═══════════╝
/ | \
/ | \
v v v
ENABLES ENABLES ENABLES
| | |
Dot Product Norms Matrix Ops
(1.3) (1.4) (2.1)
| | |
v v v
Cosine Sim L2 Reg Linear Layers
Attention Weight (Wx + b)
Contrastive Decay
|
v
Embeddings
(Word2Vec,
Positional)
USED BY:
- Every Neural Network Layer (Input/Output/Hidden)
- Optimizers (Gradient vector ∇L)
- Embedding Models (Dense representations of discrete data)
EXTENDED BY:
- Matrices: Lists of vectors
- Tensors: n-dimensional generalizations
- Subspaces: Planes/spans formed by sets of vectors
RELATED CONCEPTS:
- Vector Fields (a vector at every point in space)
- Topology (connectedness of vector spaces)
- Information Theory (vectors as codes)Final Mental Model
Think of a vector as a Super-Address.
A scalar is a house number on a 1D street. It tells you exactly one thing: how far down the street to walk.
A vector is a GPS coordinate in a multi-dimensional city.
[3, 1500, 350000, 20] is the address of a specific house in
“Feature City,” where the first avenue is “Bedrooms,” the second is
“Square Feet,” and so on.
In a Neural Network, the model doesn’t see images or words. It only sees addresses in a vast, dark, high-dimensional city. It learns to navigate this city by moving from one address to another (via matrix multiplications), adding directional offsets (biases and positional encodings), and measuring how close two addresses are (dot products).
The vector is the bridge between the messy, discrete real world and the clean, continuous mathematical world where optimization is possible. If you change one coordinate of the vector, you move the address; if you change the scale, you stretch the city.
The permanent memory aid:
Vector = Ordered List. Multi-feature Address. Arrow in space. Shape
[n]. The fundamental currency of all deep learning representations—every concept, every token, every gradient is a vector.
Comments
Post a Comment