# Manifold learning

### What is a Manifold? (The Core Idea)

Think of it this way:

- **A straight line** is a 1-dimensional manifold. If you're walking along a line, you only have one direction to go (forward/backward).
- **The surface of a sphere (like Earth's surface)** is a 2-dimensional manifold. Even though the Earth exists in 3D space, if you're on its surface, you only need two coordinates (like latitude and longitude) to define your position. You can move forward/backward and left/right on the surface, but you can't go "up" or "down" relative to the surface itself without leaving it.

So, a **manifold is a low-dimensional shape or surface that is embedded within a higher-dimensional space.** Locally, it looks flat (like a line looks flat if you zoom in, or a small patch of Earth looks flat), but globally, it can be curved or twisted.

### Why is this important for Data? (The "Data Manifold Hypothesis")

Most real-world data isn't just randomly scattered. It tends to have underlying patterns and structures. For example:

- **Images of faces:** A face image might have hundreds or thousands of pixels (dimensions). But valid "face" images don't fill up that entire high-dimensional space. They occupy a small, specific region. The variations between faces (e.g., age, expression, lighting) are limited, and these variations define a low-dimensional "face manifold" within the high-dimensional image space. You can't just randomly change pixels and get a new valid face; you have to follow the rules of what makes a face.
- **Handwritten Digits (like MNIST):** As mentioned, a 28x28 pixel image is 784 dimensions. But a "digit 7" isn't just any random combination of 784 pixels. It has a specific shape, thickness, slant. All possible "7"s form a much smaller, curved "7-manifold" within that 784-dimensional space.

The **"Data Manifold Hypothesis"** is the idea that real-world data, despite appearing in high-dimensional spaces, actually lies on or very close to such a low-dimensional manifold.

### How do Autoencoders "Learn" Manifolds?

This is where autoencoders come in. Autoencoders are neural networks designed to learn efficient representations of data. They have two main parts:

1. **Encoder:** Takes a high-dimensional input (x) and compresses it into a lower-dimensional **latent representation** (h). This latent space is where we hope to find the manifold.
2. **Decoder:** Takes the latent representation (h) and tries to reconstruct the original high-dimensional input (x^).

The core idea is that an autoencoder, by forcing the data through a bottleneck (the latent space) and then demanding accurate reconstruction, is implicitly trying to learn the most essential features of the data. If the data truly lies on a manifold, the autoencoder will learn to map points from the manifold to a compact representation in the latent space, and then map them back onto the manifold during reconstruction.

**Here's the intuitive flow:**

- **Compression:** The encoder learns to "project" the high-dimensional data points onto this lower-dimensional manifold in the latent space. It discards the "noise" or "unimportant" variations and keeps only the variations that define the manifold structure.
- **Reconstruction:** The decoder then learns to "unproject" these latent representations back into the original high-dimensional space, but critically, it projects them _back onto the manifold_. It tries to reconstruct the data as if it were a point on that learned surface.

![[Pasted image 20250611173756.png]]
**Think of it like this, using the provided Figure 14.7:**

The blue dashed line is the "identity function," meaning if the autoencoder simply copied the input to the output, the reconstruction (r(x)) would equal the input (x). But a good autoencoder doesn't just copy. It learns to capture the underlying structure.

The green line represents the "optimal reconstruction function." Notice how it's not a perfect copy. Instead, it creates "steps." These steps represent the autoencoder "pulling" any input x towards the nearest "manifold" point.

- If you perturb an input x0​ slightly (move it away from the "manifold" point), the autoencoder will try to reconstruct it _back to_ the "manifold" point (indicated by the horizontal arrows pointing back to x0​,x1​,x2​).
- The "manifold" points (x0​,x1​,x2​ in the diagram) are where the optimal reconstruction function _crosses_ the identity function. This means these are the points where the autoencoder produces the _exact_ input as output, because they are already _on_ the manifold.

This behavior, where the autoencoder is invariant to small perturbations _near_ data points and pulls inputs back to specific points, means it's learning the stable, fundamental structures that define the manifold.

### The Role of Tangent Planes (Figure 14.9)
![[Pasted image 20250611174009.png]]
The concept of "tangent planes" is a more advanced mathematical detail, but intuitively:

- Imagine a small patch on the surface of our Earth sphere. Locally, it looks flat. We can define a "tangent plane" at any point on the sphere. This plane represents the directions you can move _on the surface_ at that exact point.
- In manifold learning, these "tangent planes" represent the local directions of variation _along the manifold_. For a face manifold, a tangent plane at a specific face might describe how the pixels change if the person smiles slightly or tilts their head.
- If we can learn these local "flat" patches (like the ovals in Figure 14.9) and how they fit together, we can map out the entire manifold. Autoencoders, especially more advanced ones, implicitly learn these local variations. They learn that certain directions of change in the input data are "meaningful" (stay on the manifold), while others are "noise" (off the manifold).

### In essence:

Autoencoders are powerful tools for **unsupervised manifold learning** because they:

1. **Compress data:** They force the model to find a lower-dimensional representation (h).
2. **Reconstruct faithfully:** They penalize poor reconstruction, ensuring that the learned representation captures the essential information needed to recreate the original data.
3. **Learn robustness:** By doing so, they learn a reconstruction function that maps nearby points in the high-dimensional space back to the same (or very similar) points on the intrinsic manifold, effectively "cleaning" noisy data by snapping it to the manifold.

This ability to discover and model these hidden, low-dimensional structures is why autoencoders are so valuable for tasks like dimensionality reduction, data denoising, and understanding the intrinsic variations within complex datasets.

---
# Reparameterization Trick

The reparameterization trick is a crucial technique used in Variational Autoencoders (VAEs) and other probabilistic models to enable gradient-based optimization through stochastic (random) nodes in a computation graph.

**The Problem It Solves:**
In a VAE, we want to sample a latent variable $z$ from the approximate posterior distribution $q(z|x)$. This distribution is typically learned by the encoder network, which outputs its parameters (e.g., mean $\mu$ and variance $\sigma^2$ for a Gaussian). The act of *sampling* $z$ from $q(z|x)$ is a stochastic operation. Standard backpropagation relies on deterministic operations to compute gradients. If we were to directly sample $z$ and then compute gradients, the randomness would break the computation graph, making it impossible to backpropagate gradients through the sampling step to update the encoder's parameters ($\mu$ and $\sigma^2$).

**How it Works:**
The reparameterization trick resolves this by **separating the stochasticity from the parameters** of the distribution. Instead of directly sampling $z$ from $q(z|x)$ (e.g., $\mathcal{N}(\mu, \sigma^2)$), we introduce an auxiliary random variable $\epsilon$ that follows a simple, parameter-free distribution (typically a standard normal distribution, $\mathcal{N}(0, I)$).

Then, we express $z$ as a **deterministic function** of the parameters ($\mu, \sigma$) and this independent random variable $\epsilon$.

For a Gaussian distribution, the reparameterization trick is given by the formula:
$z = \mu + \sigma \odot \epsilon$

Where:
* $z$: The sampled latent variable vector that we want to obtain.
* $\mu$: The mean vector, which is a deterministic output of the encoder network (a function of the input $x$, i.e., $\mu(x)$).
* $\sigma$: The standard deviation vector, also a deterministic output of the encoder network (derived from the log-variance output, $\sigma(x) = \exp(\frac{\log \sigma^2(x)}{2})$).
* $\epsilon$: A sample from a standard normal distribution, $\mathcal{N}(0, I)$. This is the only stochastic part, and it's independent of the model's parameters.
* $\odot$: Denotes element-wise multiplication (Hadamard product).

![[Pasted image 20250611191708.png]]

**Why it's Crucial for VAEs:**
By using this trick:

1.  **Enables Backpropagation:** The entire computation graph from the encoder's parameters ($\mu, \sigma$) to the loss function becomes deterministic. The randomness is factored out into $\epsilon$, which is external to the network's trainable parameters. This allows gradients to flow back through $\mu$ and $\sigma$ to the encoder, enabling its training using standard optimization techniques like stochastic gradient descent.
2.  **Learnable Parameters:** The encoder can learn to output appropriate $\mu$ and $\sigma$ values, which define the shape and location of the approximate posterior $q(z|x)$, because the gradients can now reach those parameters.
3.  **Efficiency:** It makes the training process efficient and practical, as we don't need to use more complex, gradient-free optimization methods for the sampling step.


---
# Variational Inference
![[Pasted image 20250611191122.png]]
**What it is:**
Variational Inference is a method used to approximate intractable posterior distributions in latent variable models. In these models, some variables (latent variables) are not directly observed but are inferred from other observed variables, allowing for the description of complex distributions over visible variables.

**Why it's needed:**
In Bayesian statistics, the goal is often to determine the posterior distribution $p(z|x)$ of a latent variable $z$ given some observed data $x$. This posterior distribution is calculated using Bayes' rule: $p_{\theta}(z|x) = \frac{p_{\theta}(x|z)p_{\theta}(z)}{p_{\theta}(x)}$.
The problem arises because computing the denominator, $p_{\theta}(x)$ (also known as the "evidence" or "normalizing constant"), is computationally intractable. This is due to it involving an integral over the entire latent space, which becomes impossible to compute in high-dimensional spaces. Since the true posterior cannot be computed directly, variational inference provides an approximation.

**How it works (in VAEs):**
Instead of computing the true posterior $p(z|x)$, Variational Inference introduces an approximate distribution, $q(z|x)$. This approximation is often a Gaussian distribution with parameters (mean $\mu(x)$ and standard deviation $\sigma(x)$) that are learned by an encoder network. The goal is to make this approximate distribution as close as possible to the true posterior. The closeness is measured using the Kullback-Leibler (KL) divergence, $D_{KL}(q(z|x)||p(z|x))$.

### The Need for the ELBO Method (Evidence Lower Bound)

Since the true posterior $p(z|x)$ cannot be directly computed, the KL divergence cannot be minimized directly. This is where the Evidence Lower Bound (ELBO) comes in.

**What it is:**
ELBO is the objective function that Variational Autoencoders (VAEs) maximize during training. It is derived from Bayes' theorem and represents a lower bound on the marginal log-likelihood $\log p(x)$.
The relationship is expressed as: $\log p(x) \ge \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}(q(z|x)||p(z))$.
The ELBO itself is defined as: $ELBO = \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}(q(z|x)||p(z))$.
The two main components of ELBO are:

* **Expected Log-Likelihood (Reconstruction Term):** $\mathbb{E}_{q(z|x)}[\log p(x|z)]$. This term measures how well the model can reconstruct the input data $x$ from its latent representation $z$. Maximizing this term encourages the model to generate reconstructions $\hat{x}$ that are similar to the original input $x$. If this term is high, the model is not reconstructing data well.
* **KL Divergence (Regularization Term):** $D_{KL}(q(z|x)||p(z))$. This term measures the difference between the approximate posterior distribution $q(z|x)$ (learned by the encoder) and a prior distribution $p(z)$ (typically a standard normal distribution $\mathcal{N}(0,I)$). Minimizing this term acts as a regularizer, ensuring that the latent space is well-structured and the approximate posterior doesn't stray too far from the prior. If this term is high, the latent space is not well structured.

**Why it's essential for VAEs:**
Since maximizing the true log-likelihood $\log p(x)$ is intractable, VAEs instead maximize the ELBO. Equivalently, VAEs minimize the negative ELBO as their loss function: $\mathcal{L}_{VAE} = -\mathbb{E}_{q(z|x)}[\log p(x|z)] + D_{KL}(q(z|x)||p(z))$.
This objective function creates a balance:

* It ensures good reconstruction quality.
* It regularizes the latent space, forcing the approximate posterior to be close to the chosen prior.

This balance is crucial for VAEs to learn meaningful and well-behaved latent representations that can be used for generation and other tasks. The relative importance of these two terms can sometimes be controlled by a weight $\beta$ (in $\beta$-VAEs).

---

### Numerical Example: Calculating VAE Components

**Given Parameters:**

* Latent space is assumed to follow a Gaussian distribution: $z \sim \mathcal{N}(\mu, \sigma^2I)$, where $\mu$ and $\sigma$ are predicted by the encoder.
* Encoder network outputs:
    * Mean vector: $\mu=[0.5, -0.5]$
    * Log variance vector: $\log \sigma^2 = [-1, -2]$
* Standard normal sample (noise): $\epsilon=[0.2, -0.8]$, where $\epsilon \sim \mathcal{N}(0,I)$
* Prior distribution for KL divergence is a standard normal distribution: $\mathcal{N}(0,I)$

#### Step 1: Compute the actual values of $\sigma$

**Formula Used:**
The document states that $\sigma = \exp(\frac{\log \sigma^2}{2})$. This formula converts the log-variance ($\log \sigma^2$) output by the encoder into the standard deviation ($\sigma$). This is because neural networks often output log-variance instead of variance directly to ensure that the variance is always positive, as $\exp()$ will always yield a positive result.

**Calculation:**
Given $\log \sigma^2 = [-1, -2]$, we compute $\sigma$ for each element:

* For the first element: $\sigma_1 = \exp(\frac{-1}{2}) = \exp(-0.5)$
* For the second element: $\sigma_2 = \exp(\frac{-2}{2}) = \exp(-1)$

Using the provided approximations:

* $\exp(-0.5) \approx 0.6065$
* $\exp(-1) \approx 0.3679$

Therefore, $\sigma=[0.6065, 0.3679]$.

#### Step 2: Perform the reparameterization trick by generating a sample $z$

**Formula Used:**
The reparameterization trick is given by the formula: $z = \mu + \sigma \odot \epsilon$.
* $z$: The sampled latent variable vector.
* $\mu$: The mean vector predicted by the encoder.
* $\sigma$: The standard deviation vector computed in Step 1.
* $\epsilon$: A sample from a standard normal distribution $\mathcal{N}(0,I)$, which introduces the stochasticity.
* $\odot$: Denotes element-wise multiplication.

This trick is crucial because it separates the deterministic part ($\mu + \sigma \odot \epsilon$) from the stochastic part ($\epsilon$), allowing backpropagation to work through the VAE's sampling process.

**Calculation:**
Given:

* $\mu=[0.5, -0.5]$
* $\sigma=[0.6065, 0.3679]$ (from Step 1)
* $\epsilon=[0.2, -0.8]$

We compute $z$ element-wise:

* $z_1 = 0.5 + (0.6065 \times 0.2) = 0.5 + 0.1213 = 0.6213$
* $z_2 = -0.5 + (0.3679 \times -0.8) = -0.5 - 0.2943 = -0.7943$

So, the sampled latent vector is $z=[0.6213, -0.7943]$.

#### Step 3: Compute the KL divergence term of the loss function

**Formula Used:**
The KL divergence between the approximate posterior $q(z|x)=\mathcal{N}(\mu, \sigma^2I)$ and the prior $p(z)=\mathcal{N}(0,I)$ is given by: $D_{KL}(q(z|x)||p(z))=\frac{1}{2}\sum_{i=1}^{d}(\sigma_i^2 + \mu_i^2 - 1 - \log \sigma_i^2)$.
* $d$: The dimensionality of the latent space (in this case, 2).
* $\sigma_i^2$: The variance of the $i$-th latent dimension.
* $\mu_i$: The mean of the $i$-th latent dimension.
* $\log \sigma_i^2$: The log variance of the $i$-th latent dimension (as output by the encoder).

This KL divergence term acts as a regularizer in the VAE loss function, encouraging the approximate posterior to stay close to the prior.

**Calculation:**
Given:

* $\mu=[0.5, -0.5]$
* $\log \sigma^2 = [-1, -2]$
* $\sigma=[0.6065, 0.3679]$ (from Step 1)

First, calculate $\sigma_i^2$ (variance) from the $\sigma$ values:

* $\sigma_1^2 = 0.6065^2 \approx 0.368$
* $\sigma_2^2 = 0.3679^2 \approx 0.135$

Now substitute the values into the KL divergence formula, summing over the two dimensions ($i=1,2$):
$D_{KL} = \frac{1}{2}[(\sigma_1^2 + \mu_1^2 - 1 - \log \sigma_1^2) + (\sigma_2^2 + \mu_2^2 - 1 - \log \sigma_2^2)]$
$D_{KL} = \frac{1}{2}[(0.368 + 0.5^2 - 1 - (-1)) + (0.135 + (-0.5)^2 - 1 - (-2))]$
$D_{KL} = \frac{1}{2}[(0.368 + 0.25 - 1 + 1) + (0.135 + 0.25 - 1 + 2)]$
$D_{KL} = \frac{1}{2}[(0.618) + (1.385)]$
$D_{KL} = \frac{1}{2}[2.003]$
$D_{KL} = 1.0015$

So, the KL divergence is approximately 1.0015. This value would then be added to the reconstruction loss (negative expected log-likelihood) to form the total VAE loss function that is minimized during training.
