# Backpropagation in  CNN Using Inverse Filters and Chain Rule

Backpropagation in CNNs is the process of updating parameters (like filters) during training using the gradients of a loss function with respect to these parameters. (Derivative of loss with respect to each cell is backpropagated)

---

## 1. Loss Gradient with Respect to Input ($\frac{\partial L}{\partial X_i}$)

The first formula calculates the gradient of the loss ($L$) with respect to the input ($X_i$):

$$
\frac{\partial L}{\partial X_i} = \sum_{k=1}^{M} \frac{\partial L}{\partial O_k} \cdot \frac{\partial O_k}{\partial X_i}
$$

### Explanation:
- $O_k$: Represents the output of the convolution operation after applying the filter to the input $X$.
- $\frac{\partial L}{\partial O_k}$: This is the gradient of the loss with respect to the $k$-th output element.
- $\frac{\partial O_k}{\partial X_i}$: This is the gradient of the $k$-th output with respect to the $i$-th input element. This involves an **inverse convolution operation**, where the output's influence is distributed back over the input space.

The backpropagation step essentially applies a **convolution operation** between the **loss gradient ($\frac{\partial L}{\partial O}$)** and a **flipped version of the filter** (used during the forward pass) to compute $\frac{\partial L}{\partial X}$.

---

## 2. Loss Gradient with Respect to Filter ($\frac{\partial L}{\partial F}$)

The second formula calculates the gradient of the loss ($L$) with respect to the filter $F$:

$$
\frac{\partial L}{\partial F} = \frac{\partial L}{\partial O} \ast \frac{\partial O}{\partial F}
$$

For each element of $F_i$:

$$
\frac{\partial L}{\partial F_i} = \sum_{k=1}^{M} \frac{\partial L}{\partial O_k} \cdot \frac{\partial O_k}{\partial F_i}
$$

### Explanation:
- $\frac{\partial O_k}{\partial F_i}$: This is the local gradient representing how $O_k$ depends on $F_i$.
- $\frac{\partial L}{\partial O_k}$: Propagated gradient from the loss to the $k$-th output element.
- $\frac{\partial L}{\partial F_i}$: Total gradient contribution for $F_i$, computed as a weighted sum of all outputs.

In the context of convolution, this involves performing a **convolution operation** between the gradient $\frac{\partial L}{\partial O}$ and the input $X$ to update the filter $F$.

---

## Inverse Filter During Backpropagation

The **inverse filter** concept refers to flipping the filter kernel used in the forward convolution during the backward pass. This flipping aligns the spatial contribution of gradients during backpropagation.

1. For $\frac{\partial L}{\partial X_i}$:
   - Convolve $\frac{\partial L}{\partial O}$ with the flipped version of the filter $F$ (used in the forward pass).

2. For $\frac{\partial L}{\partial F_i}$:
   - Convolve $\frac{\partial L}{\partial O}$ with the **input $X$**. This computes how the filter should adjust its weights to reduce the loss.

---

## Steps in Backpropagation

1. Compute the gradient of the loss with respect to the output of the convolution layer ($\frac{\partial L}{\partial O}$) using the chain rule.
2. Use $\frac{\partial L}{\partial O}$ and perform an **inverse convolution** with the flipped filter to compute $\frac{\partial L}{\partial X}$.
3. Use $\frac{\partial L}{\partial O}$ and perform a **convolution** with the input $X$ to compute $\frac{\partial L}{\partial F}$.
4. Update parameters ($F$) using gradient descent:

$$
F \leftarrow F - \eta \cdot \frac{\partial L}{\partial F}
$$

where $\eta$ is the learning rate.

---

## Intuition

- The **loss gradient with respect to input ($\frac{\partial L}{\partial X}$)** explains how the loss changes if the input changes, propagating error signals backward into the network.
- The **loss gradient with respect to the filter ($\frac{\partial L}{\partial F}$)** tells how the filter should adapt its weights to minimize the loss in future iterations.

This is the essence of learning in CNNs—iteratively updating filters to reduce the overall loss.





# Backpropagation In CNN through Matrix Multiplication

Backpropagation in Convolutional Neural Networks (CNNs) involves propagating the error from the output layer back through the network to update the weights and biases. The process heavily relies on matrix multiplication for computational efficiency. Here's an explanation of backpropagation in CNNs focusing on the matrix operations:



## 1. Forward Pass Recap

The forward pass computes the output of the CNN by applying:

- **Convolution**: Sliding filters (kernels) over input images to compute feature maps.
- **Activation**: Applying non-linear functions (e.g., ReLU) to introduce non-linearity.
- **Pooling**: Downsampling the feature maps to reduce spatial dimensions.
- **Fully Connected Layers**: Using matrix multiplication to compute final predictions.

Intermediate results (activations and feature maps) are stored for use during backpropagation.



## 2. Backward Pass Overview

The goal of backpropagation is to compute gradients of the loss $L$ with respect to weights ($W$) and biases ($b$) to minimize the loss using gradient descent.


## 3. Gradient Computation Steps

### (a) Compute Gradient at Output Layer

At the output layer:

1. Compute the gradient of the loss $L$ with respect to the output $\mathbf{z}$ (pre-activation values):
   $$\
   \frac{\partial L}{\partial \mathbf{z}} = \frac{\partial L}{\partial \hat{\mathbf{y}}} \cdot \frac{\partial \hat{\mathbf{y}}}{\partial \mathbf{z}}
   $$
   Here, $\hat{\mathbf{y}}$ is the predicted output (e.g., softmax probabilities).

2. Backpropagate the error through the fully connected layer:
   $$
   \frac{\partial L}{\partial W} = \mathbf{a}^\top \cdot \frac{\partial L}{\partial \mathbf{z}}, \quad \frac{\partial L}{\partial \mathbf{b}} = \frac{\partial L}{\partial \mathbf{z}}
   $$
   Where:
   - $\mathbf{a}$ is the activation from the previous layer (input to the fully connected layer).
   - $W$ is the weight matrix.

   **Matrix multiplication** efficiently computes $\frac{\partial L}{\partial W}$.

---

![[Pasted image 20250108065354.png]]
# **What is Data Augmentation?**

Data augmentation is a technique used in machine learning and deep learning to artificially expand the size and diversity of a dataset by applying transformations to existing data. These transformations can include flipping, rotating, cropping, scaling, and more. It is commonly used in tasks like image classification, object detection, and natural language processing.

---

## **Key Features of Data Augmentation**
- **Purpose:**
  - To improve model performance and generalization by introducing variability in the dataset.
  
- **Applications:**
  - Used in supervised learning tasks to balance datasets, especially in cases of class imbalance or small dataset sizes.

---

## **Pros of Data Augmentation**
1. **Improves Generalization:**
   - By exposing the model to diverse variations, it performs better on unseen data.

2. **Reduces Overfitting:**
   - Helps prevent the model from memorizing the training data.

3. **Balances Imbalanced Datasets:**
   - Synthetic samples can address class imbalance, improving the model's robustness.

4. **Cost-Effective:**
   - Eliminates the need for expensive data collection and labeling.

5. **Improves Robustness:**
   - Models become more robust to variations in real-world scenarios (e.g., changes in lighting or noise).

---

## **Cons of Data Augmentation**
1. **Risk of Overfitting to Augmented Data:**
   - Excessive augmentation can make the model overly reliant on synthetic patterns.

2. **Not Always Effective:**
   - Some augmentations may not contribute significantly to performance or might distort important features.

3. **Computational Overhead:**
   - Real-time data augmentation can increase training time and computational costs.

4. **Requires Domain Knowledge:**
   - Choosing effective augmentations often requires understanding of the specific problem domain.

5. **Risk of Introducing Noise:**
   - Poorly designed augmentations can introduce irrelevant or misleading patterns.

---

## **When to Use Data Augmentation**
- When the dataset is small or imbalanced.
- When the model is prone to overfitting.
- When diverse real-world variations are critical for the model's success.

Data augmentation is a powerful technique but must be used thoughtfully to avoid negatively impacting model performance.

## **Types of Data Augmentation **

### Image :
- Rotation
- flipping 
- scaling
- cropping
- translation
- brightness adjustment 
- contrast adjustment 
- noise injections
- color jittering
- cutout masking 
### Text:
- synonym replacement: replace words with their synonyms 
- random insertion: insert random synonym into the sentence
- random deletion
- back translation: translate to and back to the source language
- text paraphrasing : rephrase the text

### Audio:

- Time stretching
- Time shifting
- Audio clipping
- Random background noise
- Random drops and increase in volume
- Pitch shifting : adjust he pitch without affecting the tempo
### Tabular:
- Table pivoting
- Model implementation 
- shuffling attributes(columns)
- Random noise in numbers
- Data interpolation
- SMOTE (Synthetic Minority Over-sampling Technique): Generate synthetic samples for minority classes
