# Introduction (RNN) vs (NN)

![[Pasted image 20250107181114.png]]
Conventional Neural Networks (NN), such as Multi-layer Perceptrons (MLPs), are feedforward networks where information flows in one direction—from input to output. These networks are designed for tasks where data points are independent of each other, and they operate by mapping an input to a fixed output. While effective for tasks like classification and regression, conventional NNs struggle with sequential or time-dependent data because they do not capture the temporal dependencies or relationships between data points across time.

In contrast, **Recurrent Neural Networks (RNNs)** are specifically designed to handle sequential data. Unlike conventional NNs, RNNs have feedback loops in their architecture that allow information to persist over time. This enables RNNs to "remember" previous inputs and use this memory to make more informed predictions or decisions based on the current input and past inputs. RNNs are widely used in tasks such as speech recognition, language modeling, and time series forecasting, where the sequence and context of the data play a critical role. By maintaining a hidden state or memory of past information, RNNs can model time dependencies and relationships within sequences, providing them with the ability to process and understand sequential data effectively.

![[Pasted image 20250107185156.png]]

# Characteristics of Sequential data
### 1. **The Importance of Temporal Order in Time Series**  

Time series data represents observations collected sequentially over time, where the order of data points is crucial. The value at a given time  **t**  often depends on the values that precede it, capturing trends, patterns, or cycles. For example, stock prices, weather patterns, and sensor readings exhibit dependencies that models leverage to make predictions. If the temporal order is disrupted (e.g., through permutation), the relationships between consecutive values break down, eliminating meaningful patterns and making the data unusable for time-series models like ARIMA, LSTMs, or other forecasting algorithms.

Permuting the order of values destroys the "signal" or inherent structure within the data. A stock price sequence such as 100, 102, 105, 110, 115 shows a clear increasing trend. After permuting, the order becomes 115, 102, 110, 100, 105, causing the loss of that trend and making it unrecognizable. In essence, the temporal sequence holds crucial information about how data evolves, and breaking this order results in meaningless data that cannot be interpreted or analyzed properly.

---

### 2. **Bag of Words (BoW) vs Sequence Analysis**  

**Bag of Words (BoW)** is a simple and widely used text representation model that disregards word order and syntax. It transforms a document or sentence into a collection of its constituent words, where each word is treated as a unique feature. For example, the sentence "The cat sat on the mat" would be represented in BoW as `{the: 2, cat: 1, sat: 1, on: 1, mat: 1}`. This approach focuses on counting the occurrences of words and doesn't account for the relationships between them. The major drawback of BoW is that it loses the meaning derived from word order and grammar, making it difficult to capture nuances like subject-verb agreement or word dependencies.

In contrast, **Sequence Analysis** captures the order and contextual relationships between words in a sentence. Models such as Recurrent Neural Networks (RNNs) and Transformers (e.g., BERT) analyze text in sequence, maintaining the dependencies between words. This allows for a richer understanding of meaning. For instance, in the sentence "I enjoy reading books," Sequence Analysis understands that "enjoy" and "reading" are connected and that "reading books" forms a meaningful unit. Sequence analysis is crucial for tasks like sentiment analysis, language translation, and text generation, where the context and order of words heavily influence the interpretation of the text.

---

### 3. **Biological Sequences in DNA and Proteins**  

Biological data, such as DNA, RNA, or protein sequences, is inherently sequential, where the order of symbols carries significant biological information. DNA sequences are composed of nucleotides (adenine [A], thymine [T], cytosine [C], and guanine [G]), while RNA replaces thymine with uracil (U). These nucleobases form the genetic code that dictates cellular functions and the development of organisms. A mutation or change in the sequence of nucleotides can lead to alterations in gene expression, potentially causing diseases or abnormalities. Thus, the correct sequence is essential for maintaining biological processes.

Protein sequences, composed of amino acids encoded by codons in DNA or RNA, follow a similar sequential structure. The order of amino acids in a protein determines its structure and function. For example, a change in a single amino acid in a protein sequence can disrupt its folding, potentially leading to non-functional or harmful proteins.  The sequential nature of biological data provides essential insights into evolutionary patterns, disease mechanisms, and therapeutic developments.


# What Makes RNN Special?

Recurrent Neural Networks (RNNs) are a unique type of neural network specifically designed for tasks involving sequential data. Their standout features make them highly effective for handling time-dependent information. Here’s a breakdown of what makes RNNs special:

- **Internal Memory**: One of the most important features of RNNs is their internal memory. Unlike traditional feedforward neural networks, RNNs have the ability to "remember" previous inputs through a feedback loop. This memory allows them to use context from past inputs to influence how they process new data, making them ideal for handling sequential data where the past is important for interpreting the present.

- **Sequential Data Processing**: RNNs excel at processing data that comes in a sequence, where the order of elements matters. This is crucial for tasks such as speech recognition, language translation, and text generation, where understanding the sequence of words or sounds is essential for producing the correct output. RNNs are designed to take in one element of the sequence at a time and adjust their memory as they move through the sequence.

- **Contextual Understanding**: RNNs analyze each input in relation to what they have seen before. This means that they don’t just treat each input in isolation but rather understand the current input in the context of all previous inputs. This ability to "look back" and use prior information is essential for tasks like understanding a sentence in NLP, where the meaning of a word can depend on the words around it.

- **Dynamic Processing**: As RNNs process each new data point, they continuously update their internal memory. This allows them to adapt to changing patterns in the data over time. For instance, in time series forecasting, RNNs can adjust their predictions based on the most recent data while keeping track of the broader trend. This dynamic processing ensures that RNNs remain flexible and can learn from new data in an ongoing manner.


## About RNN
### Applications of Recurrent Neural Networks (RNNs)

Recurrent Neural Networks (RNNs) are designed to handle sequential data and capture dependencies over time. Below are some key applications of RNNs:

- **Language Translation**: RNNs are used to translate text between languages, such as in Google Translate.
- **Speech Recognition**: RNNs help convert spoken language into written text, as seen in systems like Apple's Siri and Amazon's Alexa.
- **Time Series Forecasting**: RNNs predict future values in time-dependent data, such as stock prices or weather forecasts.
- **Text Generation**: RNNs can generate text similar to a given input, useful for tasks like conversation responses or document summarization.
- **Sentiment Analysis**: RNNs are used to analyze the sentiment of text, helping to determine whether a review is positive or negative.

### RNN Architecture 

- **Input Layer:** This layer receives the initial element of the sequence data. For example, in a sentence, it might receive the first word as a vector representation.
- **Hidden Layer**: The heart of the RNN, the hidden layer contains a set of interconnected neurons. Each neuron processes the current input along with the information from the previous hidden layer’s state. This “state” captures the network’s memory of past inputs, allowing it to understand the current element in context.
- **Activation Function:** This function introduces non-linearity into the network, enabling it to learn complex patterns. It transforms the combined input from the current input layer and the previous hidden layer state before passing it on.
- **Output Layer**: The output layer generates the network’s prediction based on the processed information. In a language model, it might predict the next word in the sequence.
- **Recurrent Connection**: A key distinction of RNNs is the recurrent connection within the hidden layer. This connection allows the network to pass the hidden state information (the network’s memory) to the next time step. It’s like passing a baton in a relay race, carrying information about previous inputs forward.

![[Pasted image 20250107191410.png]]

# Types of Recurrent Neural Networks (RNNs)

![[Pasted image 20250107194112.png]]
**Red: input (x)**
**Green: hidden layer (H)**
**Blue: output layer (Y)**
### One-to-One RNN
- Maps a single input to a single output.
- Also known as "vanilla RNN" or "fully connected RNN."
- Input: Fixed-length sequence of vectors.
- Output: Fixed-length sequence of vectors.
- Processes input one time step at a time, using the output of one step as input for the next.
- Suited for tasks requiring mapping fixed-length input to fixed-length output.
- **Not well-suited** for processing variable-length inputs.
- **Applications:** 
  - Text generation
  - Word prediction
  - Sentence prediction
  - Stock market predictions

---

### One-to-Many RNN
- Maps a single input to a sequence of outputs.
- Input: Typically a fixed-length vector (e.g., an image or text).
- Output: Sequence of vectors (e.g., words or notes).
- Often used for generating sequential data based on a single input.
- **More complex to train** than one-to-one RNNs.
- **Applications:** 
  - Auto image captioning
  - Music generation

---

### Many-to-One RNN
- Maps a sequence of inputs to a single output.
- Input: Sequence of vectors (e.g., words or time series data).
- Output: Single vector (e.g., classification label or numerical prediction).
- Used for tasks involving classification or prediction based on input sequences.
- **Simpler to train** than many-to-many RNNs.
- **Applications:** 
  - Sentiment analysis
  - Language translation

---

### Many-to-Many RNN
- Maps a sequence of inputs to a sequence of outputs.
- Input: Sequence of vectors (e.g., words or time series data).
- Output: Sequence of vectors (e.g., words or notes).
- Used for tasks involving processing and translating sequences.
- **More complex to train** than many-to-one RNNs.
- **Applications:** 
  - Machine translation
  - Text summarization



# Understanding Word Representation

Word representation refers to how words are encoded into a numerical format that computers can process. Two main techniques are used: **1-hot encoding** and **word embeddings**.

---

## 1-Hot Representation

### Definition
- A simple method where each word is represented as a unique vector in a high-dimensional space.
- Each vector has a single "1" at the position corresponding to the word and "0"s elsewhere.

### Example
If the vocabulary is `["book", "soft", "teddy bear"]`:
- "book" → `[1, 0, 0]`
- "soft" → `[0, 1, 0]`
- "teddy bear" → `[0, 0, 1]`

### Limitations
- **No Similarity Information**: All words are equally distant in this representation, meaning it cannot capture relationships between words (e.g., "soft" and "teddy bear").
- **High Dimensionality**: For large vocabularies, the vector size becomes enormous.

---

## Word Embeddings

### Definition
- A dense, low-dimensional vector representation of words where similar words are closer in the vector space.
- Unlike 1-hot encoding, embeddings capture relationships and meanings between words.

---

### Word2Vec

#### Overview
- **transforms Words into Vectors**: Each word is represented as a fixed-size vector in a continuous vector space.
- **Captures Semantic Similarity**: Words with similar meanings (e.g., "king" and "queen") are placed closer together in the vector space.
- **Facilitates Downstream Tasks**: These embeddings are used in tasks like sentiment analysis, machine translation, and question answering.

---

### Skip-gram Model

#### Objective
- It is a method used in Word2Vec to learn word embeddings. Its goal is to predict the context words surrounding a target word, capturing semantic relationships between words.
- **Predicts Context Words**: Given a target word $t$, the Skip-Gram model predicts the words that appear in its surrounding context within a fixed window size. 
- **Learns Word Embeddings**: During training, the model adjusts word vector representations so that words appearing in similar contexts are closer in the vector space.
- **Captures Semantic Meaning**: Words with similar contexts (e.g., "cat" and "dog") end up having similar embeddings.

#### Formula
$$ P(c|t) = \frac{\exp(\theta_c^\top e_t)}{\sum_{j=1}^{|V|} \exp(\theta_j^\top e_t)} $$

Here: 
- $\theta_c$: Vector for context word $c$. 
- $e_t$: Vector for target word $t$. 
- $|V|$: Vocabulary size.


---

### Negative Sampling

#### Definition
- It is a set of binary classifiers using logistic regression that aim at assessing how a given context and a given target word are likely to appear simultaneously, with the models being trained on sets of k negative examples and 1 positive example.  
- A method to make the Skip-gram model computationally efficient by focusing on a subset of word pairs.

#### Formula
$$
P(y = 1 | c, t) = \sigma(\theta_t^\top e_c)
$$

Where:
- $\sigma(x) = \frac{1}{1 + \exp(-x)}$ : Sigmoid function.
- $\theta_t$: Target word vector.
- $e_c$: Context word vector.
- $y =0/1$ : if the sample is true positive for the context word

### Key Advantage
- Reduces computation by training on a small number of negative samples instead of the entire vocabulary.

---

### GloVe (Global Vectors for Word Representation)

#### Overview
- A word embedding technique that uses a **co-occurrence matrix** to capture global statistical information about word relationships.

#### Objective
- Minimize the difference between the dot product of word vectors and the logarithm of their co-occurrence count.

#### Formula
$$
J(\theta) = \frac{1}{2} \sum_{i,j=1}^{|V|} f(X_{i,j}) \left( \theta_i^\top e_j + b_i + b_j' - \log(X_{i,j}) \right)^2
$$

Where:
- $X_{i,j}$: Co-occurrence count of word \( i \) with word \( j \).
- $(X_{i,j})$: Weighting function to handle zero counts.
- $\theta_i, e_j$: Word vectors for \( i \) and \( j \).
- $b_i, b_j'$: Bias terms.

#### Final Embedding
The final word embedding is computed as:
$$
e_w^{(final)} = \frac{e_w + \theta_w}{2}
$$

---

### Summary

1. **1-hot Encoding**: Simple but lacks semantic information and has high dimensionality.
2. **Word Embeddings**: Dense vectors that capture relationships between words.
3. **Word2Vec**:
   - **Skip-gram**: Predicts target words from context.
   - **Negative Sampling**: Improves computational efficiency.
4. **GloVe**: Combines global co-occurrence statistics with local context to learn embeddings.

## Comparing words

![[Pasted image 20250107202901.png]]

# Problems on RNN

![[Pasted image 20250107205907.png]]
![[Pasted image 20250107205909.png]]
![[Pasted image 20250107205912.png]]
![[Pasted image 20250107231305.png]]
![[Pasted image 20250107231317.png]]
![[Pasted image 20250107231326.png]]
![[Pasted image 20250107231337.png]]

# Backpropagation through time

![[Pasted image 20250107231404.png]]
![[Pasted image 20250107231412.png]]
![[Pasted image 20250107231429.png]]

![[Pasted image 20250107231556.png]]
![[Pasted image 20250107231612.png]]![[Pasted image 20250107231638.png]]
![[Pasted image 20250107231650.png]]

![[Pasted image 20250107233323.png]]


# Drawbacks of RNN

- **Vanishing and exploding gradients:** These problems occur when the backpropagation algorithm fails to update network weights effectively.Vanishing gradients happen when gradients become too small, making parameter updates insignificant. Exploding gradients happen when gradients increase exponentially, causing the RNN to become unstable.
-  **Difficulty training**: Training RNNs can be challenging, especially for long sequences.
-  **Slower than other neural networks**: RNNs are computationally slower than other neural network architectures.
- **Difficulty with parallelization**: RNNs process data sequentially, making parallelization difficult and slowing down training. This means that RNNs can't take full advantage of modern hardware architectures.
-  **Difficulty with long-term dependencies**: RNNs can learn short-term dependencies, but they have trouble with long-term dependencies. This means they might forget or ignore information that's too far back in the sequence.

# Long Short-Term Memory (LSTM)

A Long Short-Term Memory (LSTM) model is a type of Recurrent Neural Network (RNN) designed to learn and predict sequential data. LSTMs are particularly effective at capturing long-term dependencies by incorporating specialized gates that manage short-term and long-term memory. These gates include:

- **Input Gate**: Decides which information to store in the memory cell based on the current input and previous hidden state.
- **Forget Gate**: Decides which information to discard from the memory cell.
- **Output Gate**: Decides which information to output based on the current input and previous hidden state.

## Pros
- **Handling Long Sequences**: Effective at capturing long-range dependencies in sequential data, making them suitable for tasks like NLP and time series analysis.
- **Avoiding Vanishing Gradient Problem**: Gating mechanisms help prevent gradients from vanishing during backpropagation.
- **Handling Variable-Length Sequences**: Can dynamically adjust their internal states to input sequences of varying lengths, which is useful in real world scenarios.
- **Memory Cell**: Maintains important information while discarding irrelevant details, enabling context retention over long sequences.
- **Gradient Flow Control**: Gates like the forget gate allow effective propagation of gradients back in time.

## Cons
- **Computational Complexity**: More resource-intensive than simpler architectures like feedforward networks or basic RNNs.
- **Overfitting**: Susceptible to overfitting with insufficient data, requiring regularization techniques like dropout.
- **Hyperparameter Tuning**: Requires careful tuning of parameters like learning rate, number of units, and sequence length, which can be time-consuming.
- **Limited Interpretability**: Functions as a "black-box" model, making it difficult to interpret its decision-making process.
- **Long Training Times**: Training deep LSTMs on large datasets can be time-intensive and require powerful hardware like GPUs or TPUs.

## Numerical

![[Pasted image 20250108004832.png]]
![[Pasted image 20250108004842.png]]![[Pasted image 20250108004855.png]]
![[Pasted image 20250108004902.png]]


# Gated Recurrent Unit (GRU)

A Gated Recurrent Unit (GRU) is a type of Recurrent Neural Network (RNN) architecture similar to Long Short-Term Memory (LSTM). GRUs are designed to model sequential data by selectively remembering or forgetting information over time, but they have a simpler architecture compared to LSTMs, making them more computationally efficient and easier to train(GRUs are a popular alternative to LSTMs for sequential data modeling).

## Key Features

- **Simpler Architecture**: GRUs have fewer parameters than LSTMs, leading to faster training and reduced computational cost.
- **Gates**:
  - **Reset Gate**: Controls how much of the previous hidden state to forget.
  - **Update Gate**: Determines how much of the new candidate activation vector to incorporate into the current hidden state.
- **Unified State**: 
  - GRUs replace the separate **memory cell state** and **hidden state** used in LSTMs with a single state called the **candidate activation vector**.
  - This hidden state serves as both the memory and the output, simplifying the architecture while retaining the ability to capture sequential dependencies.

## GRU Architecture

The GRU architecture consists of the following components:

- **Input Layer**: Takes sequential data (e.g., words or time series) and feeds it into the GRU.  
- **Hidden Layer**: Updates the hidden state at each time step based on the current input and previous hidden state, serving as the network's memory.  
- **Reset Gate**: Determines how much of the previous hidden state to forget, producing values between 0 and 1.  
- **Update Gate**: Controls how much of the candidate activation vector to incorporate into the new hidden state, producing values between 0 and 1.  
- **Candidate Activation Vector**: A modified version of the previous hidden state, reset by the reset gate and combined with the current input, squashed between -1 and 1 using a tanh activation.  
- **Output Layer**: Produces the network's output (e.g., a number, sequence, or probability distribution) based on the final hidden state.

## Pros:
- GRU networks are similar to Long Short-Term Memory (LSTM) networks, but with fewer parameters, making them computationally less expensive and faster to train.
- GRU networks can handle long-term dependencies in sequential data by selectively remembering and forgetting previous inputs.
- GRU networks have been shown to perform well on a variety of tasks, including natural language processing, speech recognition, and music generation.
- GRU networks can be used for both sequence-to-sequence and sequence classification tasks
## Cons:
- GRU networks may not perform as well as LSTMs on tasks that require modeling very long- term dependencies or complex sequential patterns.
- GRU networks may be more prone to overfitting than LSTMs, especially on smaller datasets.
- GRU networks require careful tuning of hyperparameters, such as the number of hidden units and learning rate, to achieve good performance.
- GRU networks may not be as interpretable as other machine learning models, since the gating mechanism can make it difficult to understand how the network is making predictions


## Numerical 

![[Pasted image 20250108012738.png]]

![[Pasted image 20250108012629.png]]

![[Pasted image 20250108012654.png]]
![[Pasted image 20250108012719.png]]
