~~What if domain is large but dataset is small~~(normalization and LOOCV/LPOCV)

~~Any rule for the ratio of number of neurons~~(PCA)(2/3 rule)

~~BFGS~~, LBFGS
~~List of all AF(numerical) and cost function*~~

~~Alternates to AF*(THE FANCY ONES)~~

Types of NN

ADAGRAD, ADAM etc the math grunt*

GD and SGD from scratch

~~Math behind the NN at every step and layer* ~~

---
~~Types of Normalization~~

~~Table the AF's~~

~~Make a sample NN~~



**Phase 1: Activation Functions (AFs)**
1. **Mathematical Exploration**
    
    - Define all parameters used in the formulas.
    - Understand and document the characteristics of each AF:
    - Identify aliases (e.g., LeCun = scaled tanh) and variations.
        
2. **Research and Validation**
    
    - Search for research papers using each AF and document:
        
        - Use cases
            
        - Performance
            
        - Pros and cons
            
3. **Implementation and Experimentation**
    
    - Code all activation functions.
        
    - Run them on a common benchmark problem.
        
    - Compare their effectiveness.
        
    - Document which AFs perform best for your use case.
        

---

### ⚖️ **Phase 2: Normalization Techniques**

1. **Theory and Documentation**
    
    - Study and describe all normalization techniques(+Gaslobato Normalization):
            
            
    - Organize these into a formal paper/report.
        
2. **Experimentation**
    
    - Combine the best activation functions (from Phase 1) with different normalizations.
        
    - Run tests on the same benchmark problem.
        
    - Analyze their impact on convergence, speed, and accuracy.
        

---

### ⚙️ **Phase 3: Optimizers**

1. **List of Optimizers to Cover**
    
    - Gradient Descent (GD)
        
    - Stochastic Gradient Descent (SGD)
        
    - Adam
        
    - BFGS
        
    - L-BFGS
        
    - Self-Scaled Broyden (SS-Broy)
        
2. **Numerical Analysis and Wolfe Line Search**
    
    - Understand and extract equations for all optimizers.
        
    - Implement Wolfe Line Search:
        
        - Standalone
            
        - Integrate with optimizers like BFGS, SS-Broy, etc.
            
    - Evaluate optimizer performance with and without Wolfe.
        

---

### 📊 **Optional: Dimensionality Reduction**

- Deep dive into PCA:
    
    - Understand the intuition and math behind PCA.
        
    - Apply PCA to reduce input dimensions on your problem.
        
    - Assess whether PCA helps AF/Normalization/Optimizer combos.
### 🗂 **Workflow and Tools**

- Use **Mendeley** to:
    
    - Organize papers on AFs, normalizations, and optimizers.
        
    - Summarize key points and keep track of references (especially pros/cons and results) from each for future reference.

Machine learning application with citations of paper and not articles and online materials and link it to deep learning and proceed with explanation of NN.

Choosing the number of data points and neurons, the criteria to decide. highlight why only 1 input neurons for us with multiple examples 

Goslowbato transformation 