
SKIPPED SOLVING OF DFBETAS , DFFITS , AIC, AND BIC FOR CIE 2 CUZ ITS TOO LONG LMAO
### Preparation

In multiple linear regression (MLR), the fitted response values $\hat{y}$ are given by:

$$
\hat{y} = X \hat{\beta}
$$

Where:
- $X$: Design matrix (includes the predictor variables and the intercept).
- $\hat{\beta}$: Estimated regression coefficients.

The least squares estimator for $\hat{\beta}$ is:

$$
\hat{\beta} = (X^T X)^{-1} X^T y
$$

Substituting $\hat{\beta}$ into $\hat{y}$, we get:

$$
\hat{y} = X (X^T X)^{-1} X^T y
$$

This leads to the $\textbf{Hat Matrix}$ $H$:

$$
H = X (X^T X)^{-1} X^T
$$

Thus, the fitted values can be written as:

$$
\hat{y} = H y
$$

Properties of the Hat Matrix

Symmetric: $H = H^T$

Idempotent: $H = H^2$

Diagonal Elements: The diagonal elements $h_{ii}$ (leverage values) measure the influence of the corresponding observation on the fitted values:

$h_{ii}$ ranges from 0 to 1.

Higher $h_{ii}$ indicates greater influence on the regression fit.

Trace of H: The trace of the Hat Matrix is the sum of the diagonal elements, which equals the number of regression parameters $p$ (including the intercept):

$$
tr(H) = \sum_{i=1}^{n} h_{ii} = p
$$

| **Term**          | **Meaning**                       | **Formula**                       | **Purpose**                                   |
| ----------------- | --------------------------------- | --------------------------------- | --------------------------------------------- |
| **RSS/$SS_res$**  | Total squared errors              | $$\sum (y_i - \hat{y}_i)^2$$      | Measures overall "badness" of model fit.      |
| **MSRES**         | Average squared error (residuals) | $$\frac{\text{RSS}}{n - k - 1}$$​ | Estimates the variance of residuals.          |
| **Sigma-Squared** | True error variance (unknown)     | $$σ2≈MSRES$$                      | Describes the variability in the true errors. |

### STANDARDIZED RESIDUALS 


#### Formula for Standardized Residuals:

$$di =\frac{e_i}{\sqrt{MS_{\text{RES}}}}$$​
Where:
- $e_i$ is the **residual** for the i-th observation. A residual is the difference between the observed value and the predicted value from the regression model, i.e., $e_i = y_i - \hat{y}_i$
- $MS_{RES}$ is the **mean square residual**, which is an estimate of the **variance** of the error terms:
$$MS_{\text{RES}} = \frac{\sum_{i=1}^{n} e_i^2}{n - p}$$
where `n` is the number of observations and `p` is the number of model parameters.



#### Why Standardize the Residuals?

- **Mean of Zero**: The residuals $e_i$ have a mean of zero because the regression line minimizes the sum of squared residuals.
- **Unit Variance**: By dividing the residuals by the square root of $MS_{RES}$, the standardized residuals $d_i$ have a variance of 1 (unit variance). This allows for consistent comparison of residuals across different models and datasets.

#### Interpretation of Standardized Residuals

- $d_i$ measures how many standard deviations a given residual $e_i$ is from the mean of the residuals (which is 0).
- If $|d_i| > 3$, the observation might be an **outlier**.

#### Conclusion on Outliers

- If $|d_i| > 3$, it suggests that the residual for the $i-th$ observation is unusually large, and that observation could be an outlier.
- The threshold of **3** is commonly used because, under the assumption of normality, about **99.7%** of data points in a normal distribution lie within 3 standard deviations of the mean.








### Internally Studentized Residuals (Regular Studentized Residuals)

#### What are Internally Studentized Residuals?
Internally studentized residuals are calculated by dividing the residual of the i-th observation by the standard error that includes all observations. The formula is:

$$r_i = \frac{\epsilon_i}{\sqrt{\sigma_i^{2} (1 - h_i)}}$$

Where:
- `ε_i`: Residual for the i-th observation.
- `σ`: Residual standard error using all observations.
- `h_i`: Leverage for the i-th observation (from the hat matrix).

#### Why Include All Data Points in the Variance?
- The standard error includes all observations, which means the numerator and denominator are not fully independent.
- Internal studentized residuals assume that the influence of the i-th observation on the variance is minimal, though not completely independent.

#### Use and Practicality
- Commonly used for simplicity and computational efficiency.
- Slight dependence between the residual and the variance estimate.

---


### Externally Studentized Residuals (R-Studentized Residuals)

#### What are Externally Studentized Residuals?
Externally studentized residuals are calculated by excluding the $i$-th observation both from the numerator (residual) and the denominator (standard error). The formula is:

$$
t_i = \frac{\epsilon_i}{\sqrt{S_i^{2} (1 - h_i)}}
$$

Where:
- $\epsilon_i$: Residual of the the $i$-th observation.
- $\sigma^{(i)}$: Standard error of the residual excluding the $i$-th observation (leave-one-out variance).
- $h_i$: Leverage for the $i$-th observation.

#### Why Exclude the $i$-th Observation in Variance?
- Excluding the $i$-th observation from the variance calculation ensures that the numerator and denominator are independent.
- This breaks the circular dependence between the residual and its variance estimate, providing a more robust measure.

#### Variance Without $e_i$ (Used in External Scaling)
In the case of externally studentized residuals, we want to estimate the error variance without the influence of the $i$-th observation. We essentially recalculate the variance as if the $i$-th data point were removed from the model. This allows for a more accurate estimate of variance because it eliminates the bias introduced by the $i$-th point.

The formula for the externally studentized residual becomes:

$$
S_i^{2} = \frac{(n - p)}{(n - p - 1)} \left(MS_{\text{RES}} - \frac{e_i^2}{(1 - h_i)} \right)
$$

Where:
- $MS_{\text{RES}}$: Mean square error based on the entire dataset.
- $e_i^2$: Squared residual of the $i$-th observation.
- $h_i$: Leverage of the $i$-th observation (a measure of how influential that observation is in determining the model's fit).

The key point here is that we subtract $$\frac{e_i^2}{(1 - h_i)}$$ because this adjusts for the influence of the $i$-th point. The leverage term $h_i$ represents the influence of the $i$-th point on the fitted values of the model. The term $\frac{1}{(1 - h_i)}$ is essentially the proportion of the variance in the residuals that is due to all other points, excluding the $i$-th point.

#### Why Is $\frac{e_i^2}{(1 - h_i)}$ Included?

The reason we include $\frac{e_i^2}{(1 - h_i)}$ in the denominator of the variance estimate in the externally studentized residual is to account for the influence that the $i$-th observation has on the model fit and the error variance. Here’s why:

- **Leverage** ($h_i$) tells us how much the $i$-th observation is influencing the regression model. If $h_i$ is large, the $i$-th observation has a large impact on the model fit and residuals.
- The term $$\frac{e_i^2}{(1 - h_i)}$$ adjusts the contribution of the $i$-th residual by accounting for this leverage. If $h_i$ is large (indicating a highly influential point), the adjustment term makes the variance estimate more robust by reducing the weight of that point's residual in the overall variance calculation.

Without this adjustment, the regular variance would be biased because the $i$-th point would unduly affect the estimate of variance. In contrast, by excluding $e_i$ and adjusting for leverage, the externally studentized residual provides a more accurate estimate of the variance that would be expected without the $i$-th point.

#### Why Use Externally Studentized Residuals?
- **More robust** to individual influential points.
- **Better outlier detection**, as they account for the influence of the $i$-th observation in both the residual and the standard error.
- Follow a **t-distribution** with $n - k - 2$ degrees of freedom.
  
#### Use and Practicality
- Used for outlier detection and hypothesis testing (e.g., Bonferroni-adjusted t-tests).
- **More rigorous** than internal studentization because the $i$-th observation is excluded in both the residual and variance.

---

### Difference Between Internal and External Studentized Residuals

| Type                     | Variance Used                     | Formula                                                    | Dependence                |
| ------------------------ | --------------------------------- | ---------------------------------------------------------- | ------------------------- |
| **Internal Studentized** | Variance using all observations   | $$r_i = \frac{\epsilon_i}{\sqrt{\sigma_i^{2} (1 - h_i)}}$$ | Slight dependence exists. |
| **External Studentized** | Variance excluding the $i$-th obs | $$t_i = \frac{\epsilon_i}{\sqrt{\sigma_i^{2} (1 - h_i)}}$$ | Fully independent.        |

- **Internal studentized residuals** assume minimal influence from the $i$-th observation on the variance.
- **External studentized residuals** rigorously recompute the variance excluding the $i$-th observation.

#### Outlier Detection
- If $$|t_i| > t_{\alpha/2, n-k-2}$$, the $i$-th observation may be considered an outlier.

---



### Pressed Residuals: Explanation and Function

Pressed residuals are a statistical diagnostic tool used in regression analysis. They help identify the influence of individual data points on the regression model by measuring how much the predicted value of a specific observation changes when that observation is removed from the model.

#### Function of Pressed Residuals

Pressed residuals are particularly useful for:

1. **Detecting influential data points:** They highlight observations that disproportionately affect the model's fit.
2. **Model validation:** They assist in assessing the stability of the regression model by showing how predictions change under perturbation.

#### Formula for Pressed Residuals

The pressed residual for the $i$-th observation is defined as:

$$
r_i^{\text{press}} = \frac{e_i}{1 - h_{ii}}
$$

Where:

- $e_i = y_i - \hat{y}_i$ is the residual for the $i$-th observation.  
- $h_{ii}$ is the $i$-th diagonal element of the hat matrix $H$, which is given by:

$$
H = X(X^\top X)^{-1}X^\top
$$

Here:

- $X$ is the design matrix of predictors.
- $X^\top$ is the transpose of $X$.
- $(X^\top X)^{-1}$ is the inverse of the product $X^\top X$.

The denominator $1 - h_{ii}$ adjusts the residual to account for the leverage of the $i$-th observation, where leverage quantifies how far an observation's predictor values are from the mean of the predictor space.

#### Interpretation

- **Small pressed residuals** suggest the observation is consistent with the model.
- **Large pressed residuals** indicate that the observation may be an outlier or exert undue influence on the model.

#### Practical Notes

1. Pressed residuals are part of influence diagnostics and often used alongside other measures like Cook's distance and leverage.
2. High-leverage points ($h_{ii}$ close to 1) amplify the residual, making the pressed residual a sensitive indicator for influential data points.


### MODEL DIAGNOSIS (LEVERAGE AND INFLUENTIAL POINTS)

#### Leverage points

**Leverage points** are data points in a regression analysis that have an unusually large effect on the estimated regression line. These points are influential because their independent variable values (i.e., the $x$-values) are far from the mean of the other data points. Leverage points are not necessarily outliers in terms of the dependent variable (i.e., the $y$-values), but their position in the independent variable space gives them the potential to disproportionately affect the model's predictions and parameters.

##### Intuitive Explanation

Imagine you're fitting a line to a set of data points using linear regression. The regression line is determined by finding the best fit based on minimizing the sum of squared errors (the distance between the points and the line). The leverage of a point is essentially a measure of how much influence it has on the placement of the regression line.

- **High leverage points** are far away from the center of the other points on the $x$-axis.
- **Low leverage points** are closer to the center, i.e., they are near the mean of the $x$-values.

A point with high leverage can **pull the regression line** toward itself, even if its $y$-value isn’t particularly unusual. This is because its $x$-value is far from the others, and the line needs to adjust to fit this point.

##### Why Are Leverage Points Important?

Leverage points don’t always correspond to outliers in terms of the dependent variable. In the above example, the $y$-value of 10 is not an outlier (it’s within a reasonable range). However, because the $x$-value is so extreme, the model may give this point too much influence. This can lead to misleading conclusions or predictions.

##### Detecting Leverage Points

Leverage is typically quantified using a **hat matrix** in regression analysis, which calculates how much influence each data point has on the fitted values. Points with a **leverage value ($h_i$)** significantly greater than the average leverage are considered to have high leverage.

- The average leverage is $\frac{p}{n}$​, where $p$ is the number of predictors (including the intercept), and $n$ is the number of data points.
- If $hi​>2×n/p$ ​, point $i$ might be a leverage point.

#### Influential points

An **influential point** is an observation that not only has a high leverage (i.e., an extreme X-value), but also has a residual (the difference between the actual and predicted value) that is large. This means the point is far from the regression line and has a significant effect on the slope or fit of the model.

##### Impact of Influential Points

**Influential points** are data points that significantly affect the outcome of a statistical analysis, particularly in regression. These points can dramatically change the **slope** and **intercept** of a regression line, which can skew the analysis and lead to misleading results. An influential point may not necessarily be an outlier (it doesn’t have to be far from the other data points), but it has a large impact on the model’s parameters.

##### How Influential Points Affect the Model

- **Change in Regression Coefficients:** Removing an influential point may lead to significant changes in the regression coefficients (slope and intercept).
- **Bias in Predictions:** Influential points can distort predictions for both existing and new data, leading to biased results that don’t represent the true relationship between variables.
- **Overfitting:** In some cases, influential points can cause the model to overfit, capturing noise rather than the actual underlying trend.
##### Detecting Influential Points

To detect **influential points**, several methods can be used, such as **leverage, Cook's distance**, and **DFBETAs**.

###### 1. **Leverage**

- **Leverage** measures how much an individual data point can influence the model’s fitted values. High leverage points are far from the average of the other data points and can distort the regression model.
- Points with leverage above a certain threshold (usually $\frac{2k}{n}$​, where k is the number of predictors and n is the number of observations) may be influential.

###### 2. **Cook's Distance**

Cook’s distance measures the combined influence of both the residual and the leverage of a point. It tells us how much the estimated regression coefficients change when that point is removed. A large Cook's distance indicates that a point has a strong influence on the model.


###### 3. **DFBETAs**

DFBETAs measure how much the estimated coefficients change when a particular data point is removed. If the change in any coefficient is large, the point is influential.

#### VISUAL 
![[maxresdefault.jpg]]



### Cook's Distance

#### Intuitive Explanation

Cook's Distance is a measure used in regression analysis to detect influential data points that can have a large impact on the estimated coefficients. It helps identify data points that significantly affect the model's predictions when they are included in the dataset.

#### Intuitive Explanation:

Imagine you have a dataset, and you fit a regression model to it. Now, if you remove one data point at a time and refit the model, you might notice that the estimated regression line changes a lot for certain points, while for others, it doesn't change much. The points that cause a large change in the model are considered **influential**.

Cook's Distance combines both the **residual** (how far off the prediction is for a given point) and the **leverage** (how far a point is from the center of the data). If a point has high leverage and its removal causes a large change in the fitted regression line, it is considered influential.

#### Formula for Cook’s Distance

##### Formula 1: Conceptual Definition
$$D_i = \frac{\sum_{j=1}^n \left( \hat{Y}_j - \hat{Y}_{j(i)} \right)^2}{p \times \text{MSE}}$$
###### Parts of the Formula

1. **$\hat{Y}_j$**  
    The predicted value for the $j$-th observation when **all data points** are included in the model.
    
1. **$\hat{Y}_{j(i)}$**  
    The predicted value for the $j$-th observation when the $i$-th data point is **removed** from the model.
    
3. **$\left( \hat{Y}_j - \hat{Y}_{j(i)} \right)^2$**  
    The squared difference between the predictions with and without the $i$-th point.
    
4. **$\sum_{j=1}^n$**  
    Summation over all $n$ observations in the dataset.
    
5. **$p$**  
    The number of parameters (coefficients) in the regression model, including the intercept.
    
6. **$\text{MSE}$**  
    The Mean Squared Error of the regression model, scaling the numerator.
##### Formula 2:
$$D_i = \frac{r_i^2}{p} \cdot \frac{h_{ii}}{(1 - h_{ii})}$$

###### Parts of the Formula

1. **$e_i$**  
    The residual for the $i$-th observation, calculated as $e_i = y_i - \hat{y}_i$.
    
2. **$p$**  
    The number of parameters (coefficients) in the regression model.
    
3. **$h_{ii}$**  
    The leverage of the $i$-th observation, which is the $i$-th diagonal element of the hat matrix $H$.
    
4. **$\frac{h_{ii}}{(1 - h_{ii})^2}$**  
    A term that adjusts for the leverage of the $i$-th observation, amplifying the influence of high-leverage points.

#### Cutoff Rule

There is a commonly used rule of thumb for detecting influential points:
- If $D_i > 1$, the $i$-th observation may be influential.


### DFBETAS and DFFITS

#### DFBETAS
DFBETAS measures the **standardized change** in a regression coefficient when a specific observation $i$ is removed.
BFBETAS evaluates how much the regression coefficients (β\betaβ) change if we remove a specific observation (iii) from the dataset.

**Formula**:  
$$
DFBETAS_{j,i} = \frac{\hat{\beta}_j - \hat{\beta}_{j(i)}}{\sqrt{S^2_{(i)} C_{jj}}}
$$

  

##### Explanation of the Parts
- **$DFBETAS_{j,i}$**:
    
    - The **standardized change** in the $j$-th regression coefficient when observation $i$ is removed.
    - It measures the influence of observation $i$ on the coefficient $\hat{\beta}_j$.
- **$\hat{\beta}_j$**:
    
    - The regression coefficient for the $j$-th predictor using the **full dataset**.
- **$\hat{\beta}_{j(i)}$**:
    
    - The regression coefficient for the $j$-th predictor when the $i$-th observation is **excluded**.
- **$S^2_{(i)}$**:
    
    - The residual mean square error when observation $i$ is removed.
- **$C_{jj}$**:
    
    - The $j$-th diagonal element of the covariance matrix of the predictors.

**Threshold for Influence**:  
$|\text{DFBETAS}_{j(i)}| > \frac{2}{\sqrt{n}}$  

**Use Case**:  
Identifies influential points affecting specific regression coefficients.
BFBETAS tells you about an observation's impact on the _global structure of the model_ (coefficients).

---

#### DFFITS
DFFITS measures the **influence** of a specific observation on the predicted values.
DFFITS measures how much the predicted value of YYY changes for a specific observation iii if iii is removed from the dataset.

Formula:

$$
DFFITS_i = \frac{\hat{y}_i - \hat{y}_{i(i)}}{\sqrt{S^2_{(i)} h_{ii}}}
$$

 **$\hat{y}_i$**:
    This is the **predicted value** for observation $i$ using the **full dataset** (all observations).
 **$\hat{y}_{i(i)}$**:
    This is the **predicted value** for observation $i$ when the $i$-th observation is **excluded** from the model.
    Removing the observation helps assess its influence on the model's prediction.
 **$S^2_{(i)}$**:
    This is the **mean squared error (MSE)** of the model after removing the $i$-th observation.
    It represents the variance of the residuals without the $i$-th data point.
**$h_{ii}$**:
    This is the **leverage value** for observation $i$, which comes from the **hat matrix** $H$.
    It measures how far the observation $i$ is from the center of the predictor space.
    Higher leverage means the observation has more influence on its own predicted value.
    
 **$\sqrt{S^2_{(i)} h_{ii}}$**:
    This is the **standard error** of the predicted value $\hat{y}_i$, accounting for the variance of the residuals and the leverage of the $i$-th observation.

**Threshold for Influence**:  
$|\text{DFFITS}_i| > 2 \sqrt{\frac{p}{n}}$  

**Use Case**:  
Identifies observations that disproportionately influence the predicted values.
DFFITS tells you about an observation's impact on its _local prediction_.

### Model Validation: AIC, BIC, and Mallow’s $C_p$ Criterion

Model validation is a crucial step in regression analysis to assess how well a model fits the data while also considering the complexity of the model. Overfitting and underfitting are common issues in statistical modeling, and these criteria help balance the trade-off between the goodness of fit and model complexity.
#### Akaike Information Criterion (AIC)
AIC balances the goodness of fit and model complexity. It penalizes models with more parameters.

**Formula**:  
$AIC = 2k + 2\ln(L)$  

Where:
- $k$ is the number of parameters in the model.
- $L$ is the likelihood of the model.

#### Bayesian Information Criterion (BIC)
BIC is similar to AIC but penalizes complexity more strongly, especially with larger sample sizes.

**Formula**:  
$BIC = \ln(n)k + 2\ln(L)$  

Where:
- $n$ is the number of observations.
- $k$ is the number of parameters.
- $L$ is the likelihood.

#### Mallow's $C_p$ Criterion

Mallow's $C_p$ is a criterion used to select the best subset of predictors for regression models. It balances model fit and complexity.

The Cp criterion is defined as:

$$
C_p = \frac{SS_{\text{Res},p}}{\hat{\sigma}^2} - n + 2p
$$

Where:
- $SS_{\text{Res},p}$ is the residual sum of squares using $p$ predictors.
- $\hat{\sigma}^2$ is the estimated variance of the error term (mean squared error).
- $n$ is the number of observations.
- $p$ is the number of predictors in the model.

#### Interpretation:
- **Cp ≈ p**: The model is well-fitted and not overfitting.
- **Cp < p**: The model is underfitting (too few predictors).
- **Cp > p**: The model is overfitting (too many predictors).



