# Linear Regression Formulas

## 1. Estimation of $b_0$ and $b_1$

### **Definitions of $S_{xx}$, $S_{xy}$, and $S_{yy}$**

$$
S_{xx} = \sum (x_i - \bar{x})^2
$$

$$
S_{xy} = \sum (x_i - \bar{x})(y_i - \bar{y})
$$

$$
S_{yy} = \sum (y_i - \bar{y})^2
$$

### **Slope Estimator** $b_1$:

$$
b_1 = \frac{S_{xy}}{S_{xx}}
$$

### **Intercept Estimator** $b_0$:

$$
b_0 = \bar{y} - b_1 \bar{x}
$$

---


## 2. Formula for $\sigma^2$ (Variance of Residuals)

$$
\sigma^2 = \frac{SSR}{n - k - 1} = MSRES
$$

where:

- $SSR$ = Residual Sum of Squares  
- $n$ = Number of observations  
- $k$ = Number of predictors  
---
## 3. Standard Error (SE)

### **Standard Error of $b_1$**:

$$
SE(b_1) = \sqrt{\frac{MSRES}{S_{xx}}}
$$

### **Standard Error of $b_0$**:

$$
SE(b_0) = \sqrt{MSRES \cdot \left[ \frac{1}{n} + \frac{\bar{x}^2}{S_{xx}} \right]}
$$


---

## 4. Checking If $b_0$ and $b_1$ Are Constants

To test if the coefficients are significantly different from zero:

### **Test Statistic for $b_1$**:

$$
t = \frac{b_1 - k}{SE(b_1)}
$$

### **Test Statistic for $b_0$**:

$$
t = \frac{b_0 - k}{SE(b_0)}
$$

If $|t|$ is greater than the critical value from the **t-distribution**, we reject the null hypothesis.


---

## 5. **Significance of Regression**

To test if the regression is statistically significant:

$$
t = \frac{b_1}{SE(b_1)}
$$

This follows a **t-distribution** with $n - k - 1$ degrees of freedom.

---

## 6. **Confidence Intervals for $b_0$ and $b_1$**

For a **95% confidence interval (CI)**:

$$
b_1 \pm t_{\alpha/2, n-k-1} \cdot SE(b_1)
$$

$$
b_0 \pm t_{\alpha/2, n-k-1} \cdot SE(b_0)
$$

where $t_{\alpha/2, n-k-1}$ is the critical value from the **t-distribution**.

---

## 7. **Sum of Squares Formulas**

### **Total Sum of Squares (SST):**  

$$
SST = S_{yy}
$$

### **Regression Sum of Squares (SSREG):**  

$$
SSREG = b_1 S_{xy}
$$

### **Residual Sum of Squares (SSRES):**  

$$
SSRES = SST - SSREG
$$

---

## 8. **$R^2$ and Adjusted $R^2$**

### **Coefficient of Determination $R^2$**  

$$
R^2 = 1 - \frac{SSRES}{SST}
$$

or equivalently,

$$
R^2 = \frac{SSREG}{SST}
$$

### **Adjusted $R^2$ (Correcting for Number of Predictors)**  

$$
R^2_{\text{adj}} = 1 - \left( \frac{(1 - R^2)(n - 1)}{n - k - 1} \right)
$$

---

## 9. **ANOVA Table and F-Test for Significance of Regression**

| **Source** | **Sum of Squares (SS)** | **Degrees of Freedom (df)** | **Mean Square (MS)**          | **F-Statistic**           |
| ---------- | ----------------------- | --------------------------- | ----------------------------- | ------------------------- |
| Regression | $SSREG$                 | $k$                         | $MSREG = \frac{SSREG}{k}$     | $F = \frac{MSREG}{MSRES}$ |
| Residual   | $SSRES$                 | $n - k - 1$                 | $MSRES = \frac{SSRES}{n-k-1}$ |                           |
| Total      | $SST$                   | $n - 1$                     |                               |                           |

### **Using the F-Statistic to Test Significance**

$$
F = \frac{MSREG}{MSRES}
$$

#### **When to Reject the Null Hypothesis ($H_0$):**
- **Null Hypothesis ($H_0$)**: The regression model is **not significant** (i.e., the predictors do not explain variation in $y$).
- **Alternative Hypothesis ($H_A$)**: The regression model is **significant**.
- **Decision Rule**:  
  - If $F > F_{\alpha, k, n-k-1}$ (from the F-distribution table), **reject $H_0$**.
  - If $F \leq F_{\alpha, k, n-k-1}$, **do not reject $H_0$**.

---

## 10. **Summary Table**

| **Formula**              | **Expression**                                                             |
| ------------------------ | -------------------------------------------------------------------------- |
| $b_1$ (Slope)            | $b_1 = \frac{S_{xy}}{S_{xx}}$                                              |
| $b_0$ (Intercept)        | $b_0 = \bar{y} - b_1 \bar{x}$                                              |
| $SE(b_1)$                | $\sqrt{\frac{MSRES}{S_{xx}}}$                                              |
| $SE(b_0)$                | $\sqrt{MSRES \cdot \left[ \frac{1}{n} + \frac{\bar{x}^2}{S_{xx}} \right]}$ |
| $MSRES$ (Error Variance) | $\frac{SSR}{n - k - 1}$                                                    |
| $\text{Var}(b_1)$        | $\frac{MSRES}{S_{xx}}$                                                     |
| $\text{Var}(b_0)$        | $MSRES \cdot \left[ \frac{1}{n} + \frac{\bar{x}^2}{S_{xx}} \right]$        |
| $SSRES$                  | $S_{yy} - b_1 S_{xy}$                                                      |
| $SSREG$                  | $b_1 S_{xy}$                                                               |
| $SST$                    | $S_{yy}$                                                                   |
| $R^2$                    | $1 - \frac{SSRES}{SST}$                                                    |
| Adjusted $R^2$           | $1 - \left( \frac{(1 - R^2)(n - 1)}{n - k - 1} \right)$                    |
| $F$-Statistic            | $F = \frac{MSREG}{MSRES}$                                                  |

---

