# Logistic Regression Model

- The **success probability** is denoted as $\pi(x)$.
- The **logit of success probability** (log-odds) is defined as:  $\ln\left(\frac{\pi(x)}{1-\pi(x)}\right)$

### Linear Form of Logit

The logit is modeled linearly as:

$$\ln\left(\frac{\pi(x)}{1-\pi(x)}\right) = \alpha + \beta x$$
This converts the logistic curve to a linear regression like line.

---

### Derivation of Logistic Function

From the logit form:

$$\ln\left(\frac{\pi(x)}{1-\pi(x)}\right) = \alpha + \beta x$$

Exponentiating both sides:

$$\frac{\pi(x)}{1-\pi(x)} = e^{\alpha + \beta x}$$

Rewriting:

$$\pi(x) = \frac{e^{\alpha + \beta x}}{1 + e^{\alpha + \beta x}}$$

This is the **logistic function**, which represents an S-shaped curve.

---

### Logistic Function Properties

- **Formula**: $$\pi(x) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$
- **Interpretation of Coefficient (β\betaβ)**:
    - $β>0$: Curve ascends as $x$ increases.
    - $β<0$: Curve descends as $x$ increases.
    - $∣β∣$: A larger absolute value of $\beta$ increases the steepness of the curve.
    - $β=0$: The curve flattens.


---

### Linearization via Logit Transformation

The logistic response function:

$$\pi(x) = \frac{1}{1 + e^{-(\alpha + \beta x)}}$$

can be linearized using the **logit transformation**:

$$\ln\left(\frac{\pi(x)}{1-\pi(x)}\right) = \alpha + \beta x$$

---

### Odds and Odds Ratio

The odds are given by:

$$\text{Odds} = \frac{\pi(x)}{1-\pi(x)} = e^{\alpha + \beta x}$$

#### Odds Ratio:

- For every 1-unit increase(logit increase) in $x$, the odds multiply by $e^{\beta}$.

### Interpretation of Coefficients

1. In the **logit scale**:
    - $\beta$ represents the change in log-odds for a one-unit increase in $x$.
2. In the **odds scale**:
    - $e^{\beta}$ indicates how much the odds multiply for a one-unit increase in $x$.

---

### Estimation for $x$:
 
 $x = 21.0$
The logistic regression model is typically given by the equation:

$$
\hat{y} = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}
$$

Where:
- $\beta_0 = -12.3508$ is the intercept,
- $\beta_1 = 0.4972$ is the slope,
- $x = 21.0$ is the input.

The estimated value for $x = 21.0$ is:

$$
\hat{y} = \frac{1}{1 + e^{-(-12.3508 + 0.4972 \cdot 21.0)}}
$$

Now, substitute the values:

$$
\hat{y} = \frac{1}{1 + e^{-(-12.3508 + 10.4332)}} = \frac{1}{1 + e^{-1.9176}} = \frac{1}{1 + 0.1462} = \frac{1}{1.1462} \approx 0.873
$$

Thus, the estimated value for $x = 21.0$ is approximately $0.873$.

### Median effective level:

The median effective level is the value of $x$ where the predicted probability $\hat{y}$ is 0.5. This is achieved when the argument of the logistic function is 0, which gives:

$$
\beta_0 + \beta_1 \cdot x = 0
$$

Solving for $x$:

$$
x = -\frac{\beta_0}{\beta_1}
$$

Substitute the values of $\beta_0$ and $\beta_1$:

$$
x = -\frac{-12.3508}{0.4972} \approx 24.84
$$

Thus, the median effective level is approximately $x = 24.84$.

### Slope at a point:

The slope of the logistic regression model at a given point is the derivative of the logistic function with respect to $x$:

$$
\frac{d\hat{y}}{dx} = \hat{y} \cdot (1 - \hat{y}) \cdot \beta_1
$$

At $x = 26.3$, first, calculate $\hat{y}$ for this value:

$$
\hat{y} = \frac{1}{1 + e^{-(-12.3508 + 0.4972 \cdot 26.3)}} = \frac{1}{1 + e^{-(-12.3508 + 13.0654)}} = \frac{1}{1 + e^{0.7146}} = \frac{1}{1 + 2.045} = \frac{1}{3.045} \approx 0.328
$$

Then, substitute $\hat{y} \approx 0.328$ into the derivative formula:

$$
\frac{d\hat{y}}{dx} = 0.328 \cdot (1 - 0.328) \cdot 0.4972 = 0.328 \cdot 0.672 \cdot 0.4972 \approx 0.111
$$

Thus, the slope at $x = 26.3$ is approximately $0.111$.

### Estimated odds increase:

The odds increase at any point is given by the factor $e^{\beta_1}$.

$$
\text{Odds Increase} = e^{\beta_1}
$$

Substitute the value of $\beta_1$:

$$
\text{Odds Increase} = e^{0.4972} \approx 1.644
$$

Thus, the estimated odds increase is approximately 64% for every increment in $x$ .


### Confidence Interval (CI) for $\beta$:

The Confidence Interval for $\beta$ is calculated using the formula:

$$
CI = B \pm Z_{\alpha/2} \cdot SE(B)
$$

Where:
- $B = 0.497$ is the estimated coefficient, (given)
- $SE(B) = 0.1017$ is the standard error of the coefficient, (given)
- $Z_{\alpha/2}$ is the critical value of the standard normal distribution at the desired confidence level.

For a 95% confidence level, $Z_{\alpha/2} \approx 1.96$.

Now, substitute the values:

$$
CI = 0.497 \pm 1.96 \cdot 0.1017
$$

Performing the calculation:

$$
CI = 0.497 \pm 0.1993
$$

Thus, the confidence interval is:

$$
CI = (0.2977, 0.6963)
$$

So, the 95% Confidence Interval for $\beta$ is approximately $(0.2977, 0.6963)$.


### Significance Testing (using Z-statistic):

For significance testing using the Z-statistic, we calculate it as:

$$
Z = \frac{B}{SE(B)}
$$

Where:
- $B = 0.497$ is the estimated coefficient,(given)
- $SE(B) = 0.1017$ is the standard error of the coefficient.(given)

Now, substitute the values:

$$
Z = \frac{0.497}{0.1017} \approx 4.89
$$

For a two-tailed test at the 95% confidence level, the critical value of the Z-statistic is $1.96$.

Since $Z \approx 4.89$ is much larger than the critical value of $1.96$, we reject the null hypothesis and conclude that $\beta$ is statistically significant at the 95% confidence level.

In summary:
- The Z-statistic is approximately $4.89$, which is greater than the critical value of $1.96$.
- Therefore, we reject the null hypothesis that $\beta = 0$ and conclude that $\beta$ is statistically significant at the 95% confidence level.


## Multiple Logistic Regression

Multiple logistic regression is a type of regression analysis used when the dependent variable (outcome) is binary (e.g., success/failure, yes/no). In this model, we use two or more independent variables (predictors) to predict the likelihood of an event occurring. The formula for multiple logistic regression is:

$$
\text{logit}(P) = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_n x_n
$$

Where:
- $P$ is the probability of the outcome (e.g., success),
- $\beta_0$ is the intercept,
- $\beta_1, \beta_2, \dots, \beta_n$ are the coefficients of the predictors $x_1, x_2, \dots, x_n$,
- $\text{logit}(P)$ is the log-odds of the probability, calculated as $\log \left( \frac{P}{1-P} \right)$.

To convert the log-odds back to a probability, we use the logistic function:

$$
P = \frac{1}{1 + e^{-\text{logit}(P)}}
$$

---

### Example 1: Logistic Regression for Study Hours vs. Success

#### Given:
- $x$: 29, 15, 33, 28, 39 (hours of study)
- $y$: 0, 0, 1, 1, 1 (success/failure)
- Intercept = -64, $B = 2$ (coefficient)

#### 1. Probability of pass for the student who studied 33 hours:

For logistic regression, we calculate the log-odds first:

$$
\text{logit}(P) = \beta_0 + \beta_1 x
$$

Substitute the values:

$$
\text{logit}(P) = -64 + 2 \cdot 33 = -64 + 66 = 2
$$

Now, we convert the log-odds to probability using the logistic function:

$$
P = \frac{1}{1 + e^{-2}} = \frac{1}{1 + 0.1353} = \frac{1}{1.1353} \approx 0.8808
$$

Thus, the probability of passing for the student who studied 33 hours is approximately $0.8808$ or $88.08\%$.

#### 2. How many hours should a student study for his success rate to be more than 95%?

We need to find the value of $x$ such that $P > 0.95$. First, set up the equation for probability:

$$
0.95 = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}
$$

Solving for the log-odds:

$$
0.95 = \frac{1}{1 + e^{-( -64 + 2x )}}
$$

Multiply both sides by $1 + e^{-( -64 + 2x )}$:

$$
0.95 \cdot (1 + e^{-( -64 + 2x )}) = 1
$$

Simplify:

$$
1 + e^{-( -64 + 2x )} = \frac{1}{0.95} \approx 1.0526
$$

Subtract 1 from both sides:

$$
e^{-( -64 + 2x )} \approx 0.0526
$$

Now, take the natural logarithm of both sides:

$$
-( -64 + 2x ) = \ln(0.0526) \approx -2.944
$$

Solve for $x$:

$$
64 - 2x = 2.944
$$

$$
2x = 64 - 2.944 = 61.056
$$

$$
x = \frac{61.056}{2} = 30.528
$$

Thus, the student should study approximately $30.53$ hours to have a success rate of more than 95%.

---

### Example 2: Logistic Regression for Customer Subscription Prediction

#### Given:
- $x_1$: 87, 62, 58, 75, 52 (feature 1)
- $x_2$: 2, 3, 2, 4, 3 (feature 2)
- Intercept = -5.706, $B_1 = 0.008$, $B_2 = 0.1002$

#### Classifying customer subscription:

For logistic regression, we calculate the log-odds as:

$$
\text{logit}(P) = \beta_0 + B_1 x_1 + B_2 x_2
$$

Substitute the values for each customer to classify them:

For the first customer, where $x_1 = 87$ and $x_2 = 2$:

$$
\text{logit}(P) = -5.706 + 0.008 \cdot 87 + 0.1002 \cdot 2
$$

$$
\text{logit}(P) = -5.706 + 0.696 + 0.2004 = -4.8096
$$

Convert to probability:

$$
P = \frac{1}{1 + e^{4.8096}} \approx 0.008
$$

This probability is very low, so the customer is unlikely to subscribe to the new scheme.

You can repeat similar steps for other customers to classify them based on their predicted probabilities.



### Summarizing Predictive Power

To evaluate a classification model’s performance, we often use a **classification table** (also known as a confusion matrix). It compares the model's predictions with the actual values, helping us understand how well the model distinguishes between classes.

The **classification table** typically includes the following elements:
- **True Positives (TP)**: Correctly predicted positive cases.
- **False Positives (FP)**: Incorrectly predicted positive cases.
- **True Negatives (TN)**: Correctly predicted negative cases.
- **False Negatives (FN)**: Incorrectly predicted negative cases.

From this table, we can compute key performance metrics such as **sensitivity** and **specificity**.

---

### Choosing the $p_0$ Value (Cutoff) and Classification Table

The cutoff (threshold) for classification determines how the predicted probabilities are mapped to binary outcomes (e.g., positive or negative). For example, if the model predicts a probability $p$ for a positive outcome, we compare it with a cutoff value, $p_0$. If $p \geq p_0$, the prediction is classified as positive ($y = 1$); otherwise, it's classified as negative ($y = 0$).

In many cases, the cutoff is set at $p_0 = 0.5$, meaning if the predicted probability is greater than or equal to 50%, the model classifies the outcome as positive. However, adjusting this cutoff can affect the balance between sensitivity and specificity, depending on the application.

---

### Example

Given:
- Out of 62 cases with $y = 0$ (negative class), the model predicts $y^ = 0$ for 43.
- Out of 111 cases with $y = 1$ (positive class), the model predicts $y^ = 1$ for 75.

We can construct the classification table:

|                    | Predicted $y = 0$ | Predicted $y = 1$ |
| ------------------ | ----------------- | ----------------- |
| **Actual $y = 0$** | 43 (TN)           | 19 (FP)           |
| **Actual $y = 1$** | 36 (FN)           | 75 (TP)           |

Where:
- **True Negatives (TN)** = 43
- **False Positives (FP)** = 19
- **False Negatives (FN)** = 36
- **True Positives (TP)** = 75

#### Calculating Sensitivity and Specificity

- **Sensitivity (True Positive Rate)** is the proportion of actual positives correctly identified:

$$
\text{Sensitivity} = \frac{\text{TP}}{\text{TP} + \text{FN}} = \frac{75}{75 + 36} = \frac{75}{111} \approx 0.6757
$$

Thus, the sensitivity is approximately $0.6757$ or $67.57\%$.

- **Specificity (True Negative Rate)** is the proportion of actual negatives correctly identified:

$$
\text{Specificity} = \frac{\text{TN}}{\text{TN} + \text{FP}} = \frac{43}{43 + 19} = \frac{43}{62} \approx 0.6935
$$

Thus, the specificity is approximately $0.6935$ or $69.35\%$.

---

### Conclusion

- The **classification table** helps visualize the performance of the model.
- The **cutoff value** ($p_0$) determines how predictions are classified into positive and negative outcomes.
- The **sensitivity** and **specificity** are critical metrics for assessing the model’s predictive power:
  - **Sensitivity**: The proportion of true positives correctly predicted.
  - **Specificity**: The proportion of true negatives correctly predicted.
#### Calculating Accuracy 
The **accuracy** is the proportion of total correct predictions (both true positives and true negatives) out of all predictions: 
$$ \text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}} = \frac{75 + 43}{75 + 43 + 19 + 36} = \frac{118}{173} \approx 0.6815 $$ Thus, the accuracy is approximately $0.6815$ or $68.15\%$. 

### Conclusion - 
The **classification table** helps visualize the performance of the model. - The **cutoff value** ($p_0$) determines how predictions are classified into positive and negative outcomes. - The **sensitivity**, **specificity**, and **accuracy** are critical metrics for assessing the model’s predictive power: - **Sensitivity**: The proportion of true positives correctly predicted. - **Specificity**: The proportion of true negatives correctly predicted. - **Accuracy**: The proportion of correct predictions (both true positives and true negatives) out of all predictions.


## ROC Curve (Receiver Operating Characteristic Curve)

The **ROC curve** is a graphical representation of a classification model's performance. It helps evaluate how well the model discriminates between the positive and negative classes across different threshold values. The ROC curve plots the **True Positive Rate (Sensitivity)** against the **False Positive Rate (1 - Specificity)** at various thresholds.

#### Components of the ROC Curve:

- **True Positive Rate (TPR)** or **Sensitivity** is the proportion of actual positives correctly identified:

  $$
  \text{Sensitivity} = \frac{\text{TP}}{\text{TP} + \text{FN}}
  $$

- **False Positive Rate (FPR)** is the proportion of actual negatives incorrectly identified as positives:

  $$
  \text{FPR} = \frac{\text{FP}}{\text{FP} + \text{TN}}
  $$

![[Pasted image 20250112202323.png]]
#### The Role of the ROC Curve:

The ROC curve allows us to:
1. **Assess model performance**: The curve provides a visual representation of the trade-off between sensitivity and specificity for different cutoffs.
2. **Compare models**: A model with a higher area under the ROC curve (AUC) is considered better at distinguishing between the classes.
3. **Choose optimal cutoff**: By analyzing the ROC curve, you can choose a cutoff that balances sensitivity and specificity according to the specific needs of the problem.

The **Area Under the Curve (AUC)** is a common metric used to summarize the ROC curve. A higher AUC value (closer to 1) indicates a better model.

---

### Effect of $p_0$ (Cutoff) Values on Sensitivity and Specificity

The value of $p_0$, or the **cutoff**, plays a crucial role in the classification process. It is the threshold probability above which the model predicts the positive class ($y = 1$), and below which it predicts the negative class ($y = 0$).

- **When $p_0 = 0$**:
  - Every instance is classified as positive ($y = 1$), regardless of the predicted probability. 
  - **True Positives (TP)** will be maximized because every positive instance is correctly classified as positive. However, **False Positives (FP)** will also be maximized because every negative instance will be incorrectly classified as positive.
  - This results in:
    - **Sensitivity** = 1 (maximum) because all true positives are correctly classified.
    - **Specificity** = 0 because all true negatives are incorrectly classified as positives.
  
  In this case, you have **high sensitivity** but **low specificity**.

- **When $p_0 = 1$**:
  - Every instance is classified as negative ($y = 0$), regardless of the predicted probability. 
  - **True Negatives (TN)** will be maximized because all negative instances are correctly classified as negative. However, **False Negatives (FN)** will also be maximized because every positive instance will be incorrectly classified as negative.
  - This results in:
    - **Sensitivity** = 0 because no positive instances are correctly classified.
    - **Specificity** = 1 (maximum) because all true negatives are correctly classified.
  
  In this case, you have **high specificity** but **low sensitivity**.

- **When $p_0 = 0.5$** (typical threshold):
  - The model balances sensitivity and specificity, classifying instances based on whether the predicted probability is greater than or equal to 0.5.
  - The **ROC curve** allows for visualizing how the trade-offs change as $p_0$ is varied, and **optimal cutoff** can be selected based on specific needs (e.g., preferring higher sensitivity or specificity).

#### Summary of the Effect of $p_0$:

- **At $p_0 = 0$**, you maximize sensitivity (i.e., correctly identifying all positives) but sacrifice specificity (i.e., misclassifying many negatives as positives).
- **At $p_0 = 1$**, you maximize specificity (i.e., correctly identifying all negatives) but sacrifice sensitivity (i.e., misclassifying many positives as negatives).
- **At intermediate cutoffs** (e.g., $p_0 = 0.5$), there is a balance between sensitivity and specificity, which can be optimized based on the specific problem at hand.

By analyzing the **ROC curve**, you can select the optimal **cutoff** that balances these two metrics according to your specific needs, depending on the cost of false positives and false negatives.

--- 

### Conclusion

- The **ROC curve** helps assess and compare classification models by plotting sensitivity against the false positive rate at various cutoffs.
- The **cutoff value** ($p_0$) has a significant impact on sensitivity and specificity:
  - Lower cutoffs (closer to 0) increase sensitivity but decrease specificity.
  - Higher cutoffs (closer to 1) increase specificity but decrease sensitivity.
  - The **AUC** is a useful metric for evaluating overall model performance.
