# Unit 4: Non-Stationary Time Series and ARIMA Models

  

---

  

## 1. Introduction: The Problem of Non-Stationarity

  

### Core Concept

Most real-world time series (stock prices, GDP, sales figures) are **non-stationary**. This means they do not have:

- A constant mean ($\mu$)

- A constant variance ($\sigma^2$)

  

### Why This Matters

ARMA models (AutoRegressive Moving Average) **fail on non-stationary data**. You cannot fit them directly.

  

### The Solution

Transform the data to make it stationary. The primary method in the Box-Jenkins approach is **differencing**.

  

**Intuition:** If a stock price is wandering wildly (non-stationary), the *change* in price from day to day might be stable (stationary).

  

Once differenced to stationarity, we can fit an AR, MA, or ARMA model to the differenced series.

  

---

  

## 2. The ARIMA Model: Integrating Differencing

  

### Definition

A process $\{X_t\}$ is an **ARIMA(p, d, q)** process if, after differencing it $d$ times, it becomes a stationary ARMA(p, q) process.

  

### The Formula

$$\phi(B)(1-B)^d X_t = \theta(B)a_t$$

  

### Breaking Down the Math

  

| Symbol      | Meaning                                        |
| ----------- | ---------------------------------------------- |
| $\phi(B)$   | The AR polynomial (past predicting the future) |
| $(1-B)^d$   | The differencing operator                      |
| $B$         | Backshift operator: $BX_t = X_{t-1}$           |
| $\theta(B)$ | The MA polynomial (error terms)                |
| $a_t$       | White noise (pure randomness)                  |
  

**If $d=1$:** $(1-B)X_t = X_t - X_{t-1}$ — this is the "first difference."

  

### The "Unit Root" Concept

An ARIMA model is called **"unit-root non-stationary"**.

  

If the AR polynomial has a root of 1 (a unit root), the series does not return to a mean—it wanders. We "cancel out" this unit root by differencing.

  

---

  

## 3. The Random Walk: The Simplest Non-Stationary Series

  

The Random Walk (RW) is the classic example of non-stationary behavior. It is technically an **ARIMA(0,1,0)** model.

  

### The Formula

$$S_t = S_{t-1} + X_t$$

  

Using differencing notation:

$$\nabla S_t = S_t - S_{t-1} = X_t$$

  

### Why Is It Non-Stationary?

- **Variance grows with time:** $Var(S_t) = t\sigma^2$. As $t$ increases, variance gets larger.

- **Memory:** The current value $S_t$ is the sum (integration) of all past shocks.

  

---

  

## 4. How to Spot Non-Stationarity

  

### A. Visual Inspection (Time Series Plots)

Look for:

- **Trends** (mean is changing)

- **Changing spread** (variance is changing)

  

**Note:** Strong cycles (like animal populations) might look non-stationary, but if they are stable cycles, they might actually be stationary.

  

### B. The ACF Plot (The "Slow Decay" Signature)

This is the most reliable visual test.

  

| Series Type    | ACF Behavior                                                          |
| -------------- | --------------------------------------------------------------------- |
| Stationary     | Drops to zero quickly (exponential decay)                             |
| Non-Stationary | Decreases very slowly (linear decay), stays significant for many lags |

  

**Action:** If you see an ACF that looks like a slow slide down, you likely need to difference the data ($d=1$).

  

### C. Statistical Tests

Use the **Dickey-Fuller Test** to be mathematically sure.

  

---

  

## 5. The Dickey-Fuller (DF) Unit Root Test

  

### The Origin of the Test Equation

  

**Question:** What is $X_t = \phi X_{t-1} + a_t$ and how do we get it?

  

We select the simplest possible autoregressive model, the **AR(1) process**, as our "test dummy":

$$X_t = \phi X_{t-1} + a_t$$

  

We assume the data behaves like an AR(1) where the current value ($X_t$) depends on the immediate past value ($X_{t-1}$) plus random shock ($a_t$).

  

By using this equation, we focus entirely on the coefficient $\phi$:

- **If $\phi = 1$:** The equation becomes $X_t = X_{t-1} + a_t$ (Random Walk). Past value preserved forever, shocks accumulate → **Non-Stationary**

- **If $|\phi| < 1$:** Effect of past fades over time. Series pulls back to a mean → **Stationary**

  

### The Hypothesis

  

| Hypothesis          | Statement  | Meaning                           |
| ------------------- | ---------- | --------------------------------- |
| $H_0$ (Null)        | $\phi = 1$ | Unit Root exists → Non-Stationary |
| $H_1$ (Alternative) | $\phi < 1$ | Stationary                        |

  

### The Test Statistic

  

We estimate $\phi$ using Least Squares. **Critical:** We cannot use standard t-tables. We use **Dickey-Fuller tables**.

  

$$\hat{\tau} = \frac{\hat{\phi} - 1}{\text{SE}(\hat{\phi})}$$

  

### Interpretation

- **$\hat{\tau}$ strongly negative** (more negative than critical value) → Reject $H_0$ → Series is **Stationary**

- **$\hat{\tau}$ positive or close to zero** → Fail to reject $H_0$ → Series is **Non-Stationary** (needs differencing)

  

### Numerical Example

  

Given $X = [10, 12, 15, 18, 22, 27]$ with $X_0 = 0$:

  

| Parameter               | Value   |
| ----------------------- | ------- |
| $\hat{\phi}$            | 1.2216  |
| $\hat{\sigma}_a$        | 4.478   |
| $\text{SE}(\hat{\phi})$ | 0.1253  |
| $\hat{\tau}$            | +1.7683 |
  

**Conclusion:** $\tau$ is positive → Fail to reject unit root → **Non-stationary**

  

### When Simple DF Test Is Valid

- No intercept

- No trend

- No autocorrelation in errors

- Pure AR(1) process

  

### When to Use ADF (Augmented Dickey-Fuller)

- Data has drift or trend

- Residuals show autocorrelation

- Real-world modeling

  

**ADF Model:**

$$\Delta X_t = \gamma X_{t-1} + \sum \psi_i \Delta X_{t-i} + \varepsilon_t$$

  
| Test | Use Case                    |
| ---- | --------------------------- |
| DF   | Theoretical, AR(1) only     |
| ADF  | Practical, used in software |

  

---

  

## 6. General Form of ARIMA and Constants

  

### The General Formula

$$\phi(B)\nabla^d X_t = \delta + \theta(B)a_t$$

  

### Decoding the Notation

  

**Question:** Is $\nabla^d$ basically $(1-B)$ from the original formula?

  

**Yes, exactly.** The symbol $\nabla$ (nabla) is shorthand for the difference operator $(1-B)$.

  

| Notation       | Equivalent    | Meaning                           |
| -------------- | ------------- | --------------------------------- |
| $\nabla^1 X_t$ | $(1-B)X_t$    | First Difference: $X_t - X_{t-1}$ |
| $\nabla^2 X_t$ | $(1-B)^2 X_t$ | Second Difference                 |
| $\nabla^d$     | $(1-B)^d$     | d-th order difference             |
 

### The Confusion: $\delta$ (Constant) vs. $d$ (Differencing)

  

**Question:** Why is there $\delta$ in the general form if ARIMA is for non-stationary data?

  

These are **two completely different things**:

  

| Symbol                 | Type            | Role                                                           |
| ---------------------- | --------------- | -------------------------------------------------------------- |
| $d$ (lowercase)        | Integer         | Order of differencing. If $d \ge 1$, series is non-stationary. |
| $\delta$ (Greek delta) | Constant number | Like an intercept in regression                                |

  

**Why is $\delta$ there?**

  

$\delta$ does NOT determine stationarity—$d$ does that. $\delta$ determines the "slope" or "level":

  

| Condition                | Effect of $\delta$                                                                          |
| ------------------------ | ------------------------------------------------------------------------------------------- |
| $d = 0$ (Stationary)     | Allows series to have non-zero mean (oscillate around 100 instead of 0)                     |
| $d = 1$ (Non-Stationary) | Creates **Deterministic Trend** (steady upward/downward drift by amount $\delta$ each step) |

  

---

  

## 7. Making a Time Series Stationary: Methods

  

### Common Transformations

  

| Method             | Purpose                     | Use When                               |
| ------------------ | --------------------------- | -------------------------------------- |
| Log transformation | Stabilizes variance         | Variance increases with level          |
| Differencing       | Removes stochastic trend    | Random walk behavior                   |
| Log-differencing   | Removes exponential growth  | Exponential trends                     |
| Detrending         | Removes deterministic trend | Linear/polynomial trend via regression |

  

### Differencing

  

**First Difference:** $\Delta y_t = y_t - y_{t-1}$

  

**Example:** $y = [10, 12, 15, 18, 22, 27]$

$\Delta y = [2, 3, 3, 4, 5]$

  

**Purpose:** Removes stochastic/unit-root trend.

  

### Detrending (Regression Based)

  

**Trend Model:** $y_t = a + bt + e_t$

  

Estimated trend removed from data → detrended residuals.

  

**Purpose:** Removes deterministic linear trend while keeping long-term structure.

  

### Differencing vs. Detrending

  

| Aspect         | Differencing                 | Detrending                             |
| -------------- | ---------------------------- | -------------------------------------- |
| Trend Type     | **Stochastic** (random walk) | **Deterministic** (predictable line)   |
| Method         | Subtract previous value      | Regression to subtract trend line      |
| Used In        | ARIMA for I(d) components    | When trend is structural/meaningful    |
| Trend Behavior | Shifts randomly over time    | Perfect predictable line: $y = a + bt$ |

  

---

  

## 8. SARIMA: Handling Seasonality

  

Data often repeats patterns—retail sales peak in December, temperature peaks in Summer. SARIMA extends ARIMA to model "between-period" correlations (e.g., this January vs. last January).

  

### The Notation

  

$$\text{ARIMA}(p,d,q) \times (P,D,Q)_s$$

  

### Decoding the Brackets: $(p,d,q) \times (P,D,Q)_s$

  

The structure has two "teams":

  

**Non-Seasonal Bracket $(p,d,q)$:** Relationships between today ($t$) and immediate past ($t-1, t-2$)

  

| Parameter | Name        | Meaning                                 |
| --------- | ----------- | --------------------------------------- |
| $p$       | AR          | Does today depend on yesterday?         |
| $d$       | Integration | Did we difference to remove trend?      |
| $q$       | MA          | Does today depend on yesterday's error? |

  

**Seasonal Bracket $(P,D,Q)_s$:** Relationships between today ($t$) and seasonal past ($t-s, t-2s$)

  

| Parameter | Name                 | Meaning                                           |
| --------- | -------------------- | ------------------------------------------------- |
| $P$       | Seasonal AR          | Does this January depend on last January?         |
| $D$       | Seasonal Integration | Did we subtract last January from this January?   |
| $Q$       | Seasonal MA          | Does this January depend on last January's error? |
| $s$       | Season length        | 12 for months, 4 for quarters                     |

  

### The Formula

$$\Phi_P(B^s) \phi(B) \nabla_s^D \nabla^d X_t = \Theta_Q(B^s) \theta(B) a_t$$

  

### Breaking It Down

  

| Component                | Symbol                     | Function                                     |
| ------------------------ | -------------------------- | -------------------------------------------- |
| Non-seasonal difference  | $\nabla^d$                 | Removes trend                                |
| Seasonal difference      | $\nabla_s^D$               | Removes seasonality (e.g., $X_t - X_{t-12}$) |
| Seasonal AR/MA operators | $\Phi(B^s)$, $\Theta(B^s)$ | Look back by $s$ steps (12, 24, 36...)       |

  

### Clarifying $\Theta_Q(B^s)$

  

**Question:** Is $\Theta_Q(B^s)$ basically the delta from the general form?

  

**No.**

  

| Symbol          | Type                | What It Is                                                |
| --------------- | ------------------- | --------------------------------------------------------- |
| $\delta$        | Scalar constant     | A single number (like 5 or -0.2) that shifts the equation |
| $\Theta_Q(B^s)$ | Polynomial operator | A function representing Seasonal Moving Average           |

  

**What does $\Theta_Q(B^s)$ do?**

- Regular $\theta(B)$ looks at errors from yesterday ($a_{t-1}$)

- Seasonal $\Theta_Q(B^s)$ looks at errors from one season ago ($a_{t-s}$)

  

**Example:** For monthly ice cream sales ($s=12$), $\Theta_Q(B^{12})$ tells the model: "The unexpected shock from 12 months ago affects today's sales."

  

**Where is $\delta$ in SARIMA?**

  

Usually omitted for simplicity. If included:

$$\Phi_P(B^s) \phi(B) \nabla_s^D \nabla^d X_t = \delta + \Theta_Q(B^s) \theta(B) a_t$$

  

---

  

## 9. The Symbol Cheat Sheet: Greek Letters in Time Series

  

| Component | Non-Seasonal (Lowercase) | Seasonal (Uppercase) | ACF Behavior       | PACF Behavior      |
| --------- | ------------------------ | -------------------- | ------------------ | ------------------ |
| AR        | $\phi$ (phi)             | $\Phi$ (Big Phi)     | Decays (tails off) | Spikes (cuts off)  |
| MA        | $\theta$ (theta)         | $\Theta$ (Big Theta) | Spikes (cuts off)  | Decays (tails off) |

  

**Intuition:**

- **AR terms ($\phi$, $\Phi$):** Like ripples in a pond—effect decays slowly

- **MA terms ($\theta$, $\Theta$):** Like a light switch—effect happens for $q$ lags then cuts off

  

---

  

## 10. SARIMA Examples: ACF/PACF Patterns

  

### Model A: $SARIMA(0,0,1) \times (1,0,0)_{12}$

  

**Interpreting the brackets:**

- Non-Seasonal $(0,0,1)$: $p=0$, $d=0$, $q=1$ → **1 Non-Seasonal MA term**

- Seasonal $(1,0,0)_{12}$: $P=1$, $D=0$, $Q=0$ → **1 Seasonal AR term**

  

**Equation:**

$$X_t = \Phi X_{t-12} + a_t - \theta a_{t-1}$$

  

**ACF/PACF Behavior:**

- Seasonal AR → ACF decays exponentially at lags 12, 24, 36...

- Non-seasonal MA → ACF spikes at lag 1, then cuts off

  

### Model B: $SARIMA(0,0,1) \times (0,0,1)_{12}$

  

**Components:** Non-seasonal MA(1) and Seasonal MA(1)

  

**Equation:**

$$X_t = (1-\theta B)(1-\Theta B^{12})a_t$$

  

**Expansion:**

$$X_t = a_t - \theta a_{t-1} - \Theta a_{t-12} + \theta\Theta a_{t-13}$$

  

### The "Surprise" at Lags 11 and 13

  

**Initial expectation:** Spikes only at lags 1 and 12.

  

**Reality:** Due to the interaction term $\theta\Theta a_{t-13}$, significant autocorrelation appears at **Lag 11 and Lag 13** as well.

  

---

### not really req but just some random explanation (continue from section 13)
## 11. Deep Dive: Why Lags 11 and 13 Appear (The Math)

  

### The Setup

  

We have Model B: $SARIMA(0,0,1) \times (0,0,1)_{12}$

  

**Expanded equation:**

$$X_t = a_t - \theta a_{t-1} - \Theta a_{t-12} + \theta\Theta a_{t-13}$$

  

### The Logic: The "Overlap Game"

  

**Goal:** Find Autocovariance at lag $k$: $\gamma_k = E[X_t \cdot X_{t-k}]$

  

**Key Rule:** Error terms ($a_t$) are White Noise—uncorrelated with any other $a$. For non-zero correlation at Lag $k$, we must find a **shared $a$ term** between $X_t$ and $X_{t-k}$.

  

### Finding Lag 11: $\gamma_{11} = E[X_t \cdot X_{t-11}]$

  

**$X_t$ contains:**

- $a_t$

- $-\theta a_{t-1}$

- $-\Theta a_{t-12}$ ← **This one**

- $+\theta\Theta a_{t-13}$

  

**$X_{t-11}$ contains:** (shift every index back by 11)

- $a_{t-11}$

- $-\theta a_{t-12}$ ← **This one** (since $t-1-11 = t-12$)

- $-\Theta a_{t-23}$

- $+\theta\Theta a_{t-24}$

  

**The Match:** Both contain $a_{t-12}$!

- In $X_t$: coefficient is $-\Theta$

- In $X_{t-11}$: coefficient is $-\theta$

  

**Result:**

$$\gamma_{11} = (-\Theta) \times (-\theta) \times \sigma_a^2 = \theta\Theta\sigma_a^2$$

  

### Finding Lag 13: $\gamma_{13} = E[X_t \cdot X_{t-13}]$

  

**$X_t$ contains:**

- $a_t$

- $-\theta a_{t-1}$

- $-\Theta a_{t-12}$

- $+\theta\Theta a_{t-13}$ ← **This one**

  

**$X_{t-13}$ contains:**

- $a_{t-13}$ ← **This one**

- $-\theta a_{t-14}$

- $-\Theta a_{t-25}$

- $+\theta\Theta a_{t-26}$

  

**The Match:** Both contain $a_{t-13}$!

- In $X_t$: coefficient is $+\theta\Theta$

- In $X_{t-13}$: coefficient is $1$

  

**Result:**

$$\gamma_{13} = (\theta\Theta) \times (1) \times \sigma_a^2 = \theta\Theta\sigma_a^2$$

  

### The General Rule

  

For multiplicative MA models, correlations appear at:

- $s - 1$ (Lag 11 when $s=12$)

- $s + 1$ (Lag 13 when $s=12$)

  

This "satellite" activity around the seasonal lag is the **signature of multiplicative MA models**.

  

---

  

## 12. The Infinite Regress: Causal Chains in Time Series

  

**Question:** If Jan 2025 is affected by Dec 2023 (via Lag 13), was Dec 2023 affected by Nov 2022?

  

**Yes.** You can trace this back forever:

- Jan 2025 ← Dec 2023 (13 months prior)

- Dec 2023 ← Nov 2022 (13 months prior)

- Nov 2022 ← Oct 2021 (13 months prior)

  

### Why We Stop Worrying: Damping

  

Effects are **multiplied**, so they decay rapidly.

  

**Example:** Monthly effect = 0.5, Seasonal effect = 0.5

  

| Effect                    | Calculation          | Impact |
| ------------------------- | -------------------- | ------ |
| Lag 13                    | $0.5 \times 0.5$     | 25%    |
| "Grandfather" (26 months) | $0.25 \times 0.25$   | 6.25%  |
| "Great-grandfather"       | $0.0625 \times 0.25$ | 1.56%  |

  

The echo gets quieter the further back you go, eventually becoming negligible.

  

### The Multiplication Rule

  

In SARIMA, we **multiply** (not add) Seasonal and Monthly effects:

  

$$(1 - \theta B)(1 - \Theta B^{12}) = 1 - \theta B - \Theta B^{12} + \theta\Theta B^{13}$$

  

**The specific interaction lag depends on inputs:**

  

| Monthly Lag | Seasonal Lag  | Interaction Lag |
| ----------- | ------------- | --------------- |
| 1           | 12            | 13              |
| 2           | 12            | 14              |
| 1           | 4 (quarterly) | 5               |
| 1           | 7 (weekly)    | 8               |

  

---

  

## 13. Identifying SARIMA Models: The Strategy

  

### Step 1: Visual Plot

Look for trend and seasonality. Check if pattern repeats every $s$ time units.

  

### Step 2: Differencing

  

| Observation      | Action                             |
| ---------------- | ---------------------------------- |
| Seasonality only | Take seasonal difference $(1-B^s)$ |
| Trend only       | Take regular difference $(1-B)$    |
| Both             | Do both                            |

  

### Step 3: Examine ACF/PACF of Differenced Data

  

**Non-Seasonal lags (1, 2, 3):**

- Spikes in ACF → MA terms ($q$)

- Spikes in PACF → AR terms ($p$)

  

**Seasonal lags (12, 24, 36):**

- Spikes in ACF at 12, 24... → Seasonal MA ($Q$)

- Spikes in PACF at 12, 24... → Seasonal AR ($P$)

  

### Step 4: Estimate Parameters

Fit the model using maximum likelihood or least squares.

  

### Step 5: Check Residuals

Residuals should look like white noise. If not, return to Step 3.

  

---

  

## 14. Summary: Key Formulas and Concepts

  

### ARIMA Formula

$$\phi(B)(1-B)^d X_t = \theta(B)a_t$$

  

### General ARIMA with Constant

$$\phi(B)\nabla^d X_t = \delta + \theta(B)a_t$$

  

### SARIMA Formula

$$\Phi_P(B^s) \phi(B) \nabla_s^D \nabla^d X_t = \Theta_Q(B^s) \theta(B) a_t$$

  

### Dickey-Fuller Test Statistic

$$\hat{\tau} = \frac{\hat{\phi} - 1}{\text{SE}(\hat{\phi})}$$

  

### Key Distinctions

  

| Concept    | Stochastic   | Deterministic           |
| ---------- | ------------ | ----------------------- |
| Trend Type | Random Walk  | Linear ($y = a + bt$)   |
| Solution   | Differencing | Detrending (Regression) |
| Model      | ARIMA        | Regression + ARMA       |

  

### Stationarity Checklist

  

| Indicator | Stationary               | Non-Stationary               |
| --------- | ------------------------ | ---------------------------- |
| Mean      | Constant                 | Changing                     |
| Variance  | Constant                 | Growing                      |
| ACF       | Fast decay               | Slow decay                   |
| DF Test   | $\tau$ strongly negative | $\tau$ near zero or positive |

  

---

  

## 15. Quick Reference: Model Identification

  

### From ACF/PACF to Model

  

| ACF Pattern            | PACF Pattern           | Suggested Model                  |
| ---------------------- | ---------------------- | -------------------------------- |
| Cuts off after lag $q$ | Tails off              | MA($q$)                          |
| Tails off              | Cuts off after lag $p$ | AR($p$)                          |
| Tails off              | Tails off              | ARMA($p,q$)                      |
| Slow linear decay      | -                      | Non-stationary, difference first |

  

### SARIMA Identification at Seasonal Lags

  

| Seasonal ACF | Seasonal PACF | Suggested Seasonal Component |
|--------------|---------------|------------------------------|
| Cuts off at lag $s$ | Tails off at $s, 2s, 3s...$ | Seasonal MA ($Q$) |
| Tails off at $s, 2s...$ | Cuts off at lag $s$ | Seasonal AR ($P$) |

  

---