Predictive Analytics (AD-702 (D)) Unit 4 Notes

How unit 4 is examined

This unit covers time series basics, stationarity, ACF, trend and seasonal removal, ARMA models, PACF and forecasting; no topic was asked in recent papers, so learn the definitions and formulas.

Introduction, Examples of time series

<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Not asked since 2022</span>

Definition. <mark>A time series is a set of observations $x_t$ recorded in time order, usually at equal intervals.</mark>

Key points.

Successive values are dependent, so ordinary independent-sample methods do not apply.
Examples are daily stock prices, monthly rainfall, yearly sales and hourly electricity demand.
A series has trend, seasonality, cycles and irregular noise.
The aims are description, modelling and forecasting future values.

Stationary models and autocorrelation function

Definition. <mark>A series is weakly stationary if its mean is constant and its autocovariance depends only on the lag $h$.</mark>

Formula. $\gamma(h)=\mathrm{Cov}(X_{t+h},X_t)$ and $\rho(h)=\gamma(h)/\gamma(0)$.

Key points.

Stationarity needs constant mean, constant finite variance and $\gamma(h)$ free of $t$.
White noise, with zero mean and variance $\sigma^2$, is the simplest stationary model.
The ACF satisfies $\rho(0)=1$, $\rho(h)=\rho(-h)$ and $|\rho(h)|\le 1$.

Example. For $x=2,4,6,8$: $\bar x=5$, $\hat\gamma(0)=(9+1+1+9)/4=5$ and $\hat\gamma(1)=(3-1+3)/4=1.25$, so $\hat\rho(1)=1.25/5=0.25$. $\hat\rho(1)=0.25$

Key points (more). A random walk $X_t=X_{t-1}+Z_t$ is not stationary because its variance $t\sigma^2$ grows with time. Strict stationarity needs the whole joint distribution to be shift-invariant; weak stationarity needs only mean and covariance, which is what this unit uses.

Estimation and elimination of trend and seasonal components

Definition. ==The classical decomposition writes $X_t=m_t+s_t+Y_t$, with trend $m_t$, seasonal part $s_t$ and stationary noise $Y_t$.==

Key points.

A trend is estimated by a moving average or by fitting a polynomial with least squares.
Differencing, $\nabla X_t=X_t-X_{t-1}$, removes a trend; a polynomial trend of degree $k$ needs $k$ differences.
Seasonality of period $d$ is removed by lag-$d$ differencing, $X_t-X_{t-d}$, or by subtracting estimated seasonal averages.
The remaining residual $Y_t$ is then checked for stationarity.

Example. For $x=2,4,6,8$ the first difference is $2,2,2$, a constant, so the linear trend is gone. A quarterly series has $d=4$, so use $X_t-X_{t-4}$. A two-sided moving average $\hat m_t=\frac1{2q+1}\sum_{j=-q}^{q}X_{t+j}$ smooths noise and gives the trend estimate.

Stationary Process and ARMA Models - Basic properties and linear processes

Definition. ==A linear process is $X_t=\sum_{j=-\infty}^{\infty}\psi_j Z_{t-j}$ with $\{Z_t\}$ white noise and $\sum|\psi_j|<\infty$.==

Key points.

Such a series is stationary with mean 0 and $\gamma(h)=\sigma^2\sum_j\psi_j\psi_{j+h}$.
White noise $Z_t\sim WN(0,\sigma^2)$ has uncorrelated terms and is the building block.
ARMA models are linear processes with a finite number of parameters.
If $\psi_j=0$ for $j<0$, the process is causal, meaning it depends only on the past and present.

Example. Take $X_t=Z_t+0.5Z_{t-1}$, so $\psi_0=1,\psi_1=0.5$. Then $\gamma(0)=\sigma^2(1+0.25)=1.25\sigma^2$, $\gamma(1)=0.5\sigma^2$ and $\gamma(h)=0$ for $h\ge2$, hence $\rho(1)=0.4$. $\rho(1)=0.4$

Introduction to ARMA models

Definition. <mark>An ARMA model combines an autoregressive (AR) part, regressing on past values, with a moving-average (MA) part, regressing on past noise.</mark>

Key points.

AR(1) is $X_t=\phi X_{t-1}+Z_t$ and MA(1) is $X_t=Z_t+\theta Z_{t-1}$.
AR(1) is stationary when $|\phi|<1$, and then $\rho(h)=\phi^h$.
MA(1) has $\rho(1)=\theta/(1+\theta^2)$ and $\rho(h)=0$ for $h>1$.
The Box-Jenkins method fits ARMA in three steps: identify, estimate, check.

Example. For AR(1) with $\phi=0.6$: $\rho(1)=0.6$, $\rho(2)=0.36$, $\rho(3)=0.216$, a geometric decay. For MA(1) with $\theta=0.5$: $\rho(1)=0.5/1.25=0.4$ and zero afterwards. AR needs $|\phi|<1$; MA is always stationary.

Properties of sample mean and autocorrelation function

Definition. ==The sample mean is $\bar x=\frac1n\sum x_t$ and the sample ACF is $\hat\rho(h)=\hat\gamma(h)/\hat\gamma(0)$.==

Formula. $\hat\gamma(h)=\frac1n\sum_{t=1}^{n-h}(x_{t+h}-\bar x)(x_t-\bar x)$.

Key points.

$\bar x$ is an unbiased estimator of $\mu$, and for white noise its variance is $\sigma^2/n$.
For white noise, $\hat\rho(h)$ is approximately $N(0,1/n)$.
Hence 95% bounds $\pm1.96/\sqrt n$ are drawn on the correlogram; values outside them are significant.

Example. With $n=100$ the bounds are $\pm1.96/10=\pm0.196$; a sample $\hat\rho(1)=0.35$ lies outside, so the lag-1 correlation is significant and the series is not white noise. Bound $=\pm0.196$

Diagram. <figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u4-01" viewBox="0 0 467 277.8" width="467" height="277.8" role="img" aria-label="Correlogram idea: sample ACF at lags 0 to 3, compared against the bounds"><style>#dsfig-u4-01 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u4-01 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u4-01 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u4-01 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u4-01 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u4-01 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u4-01 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u4-01 .t{fill:#16181D;font-weight:500}#dsfig-u4-01 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u4-01 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u4-01 .dot{fill:#16181D}#dsfig-u4-01 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u4-01 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u4-01 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u4-01 .ah{fill:#454C5A}#dsfig-u4-01 .ah.hi{fill:#2340B8}#dsfig-u4-01 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u4-01 .wl .t{font-size:12px;font-weight:700}#dsfig-u4-01 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u4-01 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u4-01 .e{stroke:#B1B7C3}html.dark #dsfig-u4-01 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u4-01 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u4-01 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u4-01 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u4-01 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u4-01 .t{fill:#E6E8ED}html.dark #dsfig-u4-01 .t.inv{fill:#0F1115}html.dark #dsfig-u4-01 .kd{stroke:#E6E8ED}html.dark #dsfig-u4-01 .dot{fill:#E6E8ED}html.dark #dsfig-u4-01 .ann{fill:#8FA3FF}html.dark #dsfig-u4-01 .lbl{fill:#858D9C}html.dark #dsfig-u4-01 .ptr{fill:#8FA3FF}html.dark #dsfig-u4-01 .ah{fill:#B1B7C3}html.dark #dsfig-u4-01 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u4-01 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u4-01 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u4-01 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah1" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh1" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M53.4,155.6 L154.2,54.8" marker-end="url(#ah1)"/><path class="e" d="M180,55.5 L285.8,203.5" marker-end="url(#ah1)"/><path class="e" d="M316.8,223.1 L406.2,235" marker-end="url(#ah1)"/><circle class="n" cx="40" cy="169" r="18"/><text class="t" x="40" y="169" dy=".35em" text-anchor="middle">L0</text><circle class="n" cx="169" cy="40" r="18"/><text class="t" x="169" y="40" dy=".35em" text-anchor="middle">L1</text><circle class="n" cx="298" cy="220.6" r="18"/><text class="t" x="298" y="220.6" dy=".35em" text-anchor="middle">L2</text><circle class="n" cx="427" cy="237.8" r="18"/><text class="t" x="427" y="237.8" dy=".35em" text-anchor="middle">L3</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">Correlogram idea: sample ACF at lags 0 to 3, compared against the bounds</figcaption></figure>

Forecasting stationary time series

Definition. <mark>The best linear predictor of $X_{n+h}$ is the linear function of $X_1,\dots,X_n$ that minimises the mean squared error.</mark>

Key points.

The predictor $P_nX_{n+h}$ satisfies the orthogonality condition: the error is uncorrelated with every observed $X_i$.
For AR(1), the forecast is $\hat X_{n+h}=\phi^h X_n$.
As $h$ grows, the forecast tends to the mean and the error variance to $\gamma(0)$.

Example. AR(1), $\phi=0.6$, $\mu=0$, last value $x_n=10$: $\hat x_{n+1}=6$, $\hat x_{n+2}=3.6$, $\hat x_{n+3}=2.16$. The one-step error variance is $\sigma^2$, and the $h$-step error variance is $\sigma^2(1-\phi^{2h})/(1-\phi^2)$. Forecasts $6,\ 3.6,\ 2.16$

ARMA(p, q) processes

Definition. ==$\{X_t\}$ is ARMA(p,q) if $X_t-\phi_1X_{t-1}-\dots-\phi_pX_{t-p}=Z_t+\theta_1Z_{t-1}+\dots+\theta_qZ_{t-q}$.==

Key points.

In operator form, $\phi(B)X_t=\theta(B)Z_t$, where $B$ is the backshift operator.
It is stationary and causal if the roots of $\phi(z)=0$ lie outside the unit circle.
It is invertible if the roots of $\theta(z)=0$ lie outside the unit circle.
The orders $p$ and $q$ are chosen from the ACF and PACF, or by minimising AIC.

Example. ARMA(1,1): $X_t=0.5X_{t-1}+Z_t+0.4Z_{t-1}$. Here $\phi(z)=1-0.5z$ has root $z=2$ and $\theta(z)=1+0.4z$ has root $z=-2.5$; both lie outside the unit circle, so the process is causal and invertible. AIC $=-2\ln L+2(p+q+1)$ penalises extra parameters.

ACF and PACF

Definition. <mark>The PACF at lag $h$ is the correlation between $X_t$ and $X_{t+h}$ after removing the effect of the intermediate values.</mark>

Key points.

AR(p): ACF tails off, PACF cuts off after lag $p$.
MA(q): ACF cuts off after lag $q$, PACF tails off.
ARMA(p,q): both tail off.
These patterns identify the model orders on the correlogram.

Example. A sample ACF that decays slowly while the PACF has one significant spike at lag 1 and then falls inside the bounds suggests AR(1). The reverse pattern, a single ACF spike at lag 1 and a decaying PACF, suggests MA(1).

Model	ACF	PACF
AR(p)	tails off	cuts off after $p$
MA(q)	cuts off after $q$	tails off
ARMA(p,q)	tails off	tails off

Modeling and Forecasting with ARMA

Definition. <mark>Modelling with ARMA follows Box-Jenkins: identify the orders, estimate the parameters, check the residuals, then forecast.</mark>

Steps.

Step 1: Make the series stationary by differencing or transforming.
Step 2: Choose p and q from the ACF and PACF.
Step 3: Estimate parameters by maximum likelihood or least squares.
Step 4: Check that residuals look like white noise.
Step 5: Forecast, and give prediction intervals.

Key points. Future noise terms are replaced by zero and past terms by residuals. The forecast approaches the mean as $h$ grows.

Diagram. <figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u4-02" viewBox="0 0 510 80" width="510" height="80" role="img" aria-label="Box-Jenkins cycle. Idn identify, Est estimate, Chk diagnostic check, Fc forecast"><style>#dsfig-u4-02 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u4-02 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u4-02 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u4-02 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u4-02 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u4-02 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u4-02 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u4-02 .t{fill:#16181D;font-weight:500}#dsfig-u4-02 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u4-02 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u4-02 .dot{fill:#16181D}#dsfig-u4-02 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u4-02 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u4-02 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u4-02 .ah{fill:#454C5A}#dsfig-u4-02 .ah.hi{fill:#2340B8}#dsfig-u4-02 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u4-02 .wl .t{font-size:12px;font-weight:700}#dsfig-u4-02 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u4-02 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u4-02 .e{stroke:#B1B7C3}html.dark #dsfig-u4-02 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u4-02 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u4-02 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u4-02 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u4-02 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u4-02 .t{fill:#E6E8ED}html.dark #dsfig-u4-02 .t.inv{fill:#0F1115}html.dark #dsfig-u4-02 .kd{stroke:#E6E8ED}html.dark #dsfig-u4-02 .dot{fill:#E6E8ED}html.dark #dsfig-u4-02 .ann{fill:#8FA3FF}html.dark #dsfig-u4-02 .lbl{fill:#858D9C}html.dark #dsfig-u4-02 .ptr{fill:#8FA3FF}html.dark #dsfig-u4-02 .ah{fill:#B1B7C3}html.dark #dsfig-u4-02 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u4-02 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u4-02 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u4-02 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M59,40 L165.2,40" marker-end="url(#ah2)"/><path class="e" d="M205.2,40 L311.4,40" marker-end="url(#ah2)"/><path class="e" d="M351.4,40 L449,40" marker-end="url(#ah2)"/><path class="e" d="M313.4,40 L61,40" marker-end="url(#ah2)"/><g class="wl"><rect x="162.6" y="31" width="47.1" height="18" rx="9"/><text class="t" x="186.2" y="40" dy=".35em" text-anchor="middle">refit</text></g><circle class="n" cx="40" cy="40" r="18"/><text class="t" x="40" y="40" dy=".35em" text-anchor="middle">Idn</text><circle class="n" cx="186.2" cy="40" r="18"/><text class="t" x="186.2" y="40" dy=".35em" text-anchor="middle">Est</text><circle class="n" cx="332.4" cy="40" r="18"/><text class="t" x="332.4" y="40" dy=".35em" text-anchor="middle">Chk</text><circle class="n" cx="470" cy="40" r="18"/><text class="t" x="470" y="40" dy=".35em" text-anchor="middle">Fc</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">Box-Jenkins cycle. Idn identify, Est estimate, Chk diagnostic check, Fc forecast</figcaption></figure>

Example. For ARMA(1,1) with $\phi=0.5,\theta=0.4$, last $x_n=10$ and last residual $\hat z_n=1$: $\hat x_{n+1}=0.5(10)+0.4(1)=5.4$ and $\hat x_{n+2}=0.5(5.4)=2.7$, since future noise is set to zero. Forecasts $5.4,\ 2.7$

Last-minute revision

A time series is data recorded in time order; the terms are dependent.
Weak stationarity: constant mean, variance, and $\gamma(h)$ depending only on lag.
$\rho(h)=\gamma(h)/\gamma(0)$, with $\rho(0)=1$.
Decomposition: $X_t=m_t+s_t+Y_t$.
Differencing removes trend; lag-$d$ differencing removes seasonality.
AR(1): $\rho(h)=\phi^h$, stationary if $|\phi|<1$.
MA(1): $\rho(1)=\theta/(1+\theta^2)$, zero beyond lag 1.
Sample ACF bounds for white noise: $\pm1.96/\sqrt n$.
AR: PACF cuts off at $p$; MA: ACF cuts off at $q$.
Box-Jenkins: identify, estimate, check, forecast.

Memory hooks

AR cuts in PACF, MA cuts in ACF (A-P, M-C).
Box-Jenkins spells I-E-C-F: identify, estimate, check, forecast.
Stationary means roots outside the circle.
Trend needs difference; season needs lag-d difference.

Coverage checklist

Introduction, Examples of time series: covered; no past questions.
Stationary models and autocorrelation function: covered; no past questions.
Estimation and elimination of trend and seasonal components: covered; no past questions.
Stationary Process and ARMA Models - Basic properties and linear processes: covered; no past questions.
Introduction to ARMA models: covered; no past questions.
Properties of sample mean and autocorrelation function: covered; no past questions.
Forecasting stationary time series: covered; no past questions.
ARMA(p, q) processes: covered; no past questions.
ACF and PACF: covered; no past questions.
Modeling and Forecasting with ARMA: covered; no past questions.

Account

Predictive Analytics (AD-702 (D)) - Unit 4 Short Notes

How unit 4 is examined

Introduction, Examples of time series

Stationary models and autocorrelation function

Estimation and elimination of trend and seasonal components

Stationary Process and ARMA Models - Basic properties and linear processes

Introduction to ARMA models

Properties of sample mean and autocorrelation function

Forecasting stationary time series

ARMA(p, q) processes

ACF and PACF

Modeling and Forecasting with ARMA

Last-minute revision

Memory hooks

Coverage checklist