1. Fundamentals of Probability Theory
Axioms of Probability
-
Non-negativity: $P(A) \geq 0$ for any event $A$.
-
Normalization: $$\displaystyle P(\Omega) = 1 $$.
-
Additivity: For disjoint events $$\displaystyle A_1, A_2, \dots $$, $$\displaystyle P\left(\bigcup_{i} A_i\right) = \sum_{i} P(A_i) $$.
Conditional Probability and Independence
-
Conditional Probability:
$$\displaystyle P(A|B) = \dfrac{P(A \cap B)}{P(B)}, \quad P(B) > 0 $$.
-
Independence:
$A$ and $B$ are independent iff $$\displaystyle P(A \cap B) = P(A)P(B) $$.
Bayes' Theorem
-
Statement:
For a partition $$\displaystyle \{A_i\} $$ of the sample space,
$$ P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_j P(B|A_j)P(A_j)}. $$
-
Proof:
$$\displaystyle P(A_i|B) = \dfrac{P(A_i \cap B)}{P(B)} $$. Since $$\displaystyle \{A_i\} $$ partition $B$,
$$\displaystyle P(B) = \sum_j P(B \cap A_j) = \sum_j P(B|A_j)P(A_j) $$.
Substituting gives the result.
\boxed{P(A_i|B) = \frac{P(B|A_i)P(A_i)}{\sum_j P(B|A_j)P(A_j)}}
[!TIP] Common Pitfall: Confusing $P(A|B)$ with $P(B|A)$. Always identify the "given" event.
Independence of Complementary Events
If $A$ and $B$ are independent, then:
-
$A'$ and $B$ are independent.
-
$A$ and $B'$ are independent.
-
$A'$ and $B'$ are independent.
Proof for $A'$ and $B$:
$$\displaystyle P(A' \cap B) = P(B) - P(A \cap B) = P(B) - P(A)P(B) = P(B)(1-P(A)) = P(B)P(A') $$.
Similarly for others.
[!TIP] Use set identities: $$\displaystyle P(A' \cap B) = P(B) - P(A \cap B) $$, etc.
Application: Conditional Probability in System Reliability (Missile Launch Example)
Given:
$$\displaystyle P(A \text{ fail}) = 0.01 $$, $$\displaystyle P(B \text{ fail}) = 0.03 $$, $$\displaystyle P(B \text{ fail}|A \text{ fail}) = 0.06 $$.
-
(i) Probability of accidental launch (both fail):
$$\displaystyle P(A \cap B) = P(B|A)P(A) = 0.06 \times 0.01 = \boxed{0.0006} $$.
-
(ii) Probability that $A$ fails given $B$ fails:
$$\displaystyle P(A|B) = \dfrac{P(A \cap B)}{P(B)} = \dfrac{0.0006}{0.03} = \boxed{0.02} $$.
-
(iii) Independence check:
$$\displaystyle P(A)P(B) = 0.01 \times 0.03 = 0.0003 \neq P(A \cap B) = 0.0006 $$ → not independent.
Random Variables and Distributions
Gaussian (Normal) Distribution:
-
PDF: $$\displaystyle f_X(x) = \dfrac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{(x-\mu)^2}{2\sigma^2}} $$.
-
Mean: $\mu$, Variance: $$\displaystyle \sigma^2 $$.
-
Standardization: $$\displaystyle Z = \dfrac{X - \mu}{\sigma} \sim N(0,1) $$.
-
Probability calculation:
$$\displaystyle P(a < X < b) = \Phi\left(\dfrac{b-\mu}{\sigma}\right) - \Phi\left(\dfrac{a-\mu}{\sigma}\right) $$, where $\Phi$ is the standard normal CDF.
[!TIP] Always standardize before using $Z$-tables.
Example: $X \sim N(1,4)$ ($$\displaystyle \mu=1 $$, $$\displaystyle \sigma=2 $$). Find $$\displaystyle P(1<X<2) $$:
$$\displaystyle P(1<X<2) = P\left(0 < Z < \dfrac{2-1}{2}\right) = P(0<Z<0.5) = \Phi(0.5) - \Phi(0) = 0.6915 - 0.5 = \boxed{0.1915} $$.
Rayleigh Distribution:
-
PDF: $$\displaystyle f_X(x) = \dfrac{x}{\sigma^2} e^{-x^2/(2\sigma^2)}, \quad x \geq 0 $$.
-
Validation (prove it integrates to 1):
$$ \int_0^\infty \frac{x}{\sigma^2} e^{-x^2/(2\sigma^2)} \, dx. $$
Substitute $$\displaystyle u = \dfrac{x^2}{2\sigma^2} $$, $$\displaystyle du = \dfrac{x}{\sigma^2} dx $$:
$$ \int_0^\infty e^{-u} \, du = 1. $$
\boxed{\text{Valid PDF}}
Poisson Distribution:
-
PMF: $$\displaystyle P(X=k) = \dfrac{e^{-\lambda} \lambda^k}{k!}, \quad k=0,1,2,\dots $$
-
Moment Generating Function (MGF):
$$ M_X(t) = E[e^{tX}] = \sum_{k=0}^\infty e^{tk} \frac{e^{-\lambda} \lambda^k}{k!} = e^{-\lambda} \sum_{k=0}^\infty \frac{(\lambda e^t)^k}{k!} = e^{-\lambda} e^{\lambda e^t} = e^{\lambda(e^t - 1)}. $$
\boxed{M_X(t) = e^{\lambda(e^t - 1)}}
Moment Generating Functions (MGF)
-
Definition: $$\displaystyle M_X(t) = E[e^{tX}] $$, exists if finite in a neighborhood of $$\displaystyle t=0 $$.
-
Properties:
-
Uniqueness: If $$\displaystyle M_X(t) = M_Y(t) $$ for all $t$ in a neighborhood of 0, then $X$ and $Y$ have identical distributions.
-
MGF of Sum: For independent $X$ and $Y$, $$\displaystyle M_{X+Y}(t) = M_X(t) M_Y(t) $$.
-
Moments: $$\displaystyle E[X^n] = M_X^{(n)}(0) $$ (n-th derivative at 0).
-
[!TIP] MGF may not exist for all distributions (e.g., Cauchy).
Transformations of Random Variables
Sum of Independent Uniform Random Variables:
Let $X, Y \sim \text{Uniform}(-1,1)$ independent. Find PDF of $$\displaystyle Z = X+Y $$.
-
Method: Convolution $$\displaystyle f_Z(z) = \int_{-\infty}^\infty f_X(x) f_Y(z-x) \, dx $$.
-
$$\displaystyle f_X(x) = \dfrac{1}{2} $$ for $$\displaystyle |x|<1 $$, else 0.
-
Support: $Z \in (-2, 2)$.
-
For $$\displaystyle -2 < z < 0 $$:
$$\displaystyle f_Z(z) = \int_{-1}^{z+1} \dfrac{1}{2} \cdot \dfrac{1}{2} \, dx = \dfrac{1}{4}(z+2) $$.
-
For $$\displaystyle 0 \leq z < 2 $$:
$$\displaystyle f_Z(z) = \int_{z-1}^{1} \dfrac{1}{2} \cdot \dfrac{1}{2} \, dx = \dfrac{1}{4}(2-z) $$.
-
Result (triangular PDF):
$$ f_Z(z) = \begin{cases} \dfrac{1}{4}(z+2), & -2 < z < 0 \\[6pt] \dfrac{1}{4}(2-z), & 0 \leq z < 2 \\[6pt] 0, & \text{otherwise} \end{cases} $$
Equivalently, $$\displaystyle f_Z(z) = \dfrac{1}{4}(2 - |z|) $$ for $$\displaystyle |z|<2 $$.
\boxed{f_Z(z) = \frac{1}{4}(2 - |z|), \quad |z| < 2}
2. Joint Distributions and Expectations
Joint Distribution Functions
-
Joint CDF: $$\displaystyle F_{X,Y}(x,y) = P(X \leq x, Y \leq y) $$.
-
Properties:
-
Non-decreasing in $x$ and $y$.
-
$$\displaystyle F_{X,Y}(-\infty, y) = 0 $$, $$\displaystyle F_{X,Y}(x, -\infty) = 0 $$.
-
$$\displaystyle F_{X,Y}(\infty, \infty) = 1 $$.
-
Right-continuous.
-
For any $$\displaystyle x_1 < x_2 $$, $$\displaystyle y_1 < y_2 $$:
$$\displaystyle P(x_1 < X \leq x_2, y_1 < Y \leq y_2) = F(x_2,y_2) - F(x_1,y_2) - F(x_2,y_1) + F(x_1,y_1) $$.
-
Proof of Property 5:
Let $$\displaystyle A = \{x_1 < X \leq x_2, y_1 < Y \leq y_2\} $$. Then
$$\displaystyle A = \{X \leq x_2, Y \leq y_2\} \setminus \{X \leq x_1, Y \leq y_2\} \setminus \{X \leq x_2, Y \leq y_1\} \cup \{X \leq x_1, Y \leq y_1\} $$.
Taking probabilities and using inclusion-exclusion yields the result.
Time Average and Ergodicity
-
Time Average (for process $X(t)$):
$$\displaystyle \bar{X} = \lim_{T \to \infty} \dfrac{1}{T} \int_0^T X(t) \, dt $$.
-
Ensemble Average: $$\displaystyle \mu_X(t) = E[X(t)] $$.
-
Ergodicity: A process is mean-ergodic if $$\displaystyle \bar{X} = \mu_X(t) $$ with probability 1 (for WSS, $$\displaystyle \mu_X $$ constant). Similarly for correlation-ergodic.
-
Condition: For a WSS process, if $$\displaystyle R_{xx}(\tau) \to 0 $$ as $|\tau| \to \infty$, then the process is mean-ergodic.
[!TIP] Ergodicity allows replacing time averages with ensemble averages, essential for practical estimation from a single sample path.
3. Random Processes
Definitions and Classification
A random process $X(t)$ is a collection of random variables indexed by time $t$.
-
By time: Continuous-time ($t \in \mathbb{R}$), discrete-time ($t \in \mathbb{Z}$).
-
By state: Continuous-state, discrete-state.
-
By determinism: Deterministic ($$\displaystyle X(t)=f(t) $$), non-deterministic.
-
By stationarity: Strict-sense stationary (SSS), wide-sense stationary (WSS), non-stationary.
Stationarity
-
Strict-Sense Stationary (SSS): All finite-dimensional distributions invariant under time shift:
$$\displaystyle F_{X(t_1),\dots,X(t_n)}(x_1,\dots,x_n) = F_{X(t_1+\tau),\dots,X(t_n+\tau)}(x_1,\dots,x_n) $$ for all $\tau$, $n$, $$\displaystyle t_i $$.
-
Wide-Sense Stationary (WSS):
-
Mean constant: $$\displaystyle E[X(t)] = \mu $$ (independent of $t$).
-
Autocorrelation depends only on time difference:
$$\displaystyle R_{xx}(t_1, t_2) = R_{xx}(t_2 - t_1) $$.
-
[!TIP] WSS is weaker than SSS. All SSS are WSS if second moments exist, but converse is false.
Stationarity Analysis of Specific Processes
Process 1: $$\displaystyle X(t) = A \cos \omega t $$, $A \sim \text{Uniform}(0,\pi)$.
-
Mean: $$\displaystyle E[X(t)] = E[A] \cos \omega t = \dfrac{\pi}{2} \cos \omega t $$ → depends on $t$ → not WSS.
-
Distribution of $X(t)$ depends on $t$ (scaled by $\cos \omega t$) → not SSS.
Process 2: $$\displaystyle X(t) = \cos(\omega t + \theta) $$, $\theta \sim \text{Uniform}(0, \pi/2)$.
-
Mean:
$$\displaystyle E[X(t)] = \int_0^{\pi/2} \cos(\omega t + \theta) \cdot \dfrac{2}{\pi} \, d\theta = \dfrac{2}{\pi} [\sin(\omega t + \theta)]_0^{\pi/2} = \dfrac{2}{\pi} (\cos \omega t - \sin \omega t) $$ → depends on $t$ → not WSS.
-
Average Power ($$\displaystyle E[X(t)^2] $$):
$$\displaystyle E[\cos^2(\omega t + \theta)] = \dfrac{1}{2} + \dfrac{1}{2} E[\cos(2\omega t + 2\theta)] $$.
Compute $E[\cos(2\omega t + 2\theta)]$:
$$\displaystyle E[e^{j2\theta}] = \dfrac{2}{\pi} \int_0^{\pi/2} e^{j2\theta} d\theta = \dfrac{2}{j\pi} (e^{j\pi} - 1) = \dfrac{2}{j\pi} (-2) = \dfrac{j4}{\pi} $$.
So $$\displaystyle E[\cos(2\omega t + 2\theta)] = \operatorname{Re}\left[ e^{j2\omega t} \cdot \dfrac{j4}{\pi} \right] = -\dfrac{4}{\pi} \sin 2\omega t $$.
Thus,
$$\displaystyle E[X(t)^2] = \dfrac{1}{2} - \dfrac{2}{\pi} \sin 2\omega t $$ → time-varying.
\boxed{\text{Average power } = \frac{1}{2} - \frac{2}{\pi} \sin 2\omega t \text{ (not constant)}}
Correlation and Spectral Analysis
Autocorrelation Function: $$\displaystyle R_{xx}(\tau) = E[X(t) X(t+\tau)] $$ (for WSS, depends only on $\tau$).
Cross-correlation Function: $$\displaystyle R_{xy}(\tau) = E[X(t) Y(t+\tau)] $$.
Power Spectral Density (PSD):
-
Definition: $$\displaystyle S_{xx}(\omega) = \int_{-\infty}^{\infty} R_{xx}(\tau) e^{-j\omega\tau} d\tau $$.
-
Wiener-Khinchin Theorem: PSD and autocorrelation form a Fourier transform pair:
$$ S_{xx}(\omega) = \int_{-\infty}^{\infty} R_{xx}(\tau) e^{-j\omega\tau} d\tau, \quad R_{xx}(\tau) = \frac{1}{2\pi} \int_{-\infty}^{\infty} S_{xx}(\omega) e^{j\omega\tau} d\omega. $$
Proof Sketch:
Define truncated process $$\displaystyle X_T(t) = X(t) $$ for $0 \leq t \leq T$, else 0. Then
$$\displaystyle S_T(\omega) = \dfrac{1}{T} |X_T(\omega)|^2 $$, where $$\displaystyle X_T(\omega) = \int_0^T X(t) e^{-j\omega t} dt $$.
Take expectation: $$\displaystyle E[|X_T(\omega)|^2] = \int_0^T \int_0^T E[X(u)X(v)] e^{-j\omega(u-v)} du dv = \int_{-T}^T R_{xx}(\tau) (T-|\tau|) e^{-j\omega\tau} d\tau $$.
Divide by $T$ and let $T \to \infty$: $$\displaystyle \dfrac{1}{T} E[|X_T(\omega)|^2] \to \int_{-\infty}^{\infty} R_{xx}(\tau) e^{-j\omega\tau} d\tau = S_{xx}(\omega) $$. The inverse transform follows from Fourier inversion.
[!TIP] Wiener-Khinchin allows computing PSD directly from autocorrelation without infinite-time integrals.
Cross Power Spectral Density:
-
$$\displaystyle S_{xy}(\omega) = \int_{-\infty}^{\infty} R_{xy}(\tau) e^{-j\omega\tau} d\tau $$.
-
Properties (for general complex processes, with $$\displaystyle R_{xy}(\tau) = E[X(t) Y^*(t+\tau)] $$):
-
Conjugate Symmetry: $$\displaystyle S_{xy}(\omega) = S_{yx}^*(-\omega) $$.
Proof: $$\displaystyle R_{yx}(\tau) = E[Y(t) X^*(t+\tau)] = R_{xy}^*(-\tau) $$. Then
$$\displaystyle S_{yx}(\omega) = \int R_{yx}(\tau) e^{-j\omega\tau} d\tau = \int R_{xy}^*(-\tau) e^{-j\omega\tau} d\tau = \left[ \int R_{xy}(\tau) e^{j\omega\tau} d\tau \right]^* = S_{xy}^*(-\omega) $$.
-
Cauchy-Schwarz Inequality: $$\displaystyle |S_{xy}(\omega)| \leq \sqrt{S_{xx}(\omega) S_{yy}(\omega)} $$.
Proof: From $$\displaystyle |R_{xy}(\tau)| \leq \sqrt{R_{xx}(0) R_{yy}(0)} $$ (Cauchy-Schwarz for random variables). Taking Fourier transforms and using magnitude inequalities yields the result.
-
Uncorrelated Processes: If $X$ and $Y$ are uncorrelated ($$\displaystyle R_{xy}(\tau)=0 $$ for all $\tau$), then $$\displaystyle S_{xy}(\omega)=0 $$.
Proof: Direct from definition.
-
Linear System Response
-
Setup: LTI system with impulse response $h(t)$, frequency response $$\displaystyle H(\omega) = \int_{-\infty}^{\infty} h(t) e^{-j\omega t} dt $$. Input $X(t)$, output $$\displaystyle Y(t) = X(t) * h(t) $$.
-
Output PSD:
$$ S_{yy}(\omega) = |H(\omega)|^2 S_{xx}(\omega). $$
-
Proof:
$$\displaystyle R_{yy}(\tau) = E[Y(t) Y(t+\tau)] = E\left[ \int\int X(u) h(t-u) X(v) h(t+\tau-v) du dv \right] $$
$$\displaystyle = \int\int E[X(u) X(v)] h(t-u) h(t+\tau-v) du dv = \int\int R_{xx}(v-u) h(t-u) h(t+\tau-v) du dv $$.
Substitute $$\displaystyle \lambda = v-u $$:
$$\displaystyle R_{yy}(\tau) = \int R_{xx}(\lambda) \left[ \int h(t-u) h(t+\tau-u-\lambda) du \right] d\lambda $$.
The inner integral is the autocorrelation of $h(t)$: $$\displaystyle R_{hh}(\tau-\lambda) = \int h(t) h(t+\tau-\lambda) dt $$ (after change of variables). Thus,
$$\displaystyle R_{yy}(\tau) = R_{xx}(\tau) * R_{hh}(\tau) $$.
Fourier transform: $$\displaystyle S_{yy}(\omega) = S_{xx}(\omega) S_{hh}(\omega) $$. But $$\displaystyle S_{hh}(\omega) = |H(\omega)|^2 $$ because
$$\displaystyle |H(\omega)|^2 = H(\omega) H^*(\omega) = \int\int h(t) h^*(s) e^{-j\omega(t-s)} dt ds = \int \left[ \int h(t) h^*(t-\tau) dt \right] e^{-j\omega\tau} d\tau = \int R_{hh}(\tau) e^{-j\omega\tau} d\tau $$.
Hence, $$\displaystyle S_{yy}(\omega) = |H(\omega)|^2 S_{xx}(\omega) $$.
\boxed{S_{yy}(\omega) = |H(\omega)|^2 S_{xx}(\omega)}