Probability Theory and Stochastic processing (EC-703 (C)) - Unit 4 Short Notes

1. Fundamentals of Probability

Bayes' Theorem

Statement: For events \(A_1, A_2, \dots, A_n\) forming a partition of the sample space \(S\) with \(P(A_i) > 0\), and any event \(B\) with \(P(B) > 0\),

$$ P(A_i | B) = \frac{P(B | A_i) P(A_i)}{\sum_{j=1}^{n} P(B | A_j) P(A_j)} $$

Proof: From conditional probability definition:

$$ P(A_i \cap B) = P(B | A_i) P(A_i) = P(A_i | B) P(B) $$

Since \(B = \bigcup_{j=1}^{n} (A_j \cap B)\) and the \(A_j\) are disjoint,

$$ P(B) = \sum_{j=1}^{n} P(A_j \cap B) = \sum_{j=1}^{n} P(B | A_j) P(A_j) $$

Substituting,

$$ P(A_i | B) = \frac{P(B | A_i) P(A_i)}{\sum_{j=1}^{n} P(B | A_j) P(A_j)} $$

\boxed{P(A_i | B) = \frac{P(B | A_i) P(A_i)}{\sum_{j=1}^{n} P(B | A_j) P(A_j)}}

[!TIP] Common in exam: Apply to medical testing, machine failure diagnosis. Always verify partition condition.

Independence of Events

• Definition: Events \(A\) and \(B\) are independent iff \(P(A \cap B) = P(A) P(B)\).

• Derived Independent Events: If \(A\) and \(B\) are independent, then:

  1. \(A'\) and \(B\) are independent.

  2. \(A\) and \(B'\) are independent.

  3. \(A'\) and \(B'\) are independent.

Proof for \(A'\) and \(B\):

$$ P(A' \cap B) = P(B) - P(A \cap B) = P(B) - P(A)P(B) = P(B)(1 - P(A)) = P(B)P(A') $$

Similar proofs for others.

[!TIP] Independence is not pairwise; mutual independence requires all combinations.

Conditional Probability & Joint Probabilities

• Conditional Probability: \(P(A|B) = \frac{P(A \cap B)}{P(B)}\), \(P(B) > 0\).

• Joint Probability: \(P(A \cap B)\).

• Missile Launch Problem (Past Paper):

  Let \(A\): Relay A fails, \(P(A)=0.01\); \(B\): Relay B fails, \(P(B)=0.03\); \(P(B|A)=0.06\).

  Accidental launch: \(P(A \cap B) = P(B|A)P(A) = 0.06 \times 0.01 = 0.0006\).

  \(P(A|B)\): \(P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{0.0006}{0.03} = 0.02\).

  Independence? Check \(P(A \cap B) \stackrel{?}{=} P(A)P(B) = 0.01 \times 0.03 = 0.0003\). Since \(0.0006 \neq 0.0003\), not independent.

2. Random Variables and Distributions

Common Distributions

[!TIP] Rayleigh validation: Show \(\int_0^\infty \frac{x}{\sigma^2} e^{-x^2/(2\sigma^2)} dx = 1\) via substitution \(u = x^2/(2\sigma^2)\).

Sum of Independent Uniforms (Convolution)

For \(X \sim U(-1,1)\), \(Y \sim U(-1,1)\), independent, \(Z = X+Y\):

• Support: \([-2, 2]\)

• PDF: Triangular,

$$ f_Z(z) = \begin{cases} \frac{z+2}{4}, & -2 \leq z < 0 \\ \frac{2-z}{4}, & 0 \leq z \leq 2 \\ 0, & \text{otherwise} \end{cases} $$

3. Moment Generating Functions (MGF)

Definition & Properties

• MGF: \(M_X(t) = E[e^{tX}]\), for \(t\) in neighborhood of 0.

• Properties:

  1. Uniqueness: MGF uniquely determines distribution.

  2. Moments: \(E[X^n] = M_X^{(n)}(0)\) (n-th derivative at 0).

  3. Independence: If \(X,Y\) independent, \(M_{X+Y}(t) = M_X(t) M_Y(t)\).

MGF for Poisson Distribution

Given \(P(X=k) = \frac{e^{-\lambda}\lambda^k}{k!}\),

$$ M_X(t) = \sum_{k=0}^\infty e^{tk} \frac{e^{-\lambda}\lambda^k}{k!} = e^{-\lambda} \sum_{k=0}^\infty \frac{(\lambda e^t)^k}{k!} = e^{-\lambda} e^{\lambda e^t} = e^{\lambda(e^t - 1)} $$

\boxed{M_X(t) = e^{\lambda(e^t - 1)}}

[!TIP] Derive mean/variance: \(M'(t) = \lambda e^t e^{\lambda(e^t-1)}\), so \(E[X] = M'(0) = \lambda\); \(M''(t) = \lambda e^t (\lambda e^t + 1) e^{\lambda(e^t-1)}\), so \(\text{Var}(X) = M''(0) - (M'(0))^2 = \lambda\).

4. Joint Distributions

Joint CDF

For bivariate RV \((X,Y)\):

$$ F_{X,Y}(x,y) = P(X \leq x, Y \leq y) $$

Properties:

1. \(0 \leq F_{X,Y}(x,y) \leq 1\)

2. Non-decreasing in each argument.

3. \(\lim_{x,y \to \infty} F_{X,Y}(x,y) = 1\), \(\lim_{x \to -\infty} F_{X,Y}(x,y) = 0\), etc.

4. For \(a < b, c < d\):

$$ P(a < X \leq b, c < Y \leq d) = F_{X,Y}(b,d) - F_{X,Y}(a,d) - F_{X,Y}(b,c) + F_{X,Y}(a,c) $$

Marginal & Conditional:

• Marginal CDF: \(F_X(x) = F_{X,Y}(x, \infty)\)

• Conditional CDF: \(F_{Y|X}(y|x) = P(Y \leq y | X = x)\) (if continuous, use conditional PDF).

5. Random Processes

Definition & Examples

A random process is a collection of random variables \(\{X(t), t \in T\}\) indexed by time \(T\).

• Mathematical description: Each \(t\) maps to an RV \(X(t)\) with its own distribution; joint distributions describe dependencies across time.

Examples:

1. \(X(t) = A \cos \omega t\)

   • \(A\): random amplitude (e.g., uniform over \((0,\pi)\)).

   • At fixed \(t\), \(X(t)\) is scaled cosine; across \(t\), values correlated through \(A\).

2. \(X(t) = \cos(\omega t + \theta)\)

   • \(\theta\): random phase (e.g., uniform over \((0, \pi/2)\)).

   • Phase randomness makes process stochastic.

6. Stationarity

Strict-Sense Stationarity (SSS)

Definition: For all \(n\), all \(t_1, \dots, t_n\), and all \(\tau\),

$$ F_{X(t_1),\dots,X(t_n)}(x_1,\dots,x_n) = F_{X(t_1+\tau),\dots,X(t_n+\tau)}(x_1,\dots,x_n) $$

All finite-dimensional distributions invariant to time shift.

Wide-Sense Stationarity (WSS)

Conditions:

1. Constant mean: \(E[X(t)] = \mu_X\) (independent of \(t\)).

2. Autocorrelation depends only on time difference: \(R_X(t_1, t_2) = R_X(t_2 - t_1) = R_X(\tau)\).

Checking Stationarity (Past Paper Examples):

1. \(X(t) = A \cos \omega t\), \(A \sim U(0,\pi)\):

   • Mean: \(E[X(t)] = E[A] \cos \omega t = \frac{\pi}{2} \cos \omega t\) → depends on \(t\) → not WSS (hence not SSS).

   • Autocorrelation: \(R_X(t_1,t_2) = E[A^2] \cos \omega t_1 \cos \omega t_2\) → depends on absolute times → not WSS.

2. \(X(t) = \cos(\omega t + \theta)\), \(\theta \sim U(0, \pi/2)\):

   • Mean: \(E[X(t)] = E[\cos(\omega t + \theta)] = \cos \omega t \cdot E[\cos \theta] - \sin \omega t \cdot E[\sin \theta]\).

     \(E[\cos \theta] = \frac{2}{\pi}\), \(E[\sin \theta] = \frac{2}{\pi}\)? Actually compute:

     \(E[\cos \theta] = \frac{2}{\pi} \int_0^{\pi/2} \cos \theta d\theta = \frac{2}{\pi}\),

     \(E[\sin \theta] = \frac{2}{\pi} \int_0^{\pi/2} \sin \theta d\theta = \frac{2}{\pi}\).

     So \(E[X(t)] = \frac{2}{\pi} (\cos \omega t - \sin \omega t)\) → depends on \(t\) → not WSS.

   • Average power: \(P = R_X(0) = E[\cos^2(\omega t + \theta)] = E\left[\frac{1+\cos(2\omega t + 2\theta)}{2}\right] = \frac{1}{2} + \frac{1}{2} E[\cos(2\omega t + 2\theta)]\).

     \(E[\cos(2\omega t + 2\theta)] = \cos 2\omega t \cdot E[\cos 2\theta] - \sin 2\omega t \cdot E[\sin 2\theta]\).

     \(E[\cos 2\theta] = \frac{2}{\pi} \int_0^{\pi/2} \cos 2\theta d\theta = 0\), similarly \(E[\sin 2\theta]=0\).

     So \(P = \frac{1}{2}\).

[!TIP] For WSS, mean must be constant; autocorrelation must be a function of \(\tau = t_2 - t_1\) only. Always compute both.

7. Ergodicity

• Time average: \(\bar{X} = \lim_{T \to \infty} \frac{1}{2T} \int_{-T}^{T} X(t) dt\).

• Ensemble average: \(E[X(t)]\).

• Ergodic process: Time averages equal ensemble averages for all sample functions (with probability 1).

  Implications: Can estimate statistics (mean, autocorrelation) from a single, long observation.

[!TIP] WSS does not imply ergodic. Ergodicity requires additional conditions (e.g., mixing in time).

8. Correlation Functions

Autocorrelation Function \(R_X(\tau)\)

For WSS process: \(R_X(\tau) = E[X(t) X(t+\tau)]\) (independent of \(t\)). Properties for WSS:

1. \(R_X(\tau)\) is even: \(R_X(\tau) = R_X(-\tau)\).

2. \(R_X(0) = E[X^2(t)] \geq 0\) → maximum at \(\tau=0\) (by Cauchy-Schwarz).

3. Non-negative definite: For any \(n\), \(\tau_1,\dots,\tau_n\), complex \(c_i\),

$$ \sum_{i=1}^n \sum_{j=1}^n c_i \overline{c_j} R_X(\tau_i - \tau_j) \geq 0 $$

Cross-Correlation \(R_{XY}(\tau)\)

\(R_{XY}(\tau) = E[X(t) Y(t+\tau)]\) (if WSS in wide sense for both).

• Not necessarily even.

• \(R_{XY}(\tau) = R_{YX}(-\tau)\).

9. Power Spectral Density (PSD)

Definition & Properties

• PSD: \(S_X(\omega) = \mathcal{F}\{R_X(\tau)\} = \int_{-\infty}^{\infty} R_X(\tau) e^{-j\omega\tau} d\tau\).

• Properties:

  1. \(S_X(\omega) \geq 0\) (non-negative).

  2. \(S_X(\omega) = S_X(-\omega)\) (even function).

  3. \(R_X(\tau) = \frac{1}{2\pi} \int_{-\infty}^{\infty} S_X(\omega) e^{j\omega\tau} d\omega\).

  4. Total average power: \(P = R_X(0) = \frac{1}{2\pi} \int_{-\infty}^{\infty} S_X(\omega) d\omega\).

Wiener-Khinchin Theorem

\boxed{S_X(\omega) = \mathcal{F}{R_X(\tau)} \quad \text{and} \quad R_X(\tau) = \mathcal{F}^{-1}{S_X(\omega)}}

PSD and autocorrelation form a Fourier transform pair.

[!TIP] PSD interpretation: Power distribution over frequency. For deterministic periodic signals, PSD has impulses at harmonics.

10. Linear Systems with Random Inputs

LTI System Response

Input \(X(t)\), impulse response \(h(t)\), output \(Y(t) = X(t) * h(t)\).

Input-Output PSD Relation

For WSS input \(X(t)\) and stable LTI system:

$$ S_{YY}(\omega) = |H(\omega)|^2 S_{XX}(\omega) $$

where \(H(\omega) = \mathcal{F}\{h(t)\}\).

Proof Sketch:

1. \(R_{YY}(\tau) = E[Y(t) Y(t+\tau)] = E[(X*h)(t) (X*h)(t+\tau)]\).

2. Convolution property: \(Y(t) = \int_{-\infty}^{\infty} h(u) X(t-u) du\).

3. Substitute and use Fubini's theorem:

$$ R_{YY}(\tau) = \iint h(u) h(v) E[X(t-u) X(t+\tau-v)] du dv = \iint h(u) h(v) R_X(\tau - v + u) du dv $$

4. This is convolution: \(R_{YY}(\tau) = (h(\tau) * h(-\tau)) * R_X(\tau)\).

5. Fourier transform: \(S_{YY}(\omega) = H(\omega) H^*(\omega) S_{XX}(\omega) = |H(\omega)|^2 S_{XX}(\omega)\).

\boxed{S_{YY}(\omega) = |H(\omega)|^2 S_{XX}(\omega)}

[!TIP] Application: Filtering noise. If \(S_{XX}(\omega)\) is flat (white), output PSD shaped by \(|H(\omega)|^2\).

11. Cross Power Spectral Density

Definition

\(S_{XY}(\omega) = \mathcal{F}\{R_{XY}(\tau)\} = \int_{-\infty}^{\infty} R_{XY}(\tau) e^{-j\omega\tau} d\tau\).

Properties

1. Symmetry: \(S_{XY}(\omega) = S_{YX}^*(\omega)\).

2. If \(X=Y\), \(S_{XX}(\omega)\) is real and even (as PSD).

3. For real processes, \(R_{XY}(\tau) = R_{YX}(-\tau)\) → \(S_{XY}(\omega) = S_{YX}^*(\omega)\).

4. Cross PSD vs PSD: When \(X=Y\), cross PSD reduces to PSD.

12. Classification of Random Processes

[!TIP] Most practical processes are WSS but not necessarily SSS or ergodic. Check mean and autocorrelation for WSS.