Introduction to Probability and Statistics (AL-302) - Unit 4 Short Notes

UNIT 4: INTRODUCTION TO PROBABILITY AND STATISTICS

I. DESCRIPTIVE STATISTICS

A. Measures of Central Tendency

Definition: Values that represent the center or typical value of a dataset.

[!TIP] Exam Focus: Be prepared to calculate all three from given raw or grouped data. For grouped data, remember the median and mode formulas use the median class and modal class.

B. Measures of Dispersion

Definition: Quantify the spread or variability in data.

[!TIP] Common Pitfall: Use n-1 for sample variance (unbiased estimator). In exams, check if population or sample is specified.

C. Measures of Shape

1. Skewness

• Definition: Measure of asymmetry of a distribution.

• Measures:

  • Pearson's Coefficient: $$\displaystyle Sk_p = \frac{\bar{x} - \text{Mode}}{\sigma} $$ or $$\displaystyle Sk_p = 3\frac{\bar{x} - \text{Median}}{\sigma} $$

  • Based on Moments: $$\displaystyle \beta_1 = \frac{\mu_3^2}{\mu_2^3} $$ (where $$\displaystyle \mu_k $$ = $$\displaystyle k^{th} $$ central moment)

• Tests of Skewness:

  • $$\displaystyle Sk_p = 0 $$ → Symmetrical

  • $$\displaystyle Sk_p > 0 $$ → Positively skewed (Right)

  • $$\displaystyle Sk_p < 0 $$ → Negatively skewed (Left)

2. Kurtosis

• Definition: Measure of "tailedness" or peakedness compared to normal distribution.

• Measures (based on moments):

  • $$\displaystyle \beta_2 = \frac{\mu_4}{\mu_2^2} $$

  • $$\displaystyle \gamma_2 = \beta_2 - 3 $$ (Excess Kurtosis)

• Interpretation:

  • $$\displaystyle \gamma_2 = 0 $$ → Mesokurtic (Normal-like tails)

  • $$\displaystyle \gamma_2 > 0 $$ → Leptokurtic (Heavier tails, more outliers)

  • $$\displaystyle \gamma_2 < 0 $$ → Platykurtic (Lighter tails, fewer outliers)

II. PROBABILITY

A. Basic Concepts

• Sample Space (S): Set of all possible outcomes.

• Event (E): Subset of sample space.

• Axioms:

  1. $$\displaystyle P(S) = 1 $$

  2. $P(E) \ge 0$ for any event E

  3. For mutually exclusive events $$\displaystyle E_1, E_2, ... $$, $$\displaystyle P(\cup E_i) = \sum P(E_i) $$

B. Conditional Probability & Independence

• Conditional Probability: $$\displaystyle P(A|B) = \frac{P(A \cap B)}{P(B)} $$, provided $$\displaystyle P(B) > 0 $$.

• Independence: A and B are independent $$\displaystyle \iff P(A \cap B) = P(A)P(B) $$.

  • Also implies $$\displaystyle P(A|B) = P(A) $$ and $$\displaystyle P(B|A) = P(B) $$.

C. Bayes' Theorem

Statement: For mutually exclusive and exhaustive events $$\displaystyle B_1, B_2, ..., B_n $$,

$$P(B_i|A) = \frac{P(B_i) P(A|B_i)}{\sum_{j=1}^n P(B_j) P(A|B_j)}$$

Application: Revising probabilities with new evidence (e.g., medical testing, machine fault diagnosis).

D. Chebyshev's Inequality

Statement: For any random variable X with finite mean $\mu$ and variance $$\displaystyle \sigma^2 $$, and for any $$\displaystyle k > 0 $$,

$$P(|X - \mu| \ge k\sigma) \le \frac{1}{k^2}$$

Proof Sketch: Use Markov's inequality on $$\displaystyle Y = (X-\mu)^2 $$. Application: Provides a conservative bound on probability of deviation from mean, regardless of distribution.

III. RANDOM VARIABLES

A. Types

• Discrete RV: Takes countable values (e.g., # of heads, Poisson count).

• Continuous RV: Takes values in an interval (e.g., height, time).

B. Probability Functions & Distribution Functions

• PMF (Discrete): $$\displaystyle p(x) = P(X=x) $$, $$\displaystyle \sum p(x) = 1 $$.

• PDF (Continuous): $f(x) \ge 0$, $$\displaystyle \int_{-\infty}^{\infty} f(x)dx = 1 $$, $$\displaystyle P(a \le X \le b) = \int_a^b f(x)dx $$.

• CDF (Both): $$\displaystyle F(x) = P(X \le x) $$.

  • Discrete: $$\displaystyle F(x) = \sum_{t \le x} p(t) $$

  • Continuous: $$\displaystyle F(x) = \int_{-\infty}^x f(t)dt $$

C. Expectation (Mean)

Definition: $$\displaystyle E(X) = \sum x p(x) $$ (Discrete) or $$\displaystyle E(X) = \int_{-\infty}^{\infty} x f(x) dx $$ (Continuous). Properties:

1. $$\displaystyle E(aX + b) = aE(X) + b $$

2. Linearity: $$\displaystyle E(X_1 + X_2 + ... + X_n) = E(X_1) + E(X_2) + ... + E(X_n) $$ (No independence required).

3. If $X \ge 0$, then $E(X) \ge 0$.

D. Variance

Definition: $$\displaystyle V(X) = E[(X - \mu)^2] = E(X^2) - [E(X)]^2 $$. Properties:

1. $$\displaystyle V(aX + b) = a^2 V(X) $$

2. $$\displaystyle V(X + Y) = V(X) + V(Y) $$ if X and Y are independent.

3. $V(X) \ge 0$.

E. Moments

• $$\displaystyle r^{th} $$ Raw Moment: $$\displaystyle \mu'_r = E(X^r) $$

• $$\displaystyle r^{th} $$ Central Moment: $$\displaystyle \mu_r = E[(X - \mu)^r] $$

  • $$\displaystyle \mu_1 = 0 $$, $$\displaystyle \mu_2 = \sigma^2 $$, $$\displaystyle \mu_3 $$ relates to skewness, $$\displaystyle \mu_4 $$ relates to kurtosis.

• Relationship: $$\displaystyle \mu_2 = \mu'_2 - (\mu'_1)^2 $$, etc.

F. Moment Generating Function (MGF)

Definition: $$\displaystyle M_X(t) = E(e^{tX}) $$, for $t$ in some interval around 0. Properties:

1. $$\displaystyle M_X(0) = 1 $$.

2. $$\displaystyle M'_X(0) = E(X) $$, $$\displaystyle M''_X(0) = E(X^2) $$.

3. $$\displaystyle M_{aX+b}(t) = e^{bt} M_X(at) $$.

4. Uniqueness: If MGF exists for two RVs and they are equal in an interval around 0, the distributions are identical.

5. Independence: If X, Y independent, $$\displaystyle M_{X+Y}(t) = M_X(t) M_Y(t) $$.

G. Computing Measures from PMF/PDF

1. Mean: $$\displaystyle E(X) = \sum x p(x) $$ or $$\displaystyle \int x f(x) dx $$.

2. Median: Solve $$\displaystyle F(m) = 0.5 $$.

3. Mode: Value $x$ where $p(x)$ or $f(x)$ is maximum.

4. Variance: Use $$\displaystyle V(X) = E(X^2) - [E(X)]^2 $$.

5. Mean Deviation (from mean): $E(|X - \mu|)$.

IV. PROBABILITY DISTRIBUTIONS

A. Discrete Distributions

1. Binomial Distribution $B(n, p)$

• PMF: $$\displaystyle P(X=x) = \binom{n}{x} p^x (1-p)^{n-x} $$, $$\displaystyle x=0,1,...,n $$.

• Mean: $$\displaystyle \mu = np $$

• Variance: $$\displaystyle \sigma^2 = np(1-p) $$

• MGF: $$\displaystyle M_X(t) = (1-p + pe^t)^n $$

• Limiting Case: As $n \to \infty$, $p \to 0$, $$\displaystyle np = \lambda $$ (constant) → Poisson($\lambda$).

2. Poisson Distribution $P(\lambda)$

• PMF: $$\displaystyle P(X=x) = \frac{e^{-\lambda} \lambda^x}{x!} $$, $$\displaystyle x=0,1,2,... $$

• Mean & Variance: $$\displaystyle \mu = \sigma^2 = \lambda $$

• MGF: $$\displaystyle M_X(t) = e^{\lambda(e^t - 1)} $$

• Applications: # of events in fixed interval (time/area), rare events.

3. Hypergeometric Distribution $HG(N, K, n)$

• PMF: $$\displaystyle P(X=x) = \frac{\binom{K}{x} \binom{N-K}{n-x}}{\binom{N}{n}} $$

  • $N$: Population size, $K$: # of successes in pop., $n$: sample size.

• Mean: $$\displaystyle \mu = n \frac{K}{N} $$

• Variance: $$\displaystyle \sigma^2 = n \frac{K}{N} \frac{N-K}{N} \frac{N-n}{N-1} $$

B. Continuous Distributions

1. Normal Distribution $$\displaystyle N(\mu, \sigma^2) $$

• PDF: $$\displaystyle f(x) = \frac{1}{\sigma\sqrt{2\pi}} e^{-\frac{1}{2}\left(\frac{x-\mu}{\sigma}\right)^2} $$, $$\displaystyle -\infty < x < \infty $$.

• Properties:

  • Bell-shaped, symmetric about $\mu$.

  • Mean = Median = Mode = $\mu$.

  • Points of inflection at $\mu \pm \sigma$.

  • $\mu + 1.96\sigma$ covers 95% area.

• Standardization: $$\displaystyle Z = \frac{X-\mu}{\sigma} \sim N(0,1) $$.

• MGF: $$\displaystyle M_X(t) = e^{\mu t + \frac{1}{2}\sigma^2 t^2} $$.

• Calculations: Use Z-tables.

2. Exponential Distribution $Exp(\theta)$ (or $$\displaystyle \lambda=1/\theta $$)

• PDF: $$\displaystyle f(x) = \frac{1}{\theta} e^{-x/\theta} $$, $x \ge 0$ (scale $\theta$).

  • Or $$\displaystyle f(x) = \lambda e^{-\lambda x} $$, $x \ge 0$ (rate $\lambda$).

• Mean: $$\displaystyle \mu = \theta = 1/\lambda $$

• Variance: $$\displaystyle \sigma^2 = \theta^2 = 1/\lambda^2 $$

• MGF: $$\displaystyle M_X(t) = \frac{1}{1 - \theta t} $$, $$\displaystyle t < 1/\theta $$.

• Memoryless Property: $$\displaystyle P(X > s+t | X > s) = P(X > t) $$.

3. Gamma Distribution $Gamma(\alpha, \beta)$ (shape $\alpha$, scale $\beta$)

• PDF: $$\displaystyle f(x) = \frac{1}{\beta^\alpha \Gamma(\alpha)} x^{\alpha-1} e^{-x/\beta} $$, $$\displaystyle x > 0 $$.

• Mean: $$\displaystyle \mu = \alpha\beta $$

• Variance: $$\displaystyle \sigma^2 = \alpha\beta^2 $$

• Note: Exponential is Gamma with $$\displaystyle \alpha=1 $$.

C. Chi-square Distribution $$\displaystyle \chi^2_k $$

• Definition: Sum of squares of $k$ independent standard normal variables: $$\displaystyle X = \sum_{i=1}^k Z_i^2 \sim \chi^2_k $$.

• PDF: $$\displaystyle f(x) = \frac{1}{2^{k/2} \Gamma(k/2)} x^{(k/2)-1} e^{-x/2} $$, $$\displaystyle x > 0 $$.

• Properties:

  • Mean = $k$, Variance = $2k$.

  • Additivity: If $$\displaystyle X_1 \sim \chi^2_{k_1} $$, $$\displaystyle X_2 \sim \chi^2_{k_2} $$ independent, then $$\displaystyle X_1+X_2 \sim \chi^2_{k_1+k_2} $$.

  • Related to Normal: $$\displaystyle Z^2 \sim \chi^2_1 $$.

  • Used in inference (tests for variance, goodness-of-fit, independence).

V. BIVARIATE ANALYSIS

A. Bivariate Distributions

• Joint PMF/PDF: $p(x,y)$ or $f(x,y)$ gives probability for pair (X,Y).

• Marginal Distributions:

  • Discrete: $$\displaystyle p_X(x) = \sum_y p(x,y) $$

  • Continuous: $$\displaystyle f_X(x) = \int_{-\infty}^{\infty} f(x,y) dy $$

• Conditional Distributions:

  • Discrete: $$\displaystyle p_{Y|X}(y|x) = \frac{p(x,y)}{p_X(x)} $$

  • Continuous: $$\displaystyle f_{Y|X}(y|x) = \frac{f(x,y)}{f_X(x)} $$

• Independence: X and Y independent $$\displaystyle \iff f(x,y) = f_X(x) f_Y(y) $$ for all x,y (or equivalent for discrete).

B. Bivariate Normal Distribution

• Joint PDF:

$$f(x,y) = \frac{1}{2\pi \sigma_X \sigma_Y \sqrt{1-\rho^2}} \exp\left( -\frac{1}{2(1-\rho^2)} \left[ \frac{(x-\mu_X)^2}{\sigma_X^2} - 2\rho\frac{(x-\mu_X)(y-\mu_Y)}{\sigma_X\sigma_Y} + \frac{(y-\mu_Y)^2}{\sigma_Y^2} \right] \right)$$

• Properties:

  • Marginals: $$\displaystyle X \sim N(\mu_X, \sigma_X^2) $$, $$\displaystyle Y \sim N(\mu_Y, \sigma_Y^2) $$.

  • Conditional distributions are normal.

  • $\rho$ is Pearson's correlation coefficient.

C. Correlation

1. Pearson's Correlation Coefficient ($\rho$ or $r$)

• Definition: Measure of linear association.

• Formula (Sample $r$):

$$r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}} = \frac{S_{xy}}{S_x S_y}$$

• Properties:

  • $-1 \le r \le 1$.

  • $$\displaystyle r = 1 $$: Perfect positive linear.

  • $$\displaystyle r = -1 $$: Perfect negative linear.

  • $$\displaystyle r = 0 $$: No linear correlation (may have non-linear).

  • Independent $$\displaystyle \implies \rho = 0 $$, but $$\displaystyle \rho = 0 \not\implies $$ independent.

2. Spearman's Rank Correlation Coefficient ($$\displaystyle \rho_s $$ or $$\displaystyle r_s $$)

• Definition: Non-parametric measure based on ranks.

• Calculation:

  1. Assign ranks to X and Y values (average for ties).

  2. $$\displaystyle r_s = 1 - \frac{6 \sum d_i^2}{n(n^2-1)} $$, where $$\displaystyle d_i = \text{rank}(x_i) - \text{rank}(y_i) $$.

  • For no ties. With ties, use Pearson's formula on ranks.

• Proof of Max Value 1: Based on Cauchy-Schwarz inequality applied to rank differences.

3. Partial & Multiple Correlation

• Partial Correlation ($$\displaystyle r_{12.3} $$): Correlation between $$\displaystyle X_1 $$ and $$\displaystyle X_2 $$ holding $$\displaystyle X_3 $$ constant.

• Multiple Correlation ($$\displaystyle R_{1.23...k} $$): Correlation between $$\displaystyle X_1 $$ and its best linear prediction from $$\displaystyle X_2, ..., X_k $$.

• Application: Isolating relationship between variables while controlling for others.

4. Correlation vs. Regression

D. Regression

1. Simple Linear Regression Model: $$\displaystyle Y = \beta_0 + \beta_1 X + \varepsilon $$, where $\varepsilon$ is error with $$\displaystyle E(\varepsilon)=0 $$. 2. Regression Coefficients:

• $$\displaystyle b_1 = \frac{S_{xy}}{S_x^2} $$ (slope)

• $$\displaystyle b_0 = \bar{y} - b_1 \bar{x} $$ (intercept)

• Properties: $$\displaystyle \sum e_i = 0 $$, $$\displaystyle \sum x_i e_i = 0 $$, line passes through $(\bar{x}, \bar{y})$. 3. Method of Least Squares: Minimizes $$\displaystyle \sum e_i^2 = \sum (y_i - \hat{y}_i)^2 $$. 4. Fitting Straight Line ($$\displaystyle y = a + bx $$):

  • Normal equations:

$$\sum y = na + b\sum x$$

$$\sum xy = a\sum x + b\sum x^2$$

*   Solve for $a, b$.

5. Fitting Second Degree Parabola ($$\displaystyle y = a + bx + cx^2 $$):

*   Normal equations:

$$\sum y = na + b\sum x + c\sum x^2$$

$$\sum xy = a\sum x + b\sum x^2 + c\sum x^3$$

$$\sum x^2 y = a\sum x^2 + b\sum x^3 + c\sum x^4$$

*   Solve simultaneous equations for $a, b, c$.

6. Fitting Other Curves (e.g., $$\displaystyle y = \alpha x + \beta x^2 $$):

*   Transform to linear in parameters if possible (here, it is already linear in $\alpha, \beta$).

*   Normal equations:

$$\sum xy = \alpha\sum x^2 + \beta\sum x^3$$

$$\sum x^2 y = \alpha\sum x^3 + \beta\sum x^4$$

*   Solve for $\alpha, \beta$.

VI. INFERENTIAL STATISTICS

A. Sampling & Standard Error

• Standard Error (SE): Standard deviation of a sampling distribution.

• SE of Mean: $$\displaystyle \sigma_{\bar{x}} = \frac{\sigma}{\sqrt{n}} $$ (if $\sigma$ known), or $$\displaystyle s_{\bar{x}} = \frac{s}{\sqrt{n}} $$.

• SE of Proportion: $$\displaystyle \sigma_p = \sqrt{\frac{p(1-p)}{n}} $$.

B. Estimation

1. Point Estimation: Single value estimate (e.g., $\bar{x}$ for $\mu$). 2. Confidence Intervals:

• For $\mu$ (σ known): $$\displaystyle \bar{x} \pm z_{\alpha/2} \frac{\sigma}{\sqrt{n}} $$

• For $\mu$ (σ unknown, n small): $$\displaystyle \bar{x} \pm t_{\alpha/2, n-1} \frac{s}{\sqrt{n}} $$

• For Proportion $p$: $$\displaystyle \hat{p} \pm z_{\alpha/2} \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} $$

C. Hypothesis Testing

1. Basic Concepts

• Null Hypothesis ($$\displaystyle H_0 $$): Status quo, to be tested.

• Alternative Hypothesis ($$\displaystyle H_1 $$): What we suspect.

• Errors:

  • Type I: Reject $$\displaystyle H_0 $$ when true ($\alpha$ = significance level).

  • Type II: Fail to reject $$\displaystyle H_0 $$ when false ($\beta$).

• p-value: Probability of observing test statistic as extreme as observed, assuming $$\displaystyle H_0 $$ true. Reject $$\displaystyle H_0 $$ if p-value $$\displaystyle < \alpha $$.

• Critical Region: Set of values leading to rejection of $$\displaystyle H_0 $$.

2. Tests for Means

• z-test (large sample, σ known): $$\displaystyle z = \frac{\bar{x} - \mu_0}{\sigma/\sqrt{n}} \sim N(0,1) $$ under $$\displaystyle H_0 $$.

• t-test (small sample, σ unknown): $$\displaystyle t = \frac{\bar{x} - \mu_0}{s/\sqrt{n}} \sim t_{n-1} $$ under $$\displaystyle H_0 $$.

3. Tests for Proportions (z-test): $$\displaystyle z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} $$, use normal approximation if $$\displaystyle np_0 \ge 5 $$ and $$\displaystyle n(1-p_0) \ge 5 $$.

4. Test for Difference of Means (large samples): $$\displaystyle z = \frac{(\bar{x}_1 - \bar{x}_2) - (\mu_1 - \mu_2)_0}{\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}} $$

• If $$\displaystyle \sigma_1, \sigma_2 $$ unknown, use $$\displaystyle s_1, s_2 $$.

5. Chi-square ($$\displaystyle \chi^2 $$) Tests a. Goodness of Fit:

• Conditions: Expected frequencies $$\displaystyle E_i \ge 5 $$ (at least 80% should satisfy this), data from random sample.

• Procedure:

  1. $$\displaystyle H_0 $$: Data fits specified distribution.

  2. Compute $$\displaystyle E_i $$ based on $$\displaystyle H_0 $$.

  3. $$\displaystyle \chi^2_{cal} = \sum \frac{(O_i - E_i)^2}{E_i} $$

  4. df = (number of categories) - 1 - (number of parameters estimated).

  5. Reject $$\displaystyle H_0 $$ if $$\displaystyle \chi^2_{cal} > \chi^2_{\alpha, df} $$. b. Test for Independence (Contingency Table):

• Procedure:

  1. $$\displaystyle H_0 $$: Two categorical variables are independent.

  2. Compute $$\displaystyle E_{ij} = \frac{(\text{row } i \text{ total}) \times (\text{col } j \text{ total})}{\text{grand total}} $$.

  3. $$\displaystyle \chi^2_{cal} = \sum \sum \frac{(O_{ij} - E_{ij})^2}{E_{ij}} $$

  4. df = (rows - 1)(columns - 1).

  5. Reject $$\displaystyle H_0 $$ if $$\displaystyle \chi^2_{cal} > \chi^2_{\alpha, df} $$.

6. F-test for Equality of Variances

• Test Statistic: $$\displaystyle F = \frac{s_1^2}{s_2^2} $$ (place larger variance in numerator).

• Distribution: $$\displaystyle F \sim F_{n_1-1, n_2-1} $$ under $$\displaystyle H_0: \sigma_1^2 = \sigma_2^2 $$.

• Application: Compare two variances, check homogeneity before t-test for means.

D. Special Applications

1. Poisson Approximation to Binomial: When $n$ large, $p$ small, $$\displaystyle \lambda = np $$ moderate. 2. Normal Approximation to Binomial: When $np \ge 5$ and $n(1-p) \ge 5$, $X \sim B(n,p) \approx N(np, np(1-p))$.

*   Apply continuity correction: $P(X \le x) \approx P(Y \le x + 0.5)$ where $$\displaystyle Y \sim N(\mu, \sigma^2) $$.