Introduction
1. Distribution and Density Functions and Their Properties
Binomial Distribution
[!IMPORTANT]
Definition: A random variable $X$ is said to follow a binomial distribution with parameters $(n, p)$ if it counts the number of successes in $n$ independent Bernoulli trials, each with success probability $p$.
Parameters:
-
$n$ = number of trials (integer, $n \geq 1$)
-
$p$ = probability of success in a single trial ($0 < p < 1$)
Probability Mass Function (PMF) – Derivation
The probability of exactly $k$ successes in $n$ trials is:
\[ P(X = k) = \binom{n}{k} p^k (1-p)^{n-k} \]
where $k = 0, 1, 2, ..., n$
[!TIP]
Remember: Binomial PMF is the "n choose k" arrangement multiplied by probability of each outcome.
Derivation Steps:
-
Each trial: probability of success = $p$; failure = $1-p$
-
In $n$ trials, number of arrangements for $k$ successes: $\binom{n}{k}$
-
Probability of any such arrangement: $p^k(1-p)^{n-k}$
-
Combine:
\[ P(X = k) = \binom{n}{k} p^k (1-p)^{n-k} \]
\[ \boxed{P(X = k) = \binom{n}{k} p^k (1-p)^{n-k}} \]
Properties
-
Mean: $E[X] = np$
-
Variance: $Var(X) = np(1-p)$
-
Symmetry: Distribution is symmetric if $p=0.5$, skewed otherwise.
Worked Example
A coin is tossed $5$ times. What is the probability of exactly $2$ heads? $(p = 0.5)$
\[ P(X=2) = \binom{5}{2} (0.5)^2 (0.5)^{3} = 10 \times 0.25 \times 0.125 = 0.3125 \]
\[ \boxed{P(X=2) = 0.3125} \]
Poisson Distribution
[!IMPORTANT]
Definition: The Poisson distribution models the number of rare events occuring in a fixed interval, given the average rate $\lambda$.
Parameter:
- $\lambda$ = average rate of occurrence in an interval ($\lambda > 0$)
PMF – Derivation
Probability of $k$ events in interval:
\[ P(X = k) = \frac{e^{-\lambda} \lambda^k}{k!}, \quad k = 0, 1, 2, ... \]
Derivation from Binomial:
If $n \to \infty$, $p \to 0$ such that $np = \lambda$ (rare-event limit) in the binomial PMF:
\[ P(X = k) = \lim_{n \to \infty} \binom{n}{k} p^k (1-p)^{n-k} = \frac{e^{-\lambda} \lambda^k}{k!} \]
\[ \boxed{P(X = k) = \frac{e^{-\lambda} \lambda^k}{k!}} \]
Properties
-
Mean: $E[X] = \lambda$
-
Variance: $Var(X) = \lambda$
-
Additivity: If $X_1 \sim Poisson(\lambda_1)$ and $X_2 \sim Poisson(\lambda_2)$, then $X_1+X_2 \sim Poisson(\lambda_1+\lambda_2)$.
Worked Example
If $\lambda = 4$, find the probability that $X=2$.
\[ P(2) = \frac{e^{-4} 4^2}{2!} = \frac{e^{-4} \times 16}{2} = 8e^{-4} \]
Compute $e^{-4} \approx 0.0183$:
\[ P(2) \approx 8 \times 0.0183 = 0.146 \]
\[ \boxed{P(X = 2) = 0.146} \]
[!TIP]
Use Poisson when $n$ is large, $p$ is small, and $np$ is moderate.
Uniform Distribution
[!IMPORTANT]
Definition (Discrete): A discrete random variable $X$ is uniformly distributed if every outcome is equally likely.
Definition (Continuous): A continuous random variable $X$ is uniformly distributed on $[a,b]$ if its pdf is constant in $[a, b]$.
(a) Discrete Uniform Distribution
If $X$ takes values $x_1, x_2, ..., x_n$:
\[ P(X = x_i) = \frac{1}{n} \]
(b) Continuous Uniform Distribution
PDF Derivation (on $[a, b]$):
Since total probability = 1:
Let $f(x) = k$ for $a \leq x \leq b$
Integrate:
\[ \int_a^b k\,dx = 1 \implies k(b - a) = 1 \implies k = \frac{1}{b - a} \]
So
\[ f(x) = \begin{cases} \frac{1}{b-a}, & a \le x \le b \\ 0, & \text{otherwise} \end{cases} \]
\[ \boxed{f(x) = \frac{1}{b-a}, \quad a \le x \le b} \]
Properties
-
Mean: $E[X] = \frac{a + b}{2}$
-
Variance: $Var(X) = \frac{(b - a)^2}{12}$
Worked Example
For $X$ uniform in $[2, 6]$:
-
$E[X] = \frac{2 + 6}{2} = 4$
-
$Var(X) = \frac{(6 - 2)^2}{12} = \frac{16}{12} = 1.333$
Gaussian (Normal) Distribution
[!IMPORTANT]
Definition: A random variable $X$ is said to follow a Gaussian (normal) distribution with mean $\mu$ and variance $\sigma^2$ if its pdf is:
\[ f(x) = \frac{1}{\sqrt{2\pi}\sigma}\ \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right), \quad -\infty < x < \infty \]
PDF – Derivation
- The bell-shaped curve satisfies normalization:
\[ \int_{-\infty}^{\infty} f(x) dx = 1 \]
$\sigma$ controls spread, $\mu$ the center.
\[ \boxed{ f(x) = \frac{1}{\sqrt{2\pi}\sigma} e^{-\frac{(x-\mu)^2}{2\sigma^2}} } \]
[!TIP]
Always quote the standard form in the exam and define all parameters.
Properties
-
Mean: $E[X] = \mu$
-
Variance: $Var(X) = \sigma^2$
-
Symmetry: Symmetric about $x = \mu$
-
68-95-99.7 Rule: $\approx 68\%$ of values within $1\sigma$, $95\%$ within $2\sigma$, $99.7\%$ within $3\sigma$
Worked Example
A Gaussian random variable with mean $1$, variance $4$: Find $P(1 < X < 2)$.
Standardize:
\[ Z = \frac{X-\mu}{\sigma} = \frac{X-1}{2} \]
So for $X=1$, $z_1 = 0$. For $X=2$, $z_2 = \frac{1}{2} = 0.5$.
So,
\[ P(1 < X < 2) = P(0 < Z < 0.5) \]
From normal tables, $P(0 < Z < 0.5) \approx 0.1915$
\[ \boxed{P(1 < X < 2) = 0.1915} \]
Exponential Distribution
[!IMPORTANT]
Definition: A random variable $X$ follows an exponential distribution with rate parameter $\lambda$ if:
\[ f(x) = \begin{cases} \lambda e^{-\lambda x}, & x \geq 0 \\ 0, & x < 0 \end{cases} \]
Parameter: $\lambda > 0$
PDF – Derivation
Require: $f(x) \geq 0$ and integrates to 1:
\[ \int_0^{\infty} \lambda e^{-\lambda x} dx = \left[- e^{-\lambda x}\right]_0^{\infty} = 1 \]
So
\[ \boxed{ f(x) = \lambda e^{-\lambda x}, \quad x \ge 0 } \]
Properties
-
Mean: $E[X] = \frac{1}{\lambda}$
-
Variance: $Var(X) = \frac{1}{\lambda^2}$
-
Memoryless: $P(X > s+t | X > s) = P(X > t)$
Worked Example
If $\lambda = 2$, find $P(X < 1)$:
\[ P(X < 1) = \int_0^1 2e^{-2x} dx = \left[ -e^{-2x}\right]_0^1 = -(e^{-2}) + (1) = 1 - e^{-2} \approx 0.8647 \]
\[ \boxed{P(X < 1) = 0.8647} \]
Rayleigh Distribution
[!IMPORTANT]
Definition: A random variable $X$ has a Rayleigh distribution with parameter $\sigma$ if:
\[ f(x) = \begin{cases} \frac{x}{\sigma^2} e^{-x^2/2\sigma^2}, & x \ge 0 \\ 0, & x < 0 \end{cases} \]
Parameter: $\sigma > 0$
PDF – Derivation
Normalization requirement:
\[ \int_0^{\infty} \frac{x}{\sigma^2} e^{-x^2/2\sigma^2} dx = 1 \]
Let $y = x^2 / (2\sigma^2) \implies x = \sigma \sqrt{2y}, \, dx = \sigma \frac{1}{\sqrt{2y}} dy$
But directly,
\[ \int_0^{\infty} \frac{x}{\sigma^2} e^{-x^2/2\sigma^2} dx \]
Let $t = x^2/(2\sigma^2) \implies x = \sigma\sqrt{2t}$, $dx = \sigma \frac{1}{\sqrt{2t}} dt$
Put:
\[ = \int_0^{\infty} e^{-t} dt = 1 \]
So, it is valid as a PDF.
Properties
-
Mean: $E[X] = \sigma \sqrt{\frac{\pi}{2}}$
-
Variance: $Var(X) = \left(2 - \frac{\pi}{2}\right)\sigma^2$
Worked Example (Validity)
Show that $f(x)$ integrates to 1:
\[ \int_0^{\infty} \frac{x}{\sigma^2} e^{-x^2/2\sigma^2} dx = 1 \quad \text{[proved above]} \]
2. Conditional Distribution and Conditional Density
[!IMPORTANT]
Definition (Discrete): Conditional probability mass function of $X$ given $Y=y$ is
\[ > P_{X|Y}(x|y) = \frac{P_{X,Y}(x, y)}{P_Y(y)} > \]
if $P_Y(y) > 0$.
Definition (Continuous): Conditional density of $X$ given $Y=y$ is
\[ > f_{X|Y}(x|y) = \frac{f_{X,Y}(x, y)}{f_Y(y)} > \]
if $f_Y(y) > 0$.
Properties – Derivation
-
Non-negativity: $P_{X|Y}(x|y) \ge 0$, $f_{X|Y}(x|y) \ge 0$
-
Total Probability: $\sum_x P_{X|Y}(x|y) = 1$; $\int f_{X|Y}(x|y)dx = 1$
Proof (continuous):
\[ \int_{-\infty}^{\infty} f_{X|Y}(x|y) dx = \int_{-\infty}^{\infty} \frac{f_{X,Y}(x, y)}{f_Y(y)} dx = \frac{1}{f_Y(y)} \int_{-\infty}^{\infty} f_{X,Y}(x, y) dx = \frac{f_Y(y)}{f_Y(y)} = 1 \]
Worked Example
Suppose $P_{X,Y}(x, y)$ for $x, y \in \{0, 1\}$:
$P_{X,Y}(0,0) = 0.1,\, P_{X,Y}(0,1) = 0.2,\, P_{X,Y}(1,0) = 0.3,\, P_{X,Y}(1,1) = 0.4.$
Find $P_{X|Y}(1|1)$.
First, $P_Y(1) = P_{X,Y}(0,1) + P_{X,Y}(1,1) = 0.2 + 0.4 = 0.6$
$P_{X|Y}(1|1) = \frac{P_{X,Y}(1,1)}{P_Y(1)} = \frac{0.4}{0.6} = 0.6667$
3. Methods of Defining Conditional Event
[!IMPORTANT]
Definition: The conditional probability of $A$ given $B$ is $P(A|B) = \frac{P(A \cap B)}{P(B)}$, provided $P(B) > 0$.
Methods:
-
Verbal: "Probability that event $A$ occurs, given $B$ has occurred."
-
Set Notation: $A|B = A \cap B$
-
Probability Approach: $P(A|B) = \frac{P(A \cap B)}{P(B)}$
4. Conditional Density Properties
[!IMPORTANT]
Properties:
- Non-negativity: $f_{X|Y}(x|y) \ge 0$
- Unit integral: $\int f_{X|Y}(x|y) dx = 1$
Derivation:
As given in Section 2.
Example
If $f_{X,Y}(x, y) = 2$ for $0 < x < y < 1$, find $f_{X|Y}(x|y)$.
First, marginals:
$f_Y(y) = \int_0^y 2 dx = 2y$ for $0<y<1$
\[ f_{X|Y}(x|y) = \frac{2}{2y} = \frac{1}{y}, \quad 0 < x < y < 1 \]
5. Expectations – Introduction and Expected Value of a Random Variable
[!IMPORTANT]
Definition: Expected value of random variable $X$:
- Discrete: $E[X] = \sum_x x\,P_X(x)$
- Continuous: $E[X] = \int_{-\infty}^{\infty} x f_X(x) dx$
Calculation (Numerical)
If $X$ is discrete with $P_X(0)=0.2,\,P_X(1)=0.5,\,P_X(2)=0.3$
\[ E[X] = 0*0.2 + 1*0.5 + 2*0.3 = 0 + 0.5 + 0.6 = 1.1 \]
If $f_X(x) = \frac{1}{2},\, 0 \le x \le 2$ (uniform):
\[ E[X] = \int_0^2 x \cdot \frac{1}{2} dx = \frac{1}{2} \left[\frac{x^2}{2}\right]_0^2 = \frac{1}{2} \times 2 = 1 \]
6. Function of a Random Variable
[!IMPORTANT]
Definition: If $Y = g(X)$, the PDF of $Y$ is determined from the PDF of $X$ by transformation.
Method – Determining Distribution of $Y = g(X)$
For monotonic $g$:
If $X$ is continuous with PDF $f_X(x)$:
\[ f_Y(y) = f_X(g^{-1}(y)) \left| \frac{d}{dy}g^{-1}(y) \right| \]
Example (Numerical)
Let $X$ uniform on $[0,1]$, $Y = 2X$, find PDF of $Y$.
$g(x) = 2x \implies x = y/2$, $f_X(x) = 1$ for $0 < x < 1$
$y \in [0,2]$
\[ f_Y(y) = f_X \left( \frac{y}{2} \right) \cdot \left| \frac{d}{dy} \left( \frac{y}{2} \right) \right| = 1 \cdot \frac{1}{2},\ 0 < y < 2 \]
So
\[ \boxed{ f_Y(y) = \frac{1}{2}, \quad 0<y<2 } \]
7. Moments About the Origin
[!IMPORTANT]
Definition: $r$-th moment about the origin:
- Discrete: $\mu_r' = E[X^r] = \sum_x x^r P_X(x)$
- Continuous: $\mu_r' = E[X^r] = \int_{-\infty}^\infty x^r f_X(x) dx$
Calculation
For $X$ with $P_X(0)=0.5, P_X(1)=0.5$:
\[ \mu_1' = E[X] = 0*0.5 + 1*0.5 = 0.5 \]
\[ \mu_2' = E[X^2] = 0^2*0.5 + 1^2*0.5 = 0 + 0.5 = 0.5 \]
8. Central Moments, Variance and Skew
[!IMPORTANT]
Definitions:
- Central moment (about mean): $\mu_r = E[(X - E[X])^r]$
- Variance: $\mu_2 = \text{Var}(X)$
- Skewness: $\gamma_1 = \frac{\mu_3}{\mu_2^{3/2}}$
Relationship to Raw Moments – Derivation
\[ \mu_2 = E[X^2] - (E[X])^2 \]
General formula:
\[ \mu_2' = E[X^2], \quad \text{so} \quad \mu_2 = \mu_2' - (\mu_1')^2 \]
Example
For $X$ as in Section 7:
\[ \mu_1' = 0.5 \]
\[ \mu_2' = 0.5 \]
\[ \mu_2 = 0.5 - (0.5)^2 = 0.5 - 0.25 = 0.25 \]
- Variance: $0.25$
9. Chebyshev’s Inequality
[!IMPORTANT]
Statement: For any random variable $X$ with mean $\mu$ and variance $\sigma^2$,
\[ > P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2} > \]
for $k > 0$
Proof – Derivation
By Markov's inequality:
\[ P(|X - \mu| \ge k\sigma) = P\left( (X - \mu)^2 \ge k^2 \sigma^2 \right) \]
\[ \le \frac{E[(X - \mu)^2]}{k^2 \sigma^2} = \frac{\sigma^2}{k^2 \sigma^2} = \frac{1}{k^2} \]
\[ \boxed{P(|X - \mu| \geq k\sigma) \leq \frac{1}{k^2}} \]
Application Example
Given $E[X]=50,\,Var(X)=25$, find $P(|X-50| \ge 10)$.
Here, $\sigma = 5$, $k = \frac{10}{5} = 2$:
\[ P(|X-50| \ge 10) \le \frac{1}{2^2} = 0.25 \]
\[ \boxed{P(|X-50| \ge 10) \le 0.25} \]
10. Characteristic Function
[!IMPORTANT]
Definition: The characteristic function of a random variable $X$ is
\[ > \phi_X(w) = E\left[e^{iwX}\right] > \]
- Discrete: $\phi_X(w) = \sum_x e^{iwx}P_X(x)$
- Continuous: $\phi_X(w) = \int_{-\infty}^{\infty} e^{iwx} f_X(x) dx$
Properties and Uses
-
Unique for each distribution.
-
Used to find moments; if $\phi_X(w)$ is differentiable:
\[ E[X^k] = \left. \frac{1}{i^k} \frac{d^k \phi_X(w)}{dw^k} \right|_{w=0} \]
11. Moment Generating Function (MGF)
[!IMPORTANT]
Definition: The moment generating function (MGF) of $X$ is:
\[ > M_X(t) = E\left[e^{tX}\right] > \]
- Discrete: $M_X(t) = \sum_x e^{t x} P_X(x)$
- Continuous: $M_X(t) = \int_{-\infty}^\infty e^{t x} f_X(x) dx$
Properties – Derivation
-
$n$-th moment $E[X^n] = M_X^{(n)}(0)$ (the $n$th derivative at $t=0$)
-
$M_{aX+b}(t) = e^{bt} M_X(at)$
Poisson Distribution MGF – Derivation and Numerical
$X \sim \text{Poisson}(\lambda)$
\[ M_X(t) = E[e^{tX}] = \sum_{x=0}^{\infty} e^{t x} \frac{e^{-\lambda} \lambda^x}{x!} = e^{-\lambda} \sum_{x=0}^\infty \left( \lambda e^{t} \right)^x \frac{1}{x!} = e^{-\lambda} e^{\lambda e^{t}} = e^{\lambda (e^{t} - 1)} \]
\[ \boxed{M_X(t) = e^{\lambda (e^{t} - 1)}} \]
[!TIP]
For exam, always quote and differentiate the MGF to get moments.
12. Monotonic Transformations for a Continuous Random Variable
[!IMPORTANT]
Definition: If $Y = g(X)$ is monotonic and $X$ continuous, then
\[ > f_Y(y) = f_X(g^{-1}(y)) \left| \frac{d}{dy}g^{-1}(y) \right| > \]
Method – Derivation
-
Find inverse $g^{-1}(y)$.
-
Differentiate $g^{-1}(y)$ wrt $y$.
-
Substitute in $f_X$.
Example
Let $X$ uniform on $[0,1]$, $Y = X^2$. $g(x) = x^2 \implies x = \sqrt{y}$, $0 \le y \le 1$.
\[ f_Y(y) = f_X(\sqrt{y}) \left| \frac{d}{dy} \sqrt{y} \right| = 1 \times \frac{1}{2 \sqrt{y}}, \quad 0<y<1 \]
\[ \boxed{f_Y(y) = \frac{1}{2\sqrt{y}},\; 0<y<1} \]
13. Non-monotonic Transformations of Continuous Random Variable
[!IMPORTANT]
Process: If $g(x)$ is not monotonic, for a value $y$, find all $x_i$ such that $g(x_i) = y$.
Then,
\[ > f_Y(y) = \sum_{\text{all } x_i} \frac{f_X(x_i)}{|g'(x_i)|} > \]
Example
If $X$ uniform on $[-1, 1]$, $Y = X^2$, find $f_Y(y)$.
$g(x) = x^2$, so $x = \pm \sqrt{y}$, $0 < y < 1$
\[ f_X(x) = \frac{1}{2} \quad \text{for}~ -1 < x < 1 \]
Derivative $|g'(x)|$ at $x = \pm \sqrt{y}$ is $|2\sqrt{y}|$
So:
\[ f_Y(y) = \frac{1}{2} \times \frac{1}{2\sqrt{y}} + \frac{1}{2} \times \frac{1}{2\sqrt{y}} = \frac{1}{2\sqrt{y}}, \quad 0<y<1 \]
14. Transformation of a Discrete Random Variable
[!IMPORTANT]
Method: For $Y = g(X)$, compute $P(Y = y)$ by summing $P_X(x)$ over all $x$ such that $g(x) = y$.
Example
If $X$ takes $0, 1, 2$ with $P(0)=0.2$, $P(1)=0.5$, $P(2)=0.3$; let $Y=2X$.
-
$Y=0$ if $X=0$: $P_Y(0) = 0.2$
-
$Y=2$ if $X=1$: $P_Y(2) = 0.5$
-
$Y=4$ if $X=2$: $P_Y(4) = 0.3$
[!TIP]
Summarize formulas on a last revision sheet: binomial, Poisson, normal, exponential PDFs, and main transformation techniques.