Pattern Recognition (AL-603 (C)) - Unit 2 Short Notes

UNIT 2: FOUNDATIONAL STATISTICAL CONCEPTS

Measures of Central Tendency

Definition: Single values that describe the center of a data distribution.

[!TIP] Exam Focus: Be prepared to calculate each from a small dataset and justify which is most appropriate for a given data type (e.g., median for income data).

Measures of Dispersion (Spread)

Definition: Quantify the variability or spread of data points around the central value.

[!TIP] Key Point: Variance and SD use $n-1$ (degrees of freedom) for sample estimation to provide an unbiased estimate of population variance.

Levels of Measurement

Variables and Data Categorization

• Categorical Variables:

  • Nominal: No intrinsic order (e.g., color, country).

  • Ordinal: Has order but not fixed intervals (e.g., small/medium/large).

• Numerical (Quantitative) Variables:

  • Discrete: Countable values (integers), gaps (e.g., number of children).

  • Continuous: Any value in a range, measurable (e.g., time, weight).

• Coding Schemes: Converting categorical data to numerical for analysis.

  • Dummy Variables (0/1): For nominal/ordinal in regression.

  • Label Encoding: Assigns integers (can imply false order—use cautiously).

  • One-Hot Encoding: Creates binary column for each category (avoids ordinal assumption).

UNIT 2: INFERENTIAL STATISTICS

Statistical Inference

• Estimation:

  • Point Estimation: Single value (e.g., $\bar{x}$ estimates $\mu$).

  • Interval Estimation (Confidence Interval): Range likely to contain population parameter.

    • For mean (known $\sigma$): $$\displaystyle \bar{x} \pm z_{\alpha/2} \frac{\sigma}{\sqrt{n}} $$

    • For mean (unknown $\sigma$): $$\displaystyle \bar{x} \pm t_{\alpha/2, df} \frac{s}{\sqrt{n}} $$

• Hypothesis Testing:

  • Null Hypothesis ($$\displaystyle H_0 $$): Status quo, no effect.

  • Alternative Hypothesis ($$\displaystyle H_1 $$ or $$\displaystyle H_a $$): Research claim.

  • p-value: Probability of observing data given $$\displaystyle H_0 $$ is true. Reject $$\displaystyle H_0 $$ if p-value < $\alpha$ (significance level).

  • Type I Error ($\alpha$): Rejecting true $$\displaystyle H_0 $$.

  • Type II Error ($\beta$): Failing to reject false $$\displaystyle H_0 $$. Power = $1-\beta$.

Parametric Tests

t-test

Assumptions: Random sample, normality (or large n), homogeneity of variance (for two-sample).

Example (Potato Yield - One-sample t-test from May 2024):

$$\displaystyle H_0: \mu = 20 $$ (standard yield), $$\displaystyle H_1: \mu > 20 $$ (better).

Data: $$\displaystyle X = [21.5, 24.5, ..., 18.5] $$, $$\displaystyle n=12 $$.

1. Compute $\bar{x}$ and $s$.

2. Calculate $$\displaystyle t = \frac{\bar{x} - 20}{s/\sqrt{12}} $$.

3. Compare to $$\displaystyle t_{critical, \alpha, df=11} $$ (one-tailed). Reject $$\displaystyle H_0 $$ if $$\displaystyle t_{calc} > t_{crit} $$.

Chi-Square ($$\displaystyle \chi^2 $$) Test

Assumptions: Independent observations, expected frequencies ≥ 5 (mostly), categorical data.

[!TIP] Calculation Steps: 1) State $$\displaystyle H_0 $$ (no association/good fit). 2) Build table, compute $$\displaystyle E_i $$. 3) Calculate $$\displaystyle \chi^2_{calc} $$. 4) Find critical $$\displaystyle \chi^2 $$ from table or compute p-value. 5) Conclude.

Maximum Likelihood Estimation (MLE)

Goal: Find parameter values ($\theta$) that maximize the likelihood of observed data.

1. Likelihood Function: $$\displaystyle L(\theta | data) = \prod_{i=1}^{n} f(x_i; \theta) $$ (joint PDF/PMF).

2. Log-Likelihood: $$\displaystyle \ell(\theta) = \ln L(\theta) = \sum \ln f(x_i; \theta) $$ (easier to maximize).

3. Estimation: Solve $$\displaystyle \frac{d\ell}{d\theta} = 0 $$ for $$\displaystyle \hat{\theta}_{MLE} $$.

4. Properties:

   • Consistency: Converges to true $\theta$ as $n \to \infty$.

   • Efficiency: Achieves Cramér-Rao lower bound asymptotically.

   • Invariance: If $\hat{\theta}$ is MLE of $\theta$, then $g(\hat{\theta})$ is MLE of $g(\theta)$.

Example: For Normal($$\displaystyle \mu, \sigma^2 $$), MLE for $\mu$ is $\bar{x}$, for $$\displaystyle \sigma^2 $$ is $$\displaystyle \frac{1}{n}\sum (x_i - \bar{x})^2 $$ (note: biased, uses n not n-1).

Resampling Methods

• Bootstrap: Sample with replacement from original data to create many "bootstrap samples." Estimate sampling distribution (e.g., CI for median).

• Cross-Validation:

  • k-fold: Split data into k subsets; train on k-1, test on 1; repeat k times. Average performance.

  • Leave-One-Out (LOO): k = n. High variance, computationally expensive.

• Permutation Test: Randomly shuffle labels/group assignments to generate null distribution of test statistic. Non-parametric alternative.

UNIT 2: MODELING TECHNIQUES

Regression Analysis

Simple Linear Regression (SLR):

• Model: $$\displaystyle y = \beta_0 + \beta_1 x + \epsilon $$, $$\displaystyle \epsilon \sim N(0, \sigma^2) $$.

• Least Squares Estimation: Minimize $$\displaystyle \sum (y_i - \hat{y}_i)^2 $$.

  • $$\displaystyle \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} = \frac{S_{xy}}{S_{xx}} $$

  • $$\displaystyle \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} $$

• Interpretation: $$\displaystyle \hat{\beta}_1 $$ = change in y per unit change in x.

Multiple Linear Regression (MLR):

• Model: $$\displaystyle y = \beta_0 + \beta_1 x_1 + ... + \beta_p x_p + \epsilon $$.

• Assumptions: Linearity, independence, homoscedasticity, normality of errors, no perfect multicollinearity.

• Multicollinearity: High correlation among predictors. Diagnose with VIF (Variance Inflation Factor). VIF > 5-10 indicates problem.

• Variables:

  • Dependent (y): Outcome.

  • Independent (x): Predictors.

  • Dummy Variables: 0/1 coding for categorical predictors (e.g., gender).

  • Interaction Terms: $$\displaystyle x_1 \times x_2 $$ to model effect modification.

Multivariate Analysis

Purpose: Analyze multiple variables simultaneously to understand relationships, reduce dimensionality, or group observations.

Bayesian Modeling

Core: Bayes' Theorem: $$\displaystyle P(\theta | D) = \frac{P(D | \theta) P(\theta)}{P(D)} $$

• Posterior $P(\theta | D)$: Updated belief after seeing data.

• Likelihood $P(D | \theta)$: Probability of data given parameter.

• Prior $P(\theta)$: Initial belief about parameter.

• Process:

  1. Specify prior distribution (informative, weakly informative, flat/uniform).

  2. Write likelihood based on data model.

  3. Compute posterior (analytically for conjugate priors, or via MCMC sampling like Gibbs, HMC).

• Advantages:

  • Incorporates prior knowledge/expert opinion.

  • Provides full probability distribution (uncertainty quantification).

  • Natural for sequential/online learning.

• Disadvantages:

  • Computationally intensive for complex models.

  • Choice of prior can influence results (sensitivity analysis needed).

  • Can be subjective.

UNIT 2: DATA HANDLING AND VISUALIZATION

Data Wrangling (Munging)

Process:

1. Data Cleaning:

   • Missing Values: Delete, impute (mean/median/mode, model-based), or flag.

   • Outliers: Detect via IQR (points beyond $Q1-1.5IQR$, $Q3+1.5IQR$) or Z-score (>3). Investigate cause; cap/transform or remove if erroneous.

   • Inconsistencies: Fix formatting (dates, units), correct typos.

2. Data Transformation:

   • Normalization (Min-Max): $$\displaystyle x' = \frac{x - \min}{\max - \min} $$ → [0,1].

   • Standardization (Z-score): $$\displaystyle x' = \frac{x - \mu}{\sigma} $$ → mean=0, SD=1.

   • Encoding: One-hot for nominal, label/ordinal for ordinal.

3. Data Integration: Merge/join datasets (SQL-like), concatenate, reshape (pivot/melt).

4. Tools: Python (Pandas: df.dropna(), df.merge(), pd.get_dummies()), R (dplyr, tidyr).

Data Visualization

Custom Visualization Design

Principles (C.A.S.E.):

• Clarity: Message is immediately understandable. Avoid clutter.

• Accuracy: Represent data truthfully; no distorted scales.

• Simplicity: Minimal ink for maximum insight (Tufte's data-ink ratio).

• Storytelling: Guide viewer with annotations, logical flow, title. Chart Selection: Match plot to data type and question (e.g., trend over time → line chart; part-to-whole → stacked bar/pie [use sparingly]; distribution → histogram/box). Tools: D3.js (web, highly customizable), Tableau (drag-and-drop BI), Python: Matplotlib (low-level), Seaborn (statistical, high-level), Plotly (interactive). R: ggplot2 (grammar of graphics).

UNIT 2: TOOLS, ECOSYSTEMS, AND DATA MANAGEMENT

Visualization Libraries

Business Intelligence Tools: Power BI

• Power BI Desktop: Free application for data connection, modeling, visualization creation.

• Power BI Service: Cloud-based SaaS for publishing, sharing, collaboration, scheduling refreshes.

• Power BI Mobile: Apps for iOS/Android to view reports on-the-go.

• Key Features:

  • Data Modeling: Create relationships between tables; define calculated columns/measures.

  • DAX (Data Analysis Expressions): Formula language for custom calculations (e.g., CALCULATE, FILTER).

  • Interactive Dashboards: Drill-down, cross-filtering, bookmarks.

  • Report Sharing: Publish to web, share within organization via workspaces.

Big Data Processing Tools

Data Management and Indexing

• SQL (Relational): Structured data, fixed schema, ACID transactions. (e.g., PostgreSQL, MySQL).

• NoSQL (Non-relational): Flexible schema, horizontal scaling. Types: Document (MongoDB), Key-Value (Redis), Column-family (Cassandra), Graph (Neo4j).

• Indexing: Data structure to speed up queries.

  • B-tree: Balanced tree; sorted keys; good for range queries. Default in most RDBMS.

  • Hash Index: Key-value lookup; fast equality; not for ranges.

  • Bitmap Index: Bit vectors for low-cardinality columns; efficient for multi-condition AND/OR (e.g., data warehouses).

Data Analyst Ecosystem

Roles: Collect, clean, analyze, visualize data; translate business questions to data questions; create reports/dashboards. Common Toolchain:

1. SQL: Extract/transform data from databases.

2. Python/R: Advanced analysis, statistics, machine learning (Pandas, NumPy, SciPy, scikit-learn, tidyverse).

3. Excel/Sheets: Quick analysis, prototyping.

4. BI Tools (Power BI, Tableau): Build interactive dashboards.

5. Version Control (Git): Track code changes, collaborate. End-to-End Workflow: $$\displaystyle \text{Data Collection} \rightarrow \text{Wrangling} \rightarrow \text{Analysis} \rightarrow \text{Visualization} \rightarrow \text{Reporting} $$.

File Formats

[!TIP] Rule of Thumb: Use Parquet for analytical queries in big data (column reads). Use CSV/JSON for interchange or small data. Use Avro for row-based streaming or when schema evolves frequently.