Robotics (IT-802 (C)) Unit 2 Notes - RGPV

UNIT 2: MACHINE LEARNING FOUNDATIONS & TECHNIQUES - ROBOTICS

1. FOUNDATIONS OF MACHINE LEARNING

1.1 Learning Paradigms & Hypothesis Spaces

Learning Paradigms:
- Supervised Learning: Learn mapping from inputs to outputs using labeled examples. Robotic Example: Training a robot to grasp objects using labeled images (object type, grasp success).
- Unsupervised Learning: Find hidden patterns in unlabeled data. Robotic Example: Clustering sensor readings to discover different environmental zones.
- Reinforcement Learning: Learn through trial-and-error interactions with an environment to maximize a reward. Robotic Example: Teaching a robot to walk by rewarding forward motion.
Finite vs. Infinite Hypothesis Spaces:
- Finite Hypothesis Space (H): A set of possible models with a fixed, countable number of parameters or structures (e.g., all decision trees with depth ≤ 5).
- Infinite Hypothesis Space (H): An uncountable set of models, typically defined by continuous parameters (e.g., all possible linear functions $$\displaystyle y = wx + b $$ where $w, b \in \mathbb{R}$).
- Key Implications:
  
  | Aspect | Finite H | Infinite H | | :--- | :--- | :--- | | Model Complexity | Bounded by space size | Can be arbitrarily high | | Generalization | Guarantees possible (PAC learning) | Risk of overfitting without regularization | | Overfitting/Underfitting | Underfitting more common if H too small | Overfitting more common if H too large |
[!TIP] Exam Focus: Be prepared to contrast these spaces and link infinite spaces directly to the need for regularization (e.g., L1/L2 in regression) to control complexity and improve generalization.

1.2 Core Concepts & Evaluation

Data Splits & Cross-Validation:
- Training Set: Used to fit the model.
- Validation Set: Used to tune hyperparameters and select models.
- Test Set: Used for final, unbiased evaluation of model performance.
- Cross-Validation (e.g., k-fold): Splits data into k folds; model trained k times, each time on k-1 folds and validated on the held-out fold. Provides robust performance estimate.
Bias-Variance Trade-off:
- Bias Error: Error from erroneous assumptions (e.g., linear model for nonlinear data). High bias → Underfitting.
- Variance Error: Error from sensitivity to small fluctuations in training data. High variance → Overfitting.
- Trade-off: Decreasing bias typically increases variance, and vice-versa. Goal is to find optimal balance.
Objective of Cost/Error Function Convergence:
- Goal: Find model parameters $\theta$ that minimize a cost function $J(\theta)$ (e.g., Sum of Squared Errors - SSE).
- SSE for Linear Regression:

$$J(w, b) = \sum_{i=1}^{n} (y_i - (w x_i + b))^2$$

*   **Gradient Descent (GD):** Iterative optimization algorithm to find $\theta$ that minimizes $J(\theta)$.

$$ \theta_{new} = \theta_{old} - \alpha \cdot \nabla J(\theta_{old}) $$

    where $\alpha$ is the learning rate. Convergence means $J(\theta)$ stabilizes at a (local) minimum.

> [!TIP] **Common Pitfall:** Convergence of GD does **not** guarantee a *global* minimum for non-convex functions (like in neural networks). It finds a local minimum.

2. SUPERVISED LEARNING: REGRESSION & CLASSIFICATION

2.1 Regression Techniques

Linear & Multiple Linear Regression:
- Model: $$\displaystyle y = \beta_0 + \beta_1 x_1 + ... + \beta_p x_p + \epsilon $$
- Estimation: Least Squares Method finds $\hat{\beta}$ that minimizes SSE:

$$\text{SSE} = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

*   **Interpretation:** $$\displaystyle \beta_j $$ = change in $y$ for a one-unit change in $$\displaystyle x_j $$, holding other variables constant.

*   **Goodness-of-Fit:** **R-squared** ($$\displaystyle R^2 $$) = Proportion of variance in $y$ explained by the model. $$\displaystyle 0 \leq R^2 \leq 1 $$.

*   **Practical Problem-Solving Steps:**

    1.  Compute means $\bar{x}, \bar{y}$.

    2.  Calculate slope $$\displaystyle \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} $$.

    3.  Calculate intercept $$\displaystyle \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} $$.

    4.  Predict $\hat{y}$ for new $x$.

    5.  Compute $$\displaystyle R^2 = 1 - \frac{\text{SSE}}{\text{SST}} $$ where $$\displaystyle \text{SST} = \sum (y_i - \bar{y})^2 $$.

Logistic Regression:
- Purpose: Binary classification (predict probability $$\displaystyle P(Y=1|X) $$).
- Model: Uses sigmoid function to map linear combination to [0,1].

$$ P(Y=1|X) = \frac{1}{1 + e^{-(\beta_0 + \beta^T X)}} $$

*   **Decision Boundary:** $$\displaystyle P(Y=1|X) = 0.5 \Rightarrow \beta_0 + \beta^T X = 0 $$. This is a linear hyperplane.

*   **Odds Ratio:** $$\displaystyle \frac{P(Y=1|X)}{1-P(Y=1|X)} = e^{\beta_0 + \beta^T X} $$. A one-unit increase in $$\displaystyle x_j $$ multiplies the odds by $$\displaystyle e^{\beta_j} $$.

*   *Robotic Example:* Classify sensor data as "Obstacle" (1) or "Free" (0) based on distance readings.

2.2 Classification Techniques

Support Vector Machines (SVM) - Linear:
- Goal: Find the maximum-margin hyperplane that separates two classes.
- Key Concepts:
  - Hyperplane: $$\displaystyle w^T x + b = 0 $$.
  - Margin: Distance between hyperplane and nearest data points from each class.
  - Support Vectors: The critical training points that lie on the margin boundaries ($$\displaystyle w^T x + b = \pm 1 $$). They support the margin.
- Margin Optimization (Primal Problem Intuition):
  
  Maximize margin $$\displaystyle \propto \frac{1}{\|w\|} $$ subject to correct classification: $$\displaystyle y_i(w^T x_i + b) \geq 1 $$.
  
  Equivalent to minimizing:

$$\min_{w,b} \frac{1}{2} \|w\|^2 \quad \text{s.t.} \quad y_i(w^T x_i + b) \geq 1$$

*   **Practical Problem-Solving:** Given points, identify support vectors (closest to the separating line), write constraints, and solve for $w, b$ that satisfy $$\displaystyle y_i(w^T x_i + b) = 1 $$ for support vectors.

Decision Trees (ID3 Algorithm):
- Core Idea: Recursively split data based on attribute that yields the highest Information Gain.
- Entropy (Measure of Impurity): For a set $S$ with class proportions $$\displaystyle p_i $$,

$$ \text{Entropy}(S) = -\sum_{i} p_i \log_2 p_i $$

    (Entropy = 0 for pure node, 1 for equally mixed).

*   **Information Gain:** Reduction in entropy after splitting on attribute $A$.

$$ \text{Gain}(S, A) = \text{Entropy}(S) - \sum_{v \in \text{Values}(A)} \frac{|S_v|}{|S|} \text{Entropy}(S_v) $$

*   **Algorithm Steps:**

    1.  Start with all training examples at root.

    2.  If all examples belong to same class, make leaf with that class.

    3.  Else, for each attribute, compute Information Gain.

    4.  Select attribute with highest Gain as splitting node.

    5.  Repeat recursively for each branch/subset.

*   **Handling Continuous Attributes:** Discretize by finding a threshold that maximizes Gain (e.g., test $$\displaystyle x_j > c $$ for various $c$).

*   **Pruning:** Remove branches that have little statistical significance to reduce overfitting (post-pruning is more common).

Naïve Bayes Classifier:
- Core Assumptions:
  1. Conditional Independence: Given the class $C$, features $$\displaystyle X_1, X_2, ..., X_n $$ are conditionally independent.
  2. This implies: $$\displaystyle P(X_1, X_2, ..., X_n | C) = \prod_{i} P(X_i | C) $$.
- How Assumptions Simplify Computation:
  
  Without assumption, need joint distribution $$\displaystyle P(X_1, ..., X_n | C) $$ which requires exponential data. With assumption, only need $$\displaystyle P(X_i | C) $$ for each feature, which is tractable.
- Bayes' Theorem for Classification:

$$ P(C|X) = \frac{P(C) \prod_{i} P(X_i|C)}{P(X)} \propto P(C) \prod_{i} P(X_i|C) $$

    Predict class $$\displaystyle \hat{C} = \arg\max_C P(C) \prod_{i} P(X_i|C) $$.

*   **Posterior vs. Prior:**

    *   **Prior Probability $P(C)$:** Probability of class before seeing evidence (e.g., overall prevalence).

    *   **Posterior Probability $P(C|X)$:** Updated probability after observing features $X$.

*   **Bayesian Reasoning Application (False Positives/Negatives):**

    > [!TIP] **Classic Problem:** Use Bayes' theorem to find $P(\text{Disease}|\text{Positive Test})$.

$$ P(D|+) = \frac{P(+|D)P(D)}{P(+)} = \frac{P(+|D)P(D)}{P(+|D)P(D) + P(+|\neg D)P(\neg D)} $$

    *Robotic Example:* Updating belief about object identity ($C$) based on multiple, noisy sensor readings ($$\displaystyle X_i $$).

3. UNSUPERVISED LEARNING: CLUSTERING & DIMENSIONALITY REDUCTION

3.1 Clustering Methods

K-Means Clustering:
- Algorithm Steps:
  1. Initialization: Randomly choose $k$ initial centroids $$\displaystyle \mu_1^{(0)}, ..., \mu_k^{(0)} $$.
  2. Assignment: Assign each point $$\displaystyle x_i $$ to nearest centroid (using Euclidean distance):

$$ c_i^{(t)} = \arg\min_{j} \| x_i - \mu_j^{(t)} \|^2 $$

    3.  **Update:** Recompute centroids as mean of assigned points:

$$ \mu_j^{(t+1)} = \frac{1}{|C_j^{(t)}|} \sum_{x_i \in C_j^{(t)}} x_i $$

    4.  Repeat steps 2-3 until centroids stabilize (or max iterations).

*   **Distance Measure:** Euclidean distance $$\displaystyle d(x_i, x_j) = \sqrt{\sum_{f=1}^{d} (x_{if} - x_{jf})^2} $$.

*   **Choosing *k*: Elbow Method.** Plot Within-Cluster Sum of Squares (WCSS) vs. $k$. Look for "elbow" where decrease slows.

*   **Practical Problem-Solving:** Given initial centers, perform **one full iteration** (assignment + update) and show new centers/clusters.

Hierarchical Clustering:
- AGNES (Agglomerative): Bottom-up. Start with each point as a cluster; repeatedly merge closest pair of clusters.
- DIANA (Divisive): Top-down. Start with all points in one cluster; recursively split clusters.
- Linkage Criteria (for AGNES): Defines "closeness" between clusters.
  - Single Linkage: Min distance between points in clusters. ($$\displaystyle d_{\min} $$)
  - Complete Linkage: Max distance between points in clusters. ($$\displaystyle d_{\max} $$)
  - Average Linkage: Average distance between all pairs. ($$\displaystyle d_{avg} $$)
- Dendrogram: Tree diagram showing sequence of merges/splits. Height represents distance. Cut at a height to get clusters.
- Advantages & Limitations:
  
  | Method | Advantages | Limitations | | :--- | :--- | :--- | | AGNES | No need to specify k upfront; produces dendrogram. | Once merged, cannot undo; computationally heavy $$\displaystyle O(n^3) $$; sensitive to noise. | | DIANA | Can handle large clusters better initially. | Same as AGNES; less common. | | Hierarchical (General) | Gives hierarchy of clusters; useful for data exploration. | Not scalable for large $n$; choice of linkage greatly affects results. |
Gaussian Mixture Models (GMM):
- Concept: Assumes data points are generated from a mixture of several Gaussian distributions. Each cluster is a Gaussian with its own mean $$\displaystyle \mu_k $$ and covariance $$\displaystyle \Sigma_k $$.
- Model:

$$ p(x) = \sum_{k=1}^{K} \pi_k \mathcal{N}(x | \mu_k, \Sigma_k) $$

where $$\displaystyle \pi_k $$ is mixing coefficient ($$\displaystyle \sum \pi_k = 1 $$).

*   **Suitability for Overlapping Clusters:** Provides **soft clustering** (posterior probabilities $$\displaystyle P(z=k|x) $$) instead of hard assignments. Clusters can overlap because the Gaussians themselves can overlap.

*   **Expectation-Maximization (EM) Algorithm Intuition:**

    1.  **E-step:** Compute responsibility $$\displaystyle \gamma(z_k) $$ = posterior probability that point $$\displaystyle x_i $$ belongs to cluster $k$, given current parameters.

    2.  **M-step:** Update parameters $$\displaystyle \pi_k, \mu_k, \Sigma_k $$ using the responsibilities as soft counts.

    3.  Repeat until log-likelihood converges.

3.2 Dimensionality Reduction

Principal Component Analysis (PCA):
- Objective: Find orthogonal axes (principal components) that maximize variance in the data (equivalently, minimize reconstruction error).
- Steps:
  1. Standardize data (mean=0, variance=1).
  2. Compute covariance matrix $$\displaystyle \Sigma = \frac{1}{n-1} X^T X $$.
  3. Compute eigenvectors and eigenvalues of $\Sigma$.
  4. Sort eigenvectors by decreasing eigenvalues. Top $k$ eigenvectors = principal components.
  5. Project data: $$\displaystyle X_{\text{reduced}} = X \cdot W_k $$ where $$\displaystyle W_k $$ is matrix of top $k$ eigenvectors.
- Example: Reduce 100D sensor data to 2D for visualization while preserving maximum spread/variance.
Locally Linear Embedding (LLE):
- Core Idea: Manifold learning technique. Preserves local neighborhood geometry. Assumes data lies on a smooth, non-linear manifold.
- Process:
  1. Find $k$ nearest neighbors for each point.
  2. Compute reconstruction weights $$\displaystyle W_{ij} $$ that best linearly reconstruct each point from its neighbors (minimize $$\displaystyle \| x_i - \sum_j W_{ij} x_j \|^2 $$ subject to $$\displaystyle \sum_j W_{ij}=1 $$).
  3. Map to low-dimensional space by finding points $$\displaystyle y_i $$ that maintain these same linear relationships (minimize $$\displaystyle \| y_i - \sum_j W_{ij} y_j \|^2 $$).
- When to Prefer LLE over PCA: When data has non-linear, manifold structure (e.g., Swiss roll, a bent sheet). PCA only finds linear projections and would "crush" such structures.
- Concept of Dimensionality Reduction:
  - Curse of Dimensionality: In high dimensions, data becomes sparse, distances lose meaning, and models overfit.
  - Benefits: Visualization, noise reduction, improved model efficiency/performance, storage savings.

4. ENSEMBLE LEARNING & MODEL COMBINATION

4.1 Core Ensemble Techniques

Bagging (Bootstrap Aggregating):
- Process:
  1. Bootstrapping: Create $B$ bootstrap samples (random samples with replacement) from original training set.
  2. Parallel Training: Train $B$ base models (e.g., decision trees) independently on each bootstrap sample.
  3. Aggregation: For regression → average predictions. For classification → majority vote.
- How it Reduces Variance: Averages out the "noise" or fluctuations from individual unstable models (like deep decision trees). The variance of the ensemble is lower than variance of individual models.
- Example: Random Forest is a specialized form of bagging for decision trees (also uses random feature subsets).
Boosting (Contrast):
- Process: Sequential training. Each new model focuses on correcting errors of the previous ensemble.
- Effect: Primarily reduces bias. Builds a strong learner from weak learners.
- Examples: AdaBoost (adjusts instance weights), Gradient Boosting (fits residuals).

4.2 Advanced Combination Schemes

Stacking (Stacked Generalization):
- Process:
  1. Base Learners (Level-0): Train diverse models (e.g., SVM, DT, k-NN) on original data.
  2. Meta-Learner (Level-1): Train a meta-model (e.g., linear regression) on a new dataset. This new dataset's features are the predictions of the base learners on a hold-out validation set. The target is the original target variable.
- Role of Meta-Learners: Learns the optimal way to combine the base models' predictions. It can assign different weights or learn complex combinations, often leading to better performance than simple averaging/voting.
- Advantage: Can leverage strengths of heterogeneous models.
Comparison of Model Combination Schemes:

| Scheme | Training | Primary Goal | Typical Base Models | Key Strength | | :--- | :--- | :--- | :--- | :--- | | Bagging | Parallel | Reduce Variance | Unstable models (e.g., trees) | Robust to overfitting | | Boosting | Sequential | Reduce Bias | Weak learners | High predictive accuracy | | Stacking | Sequential (2-level) | Improve Accuracy | Heterogeneous models | Learns optimal combination |

5. NEURAL NETWORKS & ADVANCED TOPICS

5.1 Multilayer Perceptron (MLP) Fundamentals

Role of Activation Functions:
- Need for Non-linearity: Without activation functions, an MLP is just a linear transformation ($$\displaystyle y = W_2(W_1 x + b_1) + b_2 $$), no matter how many layers. It cannot learn complex, non-linear patterns.
- How they Enable Non-linearity: Each neuron applies a non-linear function $\phi(\cdot)$ to its linear input $$\displaystyle z = w^T x + b $$. Stacking layers allows composition of non-linear functions, enabling approximation of any continuous function (Universal Approximation Theorem).
- Common Functions & Properties:
  
  | Function | Formula | Range | Derivative | Use in Robotics | | :--- | :--- | :--- | :--- | :--- | | Sigmoid | $$\displaystyle \frac{1}{1+e^{-z}} $$ | (0,1) | $\sigma(z)(1-\sigma(z))$ | Output layer for binary classification (probability). | | Tanh | $$\displaystyle \frac{e^z - e^{-z}}{e^z + e^{-z}} $$ | (-1,1) | $$\displaystyle 1 - \tanh^2(z) $$ | Hidden layers; zero-centered output. | | ReLU | $\max(0, z)$ | [0, ∞) | 1 if $$\displaystyle z>0 $$, else 0 | Most common hidden layer; sparse activation, mitigates vanishing gradient. |
- Robotic Relevance: Non-linear activation is crucial for learning complex control policies from high-dimensional sensor data (e.g., images, lidar).

5.2 Specialized Architectures & Techniques

Self-Organizing Map (SOM):
- Type: Unsupervised, competitive learning network.
- Goal: Create a low-dimensional (usually 2D), topology-preserving map of input space. Similar inputs map to nearby neurons.
- Process:
  1. Initialization: Neurons on a grid have weight vectors $$\displaystyle w_j $$ of same dimension as input.
  2. Competition: For input $x$, find Best Matching Unit (BMU) $c$ with weight closest to $x$.
  3. Cooperation: BMU and its neighborhood neurons on the grid adjust their weights towards $x$.
  4. Adaptation: $$\displaystyle w_j(t+1) = w_j(t) + \alpha(t) \cdot h_{cj}(t) \cdot (x - w_j(t)) $$ where $$\displaystyle h_{cj} $$ is neighborhood function (decreases with distance from BMU and time).
  5. Repeat, gradually reducing learning rate $\alpha$ and neighborhood radius.
- Application Example: Clustering and visualizing high-dimensional robot sensor data (e.g., tactile array patterns) on a 2D grid to identify distinct contact states or terrain types.
- DiagramCANVAS: A 2D grid of neurons. An input vector in high-D space is shown. Arrows point from the BMU and its neighbors towards the input vector, indicating weight update direction. The grid shows a smooth coloring gradient, representing the topology preservation.
Frequent Pattern Mining & Apriori Algorithm:
- Concept: Find itemsets (sets of items) that appear together in transactions frequently.
- Key Metrics:
  - Support: $$\displaystyle P(\text{Itemset}) = \frac{\text{transactions containing itemset}}{\text{total transactions}} $$. Measures frequency.
  - Confidence: $$\displaystyle P(B|A) = \frac{\text{Support}(A \cup B)}{\text{Support}(A)} $$. Measures conditional likelihood.
- Importance (Market Basket Analysis): Discover associations like {Diapers} -> {Beer}. Robotic Extension: Discover frequent sequences of sensor events or joint actions in task demonstrations.
- Apriori Algorithm (State the Algorithm):
  1. Set minimum support threshold min_supp.
  2. Generate L1: Find all frequent 1-itemsets (support ≥ min_supp).
  3. k=2; while L_{k-1} ≠ ∅:
    - Generate candidate k-itemsets $$\displaystyle C_k $$ by joining L_{k-1} with itself (apriori property: all subsets of a frequent itemset must be frequent).
    - Prune $$\displaystyle C_k $$: Remove candidates with any (k-1)-subset not in L_{k-1}.
    - Count support of candidates in $$\displaystyle C_k $$ by scanning DB.
    - L_k = {c ∈ C_k | support(c) ≥ min_supp}
    - k = k+1
  4. Return $$\displaystyle \bigcup_k L_k $$.
- Apriori Property: "All non-empty subsets of a frequent itemset must be frequent." Used for pruning candidate itemsets.

6. PROBABILISTIC GRAPHICAL MODELS

6.1 Bayesian Networks (Belief Networks)

Constructing a Bayesian Belief Network:
1. Define Nodes: Identify random variables of interest (e.g., Malaria, Headache, Fever).
2. Define Edges (Dependencies): Draw directed edges from cause to effect (parent → child). Encodes conditional independence: a node is conditionally independent of its non-descendants given its parents.
3. Specify Conditional Probability Tables (CPTs): For each node, specify $P(\text{Node} | \text{Parents})$. For a node with $k$ parent states and $m$ own states, CPT has $m \times k$ entries.
Joint Probability Distribution:

From network structure (chain rule of probability using conditional independencies):

$$ P(X_1, X_2, ..., X_n) = \prod_{i=1}^{n} P(X_i | \text{Parents}(X_i)) $$

*Example (Malaria/Headache):*

Nodes: `M` (Malaria), `H` (Headache). Edge: `M → H`.

Joint: $$\displaystyle P(M, H) = P(M) \cdot P(H|M) $$.

Given $$\displaystyle P(M)=0.03 $$, $$\displaystyle P(H|M)=0.6 $$, $$\displaystyle P(H|\neg M)=0.2 $$.

CPT:

| M | P(H=yes\|M) | P(H=no\|M) |

| :--- | :--- | :--- | | yes | 0.6 | 0.4 | | no | 0.2 | 0.8 |

**Joint Probability Table:**

| M | H | Joint Probability |

| :--- | :--- | :--- | | yes | yes | $$\displaystyle 0.03 \times 0.6 = 0.018 $$ | | yes | no | $$\displaystyle 0.03 \times 0.4 = 0.012 $$ | | no | yes | $$\displaystyle 0.97 \times 0.2 = 0.194 $$ | | no | no | $$\displaystyle 0.97 \times 0.8 = 0.776 $$ |

6.2 Bayesian Learning

Bayesian Learning Framework:
- Prior Probability $P(\theta)$: Belief about parameter $\theta$ before seeing data.
- Likelihood $P(D|\theta)$: Probability of observing data $D$ given parameter $\theta$.
- Posterior Probability $P(\theta|D)$: Updated belief about $\theta$ after seeing data $D$.
- Bayes' Theorem:

$$ P(\theta|D) = \frac{P(D|\theta) P(\theta)}{P(D)} \propto P(D|\theta) P(\theta) $$

Example (Robot Belief Update):
- Prior: Robot believes probability of "door" vs "wall" ahead based on map (e.g., $$\displaystyle P(\text{Door})=0.3 $$).
- Likelihood: Sensor model: $$\displaystyle P(\text{Reading}=\text{"gap"}|\text{Door})=0.9 $$, $$\displaystyle P(\text{Reading}=\text{"gap"}|\text{Wall})=0.1 $$.
- Evidence: Robot sensor reads "gap".
- Posterior: $$\displaystyle P(\text{Door}|\text{"gap"}) \propto P(\text{"gap"}|\text{Door})P(\text{Door}) = 0.9 \times 0.3 = 0.27 $$. Normalize with $P(\text{"gap"})$ to get final posterior. Belief in "door" increases.

EXAMINATION FOCUS PRIORITY (Based on Historical Frequency)

VERY HIGH FREQUENCY (Appears in both papers or multiple forms):
- Clustering: K-Means (numerical iteration), Hierarchical (AGNES/DIANA comparison, linkage), GMM (EM intuition, soft clustering).
- Regression: Linear (least squares, prediction, $$\displaystyle R^2 $$), Logistic (sigmoid, decision boundary).
- Dimensionality Reduction: PCA (objective, eigen-analysis) vs. LLE (manifold, when to use).
- SVM: Theory (hyperplane, margin, support vectors) & margin calculation from points.
- Naïve Bayes: Assumptions (conditional independence), posterior calculation, Bayesian reasoning with false positives/negatives (Covid test problem).
- Ensemble Learning: Bagging (variance reduction), Stacking (meta-learner role), comparison (Bagging vs Boosting vs Stacking).
HIGH FREQUENCY (Appears once with detailed theory):
- Hypothesis Spaces: Finite vs. Infinite (implications on complexity/generalization).
- Decision Trees: ID3 algorithm, Entropy & Information Gain calculation.
- Bayesian Networks: Construction (nodes, edges, CPTs), joint probability from graph.
- MLP: Role of activation functions (need for non-linearity, ReLU/Tanh/Sigmoid).
- Frequent Pattern Mining: Apriori algorithm (state), support/confidence.
APPLICATION/PROBLEM-SOLVING FOCUS:
- Numerical Problems: Linear regression (fit equation, predict, $$\displaystyle R^2 $$), K-Means (first iteration, final clusters), Euclidean distance, Bayesian probability (false positive/negative), Apriori (min_support counting).
- Algorithmic Steps: K-Means iteration, constructing a simple Bayesian Network, ID3 tree building (conceptual, using Info Gain), EM for GMM (conceptual E/M steps).

Account

Robotics (IT-802 (C)) - Unit 2 Short Notes