Machine Learning (IT-802 (A)) - Unit 4 Short Notes

I. FOUNDATIONS & CORE CONCEPTS OF MACHINE LEARNING

Definition & Perspectives

Machine Learning (ML) is a subset of AI that enables systems to learn patterns from data without explicit programming.

• AI vs. ML vs. Deep Learning: AI is the broadest field; ML is a subset using algorithms to learn from data; Deep Learning is a subset of ML using deep neural networks.

• Importance: Handles big data, automates decision-making, enables personalization (recommendation systems), and solves complex problems (image/speech recognition).

• Major Limitations: Data quality and bias, model interpretability ("black box"), computational cost, overfitting, and ethical concerns (privacy, fairness).

Learning Paradigms

• Supervised Learning: Labeled data; Classification (discrete output, e.g., spam detection) and Regression (continuous output, e.g., price prediction).

• Unsupervised Learning: Unlabeled data; Clustering (grouping, e.g., customer segmentation) and Dimensionality Reduction (e.g., PCA).

• Reinforcement Learning (RL): Agent learns via rewards/penalties from environment interaction (e.g., game playing).

• Semi-supervised & Self-supervised: Mix of labeled/unlabeled data; self-supervised creates pretext tasks from unlabeled data (e.g., masked language modeling).

Hypothesis Space & Inductive Bias

• Hypothesis Space: Set of all possible models considered.

  • Finite: Limited complexity, risk of underfitting.

  • Infinite: Highly flexible, risk of overfitting.

• Inductive Bias: Assumptions that guide learning (e.g., smoothness, simplicity).

• Overfitting vs. Underfitting:

  • Overfitting: Low training error, high test error (model too complex). Solutions: regularization, more data, pruning.

  • Underfitting: High training and test error (model too simple). Solutions: increase complexity, feature engineering.

Data & Preprocessing

• Necessity: Raw data is often noisy, incomplete, or in incompatible formats; preprocessing ensures model stability and convergence.

• Normalization/Standardization:

  • Min-Max: $$\displaystyle x' = \frac{x - \min}{\max - \min} $$ (scales to [0,1]).

  • Z-score: $$\displaystyle x' = \frac{x - \mu}{\sigma} $$ (mean=0, std=1).

  Crucial for gradient-based algorithms (e.g., neural networks).

• Categorical Data:

  • One-hot Encoding: Creates binary columns; no ordinal assumption; increases dimensionality.

  • Label Encoding: Assigns integers; assumes ordinal relationship; no dim increase but may mislead models.

• High-Dimensional Data: Curse of dimensionality (sparsity, distance concentration). Solutions: feature selection, dimensionality reduction (PCA).

• Data Augmentation: Artificially increase dataset size (e.g., image rotation, flipping, noise addition) to improve generalization.

[!TIP]

Common Pitfall: Using label encoding for nominal categories (e.g., colors) introduces false ordinality; always use one-hot for non-ordinal data.

II. EVALUATION, VALIDATION & OPTIMIZATION

Model Evaluation Metrics

• Regression:

  • MSE: $$\displaystyle \text{MSE} = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{y}_i)^2 $$

  • RMSE: $\sqrt{\text{MSE}}$

  • MAE: $$\displaystyle \frac{1}{n} \sum |y_i - \hat{y}_i| $$

  • R²: $$\displaystyle 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}} $$ (proportion of variance explained).

• Classification:

  • Confusion Matrix:

    | | Predicted + | Predicted - | |---|-------------|-------------| | Actual + | TP | FN | | Actual - | FP | TN |

  • Accuracy = $$\displaystyle \frac{TP+TN}{Total} $$

  • Precision = $$\displaystyle \frac{TP}{TP+FP} $$ (positive predictive value)

  • Recall = $$\displaystyle \frac{TP}{TP+FN} $$ (sensitivity)

  • F1-Score = $$\displaystyle 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} $$

  • ROC-AUC: Area under ROC curve (trade-off between TPR and FPR).

• BLEU Score (NLP):

  • n-gram precision: $$\displaystyle \frac{\text{matching n-grams}}{\text{total n-grams in candidate}} $$

  • Brevity penalty: penalizes short translations; $$\displaystyle BP = 1 $$ if $$\displaystyle c > r $$, else $$\displaystyle e^{(1-r/c)} $$.

  • BLEU = $$\displaystyle BP \cdot \exp\left(\sum_{n=1}^N w_n \log p_n\right) $$.

Experimental Design & Validation

• Train/Validation/Test Sets:

  • Train: model fitting.

  • Validation: hyperparameter tuning.

  • Test: final unbiased evaluation.

• Cross-Validation:

  • k-fold: Split data into k subsets; train on k-1, validate on 1; repeat k times. Reduces variance, better performance estimate.

• Resampling: Bootstrap (sampling with replacement) to estimate stability.

• Hyperparameter Tuning: Adjust parameters not learned by training (e.g., learning rate, layers). Justification: significantly improves performance; prevents overfitting/underfitting.

Optimization Algorithms

• Gradient Descent:

  • Batch: Full dataset per update (stable, slow).

  • Stochastic: One sample per update (fast, noisy).

  • Mini-batch: Compromise (common in practice).

• Advanced Optimizers:

  | Optimizer | Key Idea | Benefit | |-----------|----------|---------| | Momentum | Accumulate velocity | Accelerates convergence, dampens oscillations | | AdaGrad | Per-parameter adaptive LR | Handles sparse data | | RMSProp | Moving average of squared gradients | Adapts LR, solves AdaGrad diminishing | | Adam | Momentum + RMSProp | Default choice, fast convergence |

• Convex Optimization: If loss function is convex, gradient descent finds global optimum; many ML losses are non-convex (neural networks).

Loss Functions & Regularization

• Role of Loss: Measures error; gradients guide backpropagation and weight updates.

• Common Loss Functions:

  • MSE: regression.

  • Cross-Entropy: classification (binary: $-[y\log\hat{y} + (1-y)\log(1-\hat{y})]$; multi-class: $$\displaystyle -\sum y_i \log \hat{y}_i $$).

  • Hinge: SVM (max(0, 1 - y·f(x))).

• Regularization: Prevents overfitting by adding penalty to loss.

  • L1 (Lasso): Adds $$\displaystyle \lambda \sum |w| $$ → sparsity (some weights exactly zero).

  • L2 (Ridge): Adds $$\displaystyle \lambda \sum w^2 $$ → small weights, smooth.

• Dropout: Randomly deactivate neurons during training (e.g., 50%); reduces co-adaptation.

• Batch Normalization: Normalize layer inputs (mean=0, var=1) per mini-batch; stabilizes training, allows higher LR.

[!TIP]

Key Distinction: L1 produces sparse models (feature selection); L2 does not. Dropout is applied during training only; at inference, all neurons active with scaled weights.

III. NEURAL NETWORKS & DEEP LEARNING ARCHITECTURES

Artificial Neural Networks (ANN) & Perceptrons

• Biological vs. Artificial Neuron: Biological: dendrites (input), soma (processing), axon (output). Artificial: weighted sum + bias + activation.

• Single-Layer Perceptron: $$\displaystyle y = f(\sum w_i x_i + b) $$; learns linear boundaries; limited (cannot learn XOR).

• Multi-Layer Perceptron (MLP): Input layer → hidden layers (non-linear) → output layer. Universal approximator.

• Parallel Processing: Matrix operations (e.g., batch matrix multiplication) leverage GPUs for efficient computation.

Backpropagation

• Purpose: Compute gradients of loss w.r.t. weights via chain rule; enable gradient descent.

• Algorithm Steps:

  1. Forward pass: compute output and loss.

  2. Backward pass: compute $$\displaystyle \frac{\partial \text{Loss}}{\partial w} $$ layer by layer from output to input.

  3. Update weights: $$\displaystyle w \leftarrow w - \eta \frac{\partial \text{Loss}}{\partial w} $$.

• Vanishing/Exploding Gradients:

  • Vanishing: Gradients shrink exponentially in early layers (sigmoid/tanh); early layers learn slowly.

  • Exploding: Gradients grow (poor initialization); unstable training.

  • Solutions: ReLU activation, proper initialization (He/Xavier), batch normalization, gradient clipping.

Activation Functions

[!TIP]

Exam Focus: ReLU is default for hidden layers; Softmax for multi-class output; Sigmoid/Tanh for specific needs (RNNs, binary output).

Convolutional Neural Networks (CNN)

• Core Architecture:

  • Convolutional Layers: Apply filters (kernels) to extract features (edges, textures).

  • Pooling/Sub-sampling Layers: Downsample (Max: take max; Average: take mean); reduces spatial size, provides translation invariance.

  • Fully Connected (FC) Layers: At end; combine features for classification/regression.

• Why CNN for Images?

  • Parameter Sharing: Same filter across spatial locations → fewer parameters.

  • Spatial Hierarchy: Early layers learn low-level features (edges), deeper layers learn high-level (objects).

• Convolution Operations:

  • Standard: Filter slides over input; output size: $$\displaystyle \frac{W - K + 2P}{S} + 1 $$ (W=input size, K=kernel, P=padding, S=stride).

  • Padding:

    • Same: Output size = input size ($$\displaystyle P = \frac{K-1}{2} $$); preserves spatial information.

    • Valid: No padding ($$\displaystyle P=0 $$); output shrinks.

  • Stride: Step size of filter; larger stride → smaller output.

• 1×1 Convolution:

  • Purpose:

    1. Feature/channel reduction (e.g., reduce 256 channels to 64).

    2. Channel mixing (combine information across channels).

    3. Increase nonlinearity without spatial change.

  • Used in Inception, residual blocks.

• Flattening: Convert 3D feature maps (H×W×C) to 1D vector for FC layers.

• Advanced Architectures:

  • Inception Module: Parallel convolutions (1×1, 3×3, 5×5) + max pooling; concatenate outputs. Benefits: multi-scale feature extraction, efficient computation (1×1 reduces channels first).

  • ImageNet Competition:

    • AlexNet (2012): First deep CNN (8 layers), ReLU, Dropout, GPU training.

    • VGG (2014): Very deep (16-19 layers), small 3×3 filters, uniform architecture.

    • ResNet (2015): Skip connections (residual blocks), enables very deep networks (100+ layers), solves vanishing gradient.

[!TIP]

Common Exam Question: "Explain 1×1 convolution" – emphasize channel reduction and nonlinearity. "Padding types" – same preserves size, valid reduces.

Recurrent Neural Networks (RNN) & Long Short-Term Memory (LSTM)

• RNN Architecture:

  • Hidden state $$\displaystyle h_t = f(W_{hh} h_{t-1} + W_{xh} x_t + b) $$; carries temporal information.

  • Types:

    • One-to-One: standard feedforward.

    • One-to-Many: e.g., image captioning.

    • Many-to-One: e.g., sentiment analysis.

    • Many-to-Many: e.g., machine translation.

• Vanishing Gradient in Vanilla RNNs: Gradients decay exponentially over time steps; struggles with long-term dependencies.

• LSTM Architecture:

  • Cell State $$\displaystyle c_t $$: "Conveyor belt" for long-term information.

  • Gates:

    • Forget Gate: $$\displaystyle f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) $$ → what to drop from cell state.

    • Input Gate: $$\displaystyle i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) $$; $$\displaystyle \tilde{c}_t = \tanh(W_c \cdot [h_{t-1}, x_t] + b_c) $$ → what to store.

    • Update: $$\displaystyle c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t $$.

    • Output Gate: $$\displaystyle o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) $$; $$\displaystyle h_t = o_t \odot \tanh(c_t) $$.

  • Handles long-term dependencies via additive cell state updates.

• GRU vs. LSTM vs. Vanilla RNN:

  | Feature | Vanilla RNN | LSTM | GRU | |---------|-------------|------|-----| | Gates | None | Forget, Input, Output | Reset, Update | | State | Hidden only | Hidden + Cell | Hidden only (merged) | | Parameters | Fewest | Most | Between | | Long-term | Poor | Good | Good (simpler) |

• Role in NLP & Sequence Generation: LSTMs/GRUs process sequences (words, time steps); used in ChatGPT (Transformer-based now, but RNNs were foundational).

Autoencoders

• Architecture:

  • Encoder: $$\displaystyle z = f(x) $$ compresses input to latent space (bottleneck).

  • Decoder: $$\displaystyle \hat{x} = g(z) $$ reconstructs input.

• Use in Unsupervised Learning: Learn efficient data representations without labels.

• Applications: Dimensionality reduction (latent space), denoising (train with noisy input, clean output), anomaly detection (high reconstruction error).

Transfer Learning

• Types:

  • Feature Extraction: Use pretrained network as fixed feature extractor; only train new classifier on top.

  • Fine-tuning: Unfreeze some layers (usually deeper ones) and train with low LR; adapts features to new task.

• Benefits: Less training data needed, faster convergence, better performance on small datasets.

IV. CORE ALGORITHMS: SUPERVISED & UNSUPERVISED

Support Vector Machines (SVM)

• Goal: Find optimal hyperplane $$\displaystyle w^T x + b = 0 $$ that maximizes margin (distance to nearest points).

• Support Vectors: Training points closest to hyperplane; define margin and decision boundary. Only these affect the model.

• Linear SVM for Linearly Separable Data: Solve quadratic programming: minimize $$\displaystyle \frac{1}{2}||w||^2 $$ subject to $$\displaystyle y_i(w^T x_i + b) \geq 1 $$.

• Performance in High-Dimensional Spaces: Effective when dimensions >> samples (e.g., bioinformatics, text); kernel trick avoids explicit high-dim mapping.

• Kernel Trick: Implicitly map data to high-dim via kernel function $$\displaystyle K(x_i, x_j) = \phi(x_i)^T \phi(x_j) $$. Common kernels: linear, polynomial, RBF (Gaussian).

Decision Trees

• Algorithm (ID3):

  1. Start with all examples at root.

  2. For each attribute, compute Entropy $$\displaystyle H(S) = -\sum p_i \log_2 p_i $$ (impurity).

  3. Compute Information Gain $$\displaystyle IG(S, A) = H(S) - \sum \frac{|S_v|}{|S|} H(S_v) $$.

  4. Select attribute with max IG; split.

  5. Recurse on subsets.

• Entropy: Measures uncertainty; max at uniform distribution ($$\displaystyle H=1 $$ for binary), min at pure ($$\displaystyle H=0 $$).

• Issues: Overfitting (deep trees), bias toward multi-valued attributes, instability (small data changes → different tree).

Ensemble Methods

• Bagging (Bootstrap Aggregation):

  • Create B bootstrap samples; train B models (e.g., decision trees).

  • Aggregate: average (regression) or majority vote (classification).

  • Variance Reduction: Decorrelates errors (e.g., Random Forest: bagging + random feature subset per split).

• Boosting:

  • Train weak learners sequentially; each focuses on errors of previous.

  • Example: AdaBoost – weight updates: misclassified points get higher weight.

  • Reduces bias.

• Stacking:

  • Train base models (e.g., SVM, RF, NN).

  • Use their predictions as features for a meta-learner (e.g., linear regression).

  • Combines heterogeneous models; meta-learner learns optimal combination.

• Comparison:

  | Scheme | Diversity Source | Primary Goal | Example | |--------|------------------|--------------|---------| | Bagging | Bootstrap samples | Variance reduction | Random Forest | | Boosting | Sample re-weighting | Bias reduction | AdaBoost, XGBoost | | Stacking | Heterogeneous models | Combine strengths | Meta-learner |

Clustering

• K-Means:

  1. Initialize K centroids (random or k-means++).

  2. Assign each point to nearest centroid (Euclidean distance).

  3. Update centroids as mean of assigned points.

  4. Repeat until convergence.

  • Sensitive to initialization, assumes spherical clusters, requires K.

• Hierarchical Clustering:

  • AGNES (Agglomerative): Bottom-up; start with each point as cluster; merge closest pairs (using linkage: single, complete, average).

  • DIANA (Divisive): Top-down; start with one cluster; split recursively.

  • Adaptive Hierarchical Clustering: Adjusts number of clusters based on density or connectivity (e.g., DBSCAN-inspired).

  • Advantages: No need to specify K; produces dendrogram.

  • Limitations: Computationally expensive ($$\displaystyle O(n^3) $$), irreversible merges/splits.

• Expectation-Maximization (EM):

  • E-step: Estimate latent variables (e.g., cluster assignments) given current parameters.

  • M-step: Maximize expected complete-data log-likelihood to update parameters.

  • Iterates until convergence; handles missing data naturally.

• Gaussian Mixture Models (GMM):

  • Model clusters as Gaussian distributions; soft assignments (probabilistic).

  • Parameters: means $$\displaystyle \mu_k $$, covariances $$\displaystyle \Sigma_k $$, mixing coefficients $$\displaystyle \pi_k $$.

  • Suitable for Overlapping Clusters: Soft assignments allow points to belong partially to multiple clusters.

Dimensionality Reduction

• Purpose:

  • Curse of dimensionality: sparsity, distance concentration, computational cost.

  • Visualization (2D/3D), noise reduction, feature extraction, speed up training.

• Principal Component Analysis (PCA):

  • Find orthogonal axes (principal components) that maximize variance.

  • Algorithm:

    1. Standardize data.

    2. Compute covariance matrix $$\displaystyle C = \frac{1}{n} X^T X $$.

    3. Eigen decomposition: $$\displaystyle C = V \Lambda V^T $$; columns of $V$ are PCs.

    4. Project: $$\displaystyle X_{\text{red}} = X V_k $$ (top k eigenvectors).

  • Maximizes variance; preserves global structure.

• Locally Linear Embedding (LLE):

  • Preserves local neighborhoods: each point reconstructed from its neighbors.

  • Non-linear; good for manifold data (e.g., Swiss roll).

  • Prefer LLE over PCA when data lies on a nonlinear manifold and local structure is more important than global variance.

• Other Techniques:

  • Factor Analysis: Latent factors with noise.

  • Partial Least Squares (PLS): Supervised; maximizes covariance between features and target.

  • Backward Elimination: Wrapper method; iteratively remove least significant feature based on model performance.

[!TIP]

PCA vs. LLE: PCA is linear, global; LLE is nonlinear, local. Use LLE for nonlinear manifolds (e.g., images of rotating object).

V. PROBABILISTIC & BAYESIAN METHODS

Probability in ML & Bayes' Theorem

• Fundamental Role: Handles uncertainty, provides framework for probabilistic models (e.g., Naïve Bayes, Bayesian Networks).

• Bayes' Theorem:

$$P(A|B) = \frac{P(B|A) P(A)}{P(B)}$$

• Posterior: $P(A|B)$ (updated belief after evidence).

• Prior: $P(A)$ (initial belief).

• Likelihood: $P(B|A)$ (probability of evidence given hypothesis).

• Evidence: $P(B)$ (normalizing constant).

• Application in Classification (Naïve Bayes): Compute $$\displaystyle P(\text{class}|\text{features}) \propto P(\text{class}) \prod P(\text{feature}|\text{class}) $$.

Naïve Bayes Classifier

• Assumptions:

  • Conditional Independence: Features independent given class: $$\displaystyle P(x_1,...,x_n|y) = \prod_i P(x_i|y) $$.

  • Often violated in practice but works well (e.g., text classification).

• Simplification: Avoids joint probability over all features; reduces parameters from exponential to linear.

• Performance in Noise: Robust to irrelevant features due to independence assumption; but correlated features can overcount evidence.

Bayesian Learning & Networks

• Bayesian Learning Perspective: Treat parameters as random variables; update beliefs (posterior) given data: $P(\theta|D) \propto P(D|\theta) P(\theta)$.

• Bayesian Belief Networks (BBN):

  • Structure: Directed acyclic graph (DAG). Nodes = random variables; edges = direct conditional dependencies.

  • Conditional Probability Tables (CPTs): For each node, specify $P(\text{node}|\text{parents})$.

  • Joint Probability Distribution: Product of CPTs: $$\displaystyle P(X_1,...,X_n) = \prod_i P(X_i | \text{Parents}(X_i)) $$.

  • Inference: Compute posterior given evidence (e.g., variable elimination, belief propagation).

• Example: Given CPTs, compute $$\displaystyle P(\text{car value}|\text{mileage=Lo, engine=Bad}) $$ by summing over hidden variables.

Bayesian Theorem - Worked Examples

• Medical Test:

  • $$\displaystyle P(\text{disease}) = 0.01 $$, $$\displaystyle P(\text{+}|\text{no disease}) = 0.05 $$ (false positive), $$\displaystyle P(\text{-}|\text{disease}) = 0.1 $$ (false negative).

  • Compute $$\displaystyle P(\text{disease}|\text{+}) = \frac{P(\text{+}|\text{disease})P(\text{disease})}{P(\text{+})} $$.

• From CPTs: Use joint probability and marginalization.

VI. REINFORCEMENT LEARNING (RL)

Fundamentals & Framework

• Key Components:

  • Agent: Learner/decision-maker.

  • Environment: World agent interacts with.

  • State $s$: Situation at time t.

  • Action $a$: Agent's choice.

  • Reward $r$: Immediate feedback from environment.

• Markov Decision Process (MDP):

  • Defined by $(S, A, P, R, \gamma)$:

    • $S$: state space.

    • $A$: action space.

    • $P(s'|s,a)$: transition probability.

    • $R(s,a)$: reward function.

    • $\gamma \in [0,1]$: discount factor (future rewards).

  • Markov Property: Future depends only on current state/action, not history.

• Difference from Supervised/Unsupervised:

  • No labeled dataset; sequential decisions; delayed rewards; exploration-exploitation trade-off.

RL Problem Solving

• Value Iteration:

  • Update state-value: $$\displaystyle V_{k+1}(s) \leftarrow \max_a \sum_{s'} P(s'|s,a) [R(s,a) + \gamma V_k(s')] $$.

  • Iterate until $$\displaystyle ||V_{k+1} - V_k|| < \epsilon $$.

  • Derive policy: $$\displaystyle \pi(s) = \arg\max_a \sum_{s'} P(s'|s,a)[R(s,a) + \gamma V(s')] $$.

• Policy Iteration:

  1. Policy Evaluation: Compute $$\displaystyle V^{\pi} $$ for current policy $\pi$ (solve linear system).

  2. Policy Improvement: Update $\pi$ greedily: $$\displaystyle \pi'(s) = \arg\max_a \sum_{s'} P(s'|s,a)[R(s,a) + \gamma V^{\pi}(s')] $$.

  3. Repeat until $\pi$ stable.

• Comparison:

  | Aspect | Value Iteration | Policy Iteration | |--------|----------------|------------------| | Steps | Value update + extraction | Evaluation + improvement | | Convergence | Often faster | Each evaluation may be costly | | Policy | Implicit | Explicit |

Q-Learning

• Off-policy TD Control: Learns optimal policy independent of behavior policy.

• Q-value (Action-Value): $Q(s,a)$ = expected cumulative discounted reward starting from $s$, taking $a$, then following optimal policy.

• Update Rule:

$$Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma \max_{a'} Q(s',a') - Q(s,a) \right]$$

• $\alpha$: learning rate.

• Uses max over actions → off-policy.

• Guiding Actions:

  • Exploitation: Choose $$\displaystyle a = \arg\max_a Q(s,a) $$.

  • Exploration: $\epsilon$-greedy: with prob $\epsilon$, random action.

• Algorithm (deterministic rewards/actions):

  1. Initialize $Q(s,a)$ arbitrarily.

  2. For each episode:

     • Initialize state $s$.

     • While $s$ not terminal:

       • Choose $a$ (e.g., $\epsilon$-greedy).

       • Take $a$, observe $r, s'$.

       • Update $Q(s,a)$.

       • $$\displaystyle s \leftarrow s' $$.

SARSA

• On-policy TD Control: Learns policy being followed.

• Update Rule:

$$Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma Q(s',a') - Q(s,a) \right]$$

where $a'$ is action actually taken in $s'$ (from current policy).

• Difference from Q-learning:

  • Q-learning uses $$\displaystyle \max_{a'} Q(s',a') $$ (optimistic, off-policy).

  • SARSA uses next action $a'$ from behavior policy (conservative, on-policy).

  • SARSA considers exploration in updates; Q-learning assumes optimal future.

Advanced RL Models

• Actor-Critic Model:

  • Actor: Policy network $\pi(a|s)$; selects actions.

  • Critic: Value network $V(s)$ or $Q(s,a)$; evaluates actions.

  • Interaction:

    1. Actor selects action $a \sim \pi(\cdot|s)$.

    2. Environment returns $r, s'$.

    3. Critic computes advantage $$\displaystyle A(s,a) = Q(s,a) - V(s) $$ (or TD error).

    4. Actor updates policy to increase probability of high-advantage actions.

    5. Critic updates value estimates (e.g., MSE to TD target).

  • Benefits: Lower variance than pure policy gradient; more sample efficient.

• Advanced Actor-Critic Models:

  • A2C (Advantage Actor-Critic): Synchronous; multiple workers, shared gradient.

  • A3C (Asynchronous Advantage Actor-Critic): Asynchronous workers; faster training.

  • PPO (Proximal Policy Optimization): Clipped objective; stable updates; state-of-the-art.

  • DDPG (Deep Deterministic Policy Gradient): For continuous actions; deterministic policy + Ornstein-Uhlenbeck noise.

RL Frameworks & Applications

• Popular Frameworks:

  • OpenAI Gym: Standard environments (Atari, MuJoCo).

  • TensorFlow Agents: TF-based library.

  • Stable Baselines3: PyTorch implementations of PPO, SAC, etc.

• Applications:

  • Games: AlphaGo (board games), DQN (Atari).

  • Robotics: Control, locomotion.

  • Autonomous driving: Decision-making.

  • Resource management: Data center cooling.

VII. ADVANCED & EMERGING TOPICS

Natural Language Processing (NLP)

• ML Applications:

  • Speech-to-text: ASR (Automatic Speech Recognition) – convert audio to text (e.g., DeepSpeech).

  • Speaker Identification: Verify/identify speaker from voice (embedding extraction).

  • Synthesis: Text-to-speech (TTS) – generate natural speech (e.g., Tacotron, WaveNet).

• NLP Pipeline: Tokenization → Embedding (Word2Vec, GloVe, BERT) → Model (RNN/LSTM, Transformer) → Output.

• Role of LSTMs/Transformers:

  • LSTMs: Handle sequences, long-term dependencies (used in early seq2seq).

  • Transformers: Self-attention; parallelization; power ChatGPT (GPT series), BERT.

Computer Vision (CV)

• ML Applications:

  • Image recognition (ResNet, EfficientNet).

  • Object detection (YOLO, Faster R-CNN).

  • Segmentation (U-Net).

• CNN Implementation Frameworks: TensorFlow/Keras: Conv2D, MaxPool2D, Flatten, Dense.

One-Shot Learning

• Definition: Learn from very few examples (often one per class).

• Difference from Traditional Supervised: Requires orders of magnitude less data; uses prior knowledge/metric learning.

• Beneficial Scenarios:

  • Rare event detection (e.g., new disease).

  • Face recognition with few images per person.

  • Customization tasks (personalized models).

• Techniques: Siamese networks, metric learning, data augmentation, transfer learning.

Self-Supervised Learning

• Concept: Create pretext tasks from unlabeled data to learn representations.

• Pretext Tasks:

  • Predict rotation of image.

  • Masked language modeling (BERT): predict masked words.

  • Contrastive learning (SimCLR): bring similar samples closer.

• Importance: Reduces need for labeled data; foundational for large models (GPT, BERT).

Generative AI

• Generative vs. Discriminative:

  • Generative: Models data distribution $p(x)$; generates new samples (GANs, VAEs, Diffusion).

  • Discriminative: Models $p(y|x)$; classifies/regresses.

• Key Architectures:

  • GANs (Generative Adversarial Networks): Generator vs. Discriminator; adversarial training.

  • VAEs (Variational Autoencoders): Latent variable model; learns smooth latent space.

  • Diffusion Models: Forward noising + reverse denoising; state-of-the-art for images (DALL-E 2, Stable Diffusion).

• Context: ChatGPT (autoregressive generative), DALL-E (diffusion).

Attention Models

• Brief: Mechanism allowing model to focus on relevant parts of input.

• Self-Attention: Compute attention scores between all positions in sequence; weighted sum.

• Transformers: Built on multi-head self-attention; enables parallel processing, long-range dependencies; foundation of modern NLP (GPT, BERT).

[!TIP]

Emerging Topics: Self-supervised learning and generative AI are heavily featured in recent papers (2024-2025). Be ready to define and give examples.