Optimization Techniques in Machine Leaning (AL-503 (C)) - Unit 2 Short Notes

UNIT 2: OPTIMIZATION TECHNIQUES & ARCHITECTURES IN DEEP LEARNING

I. FOUNDATIONS OF NEURAL NETWORKS & LEARNING PARADIGMS

A. Learning Paradigms

• Supervised Learning: Model learns a mapping from input features x to an output label y using labeled training data (x, y). The goal is to predict y for unseen x.

  • Examples: Image classification (CNN), Sentiment analysis (RNN/Transformer), Machine translation.

• Unsupervised Learning: Model learns patterns or structures from unlabeled data x alone. No explicit output labels are provided.

  • Examples: Clustering (K-Means), Dimensionality reduction (PCA, Autoencoders), Anomaly detection.

• Reinforcement Learning (RL): An agent learns to make decisions by performing actions in an environment to maximize a cumulative reward signal. It learns from interaction and delayed feedback, not from labeled (input, output) pairs.

  • Key Difference from Supervised: No correct answer is provided; the agent must discover rewarding actions through trial and error.

  • Deep RL Application: Deep Q-Network (DQN) uses a deep neural network to approximate the Q-value function, enabling RL in high-dimensional state spaces (e.g., playing Atari games from pixels).

[!TIP] Exam often asks for examples for each paradigm. Be ready to cite specific deep learning models (CNN for supervised, Autoencoder for unsupervised, DQN for RL).

B. Basic Neural Network Architecture

A simple feedforward neural network (FNN) or Multilayer Perceptron (MLP) consists of:

1. Input Layer: Receives the feature vector x.

2. Hidden Layer(s): One or more layers of neurons (units). Each neuron performs a linear transformation followed by a non-linear activation.

   • Linear Transformation: z = Wx + b, where W is the weight matrix, b is the bias vector.

3. Output Layer: Produces the final prediction ŷ. Its activation function depends on the task (e.g., Softmax for classification, Linear for regression).

Weights (W) and Biases (b) are the learnable parameters of the network.

C. Activation Functions

Definition: A mathematical function applied to the output of a neuron (z) to introduce non-linearity, enabling the network to learn complex, non-linear relationships.

[!TIP] ReLU is the most frequently examined. Be prepared to explain its significance (solves vanishing gradient for positive inputs, sparse activations) and its "Dying ReLU" problem.

II. THE TRAINING PROCESS: BACKPROPAGATION & OPTIMIZATION

A. Backpropagation Algorithm

Core Principle: Efficiently compute the gradient of the loss function L with respect to all network parameters (W, b) using the chain rule of calculus, propagating errors backward from the output layer to the input layer.

Step-by-Step Process:

1. Forward Pass: Input x is passed through the network layer-by-layer to compute the prediction ŷ and the loss L(ŷ, y).

2. Backward Pass (Gradient Flow):

   • Compute ∂L/∂ŷ at the output layer.

   • Apply the chain rule recursively to compute ∂L/∂z, ∂L/∂W, ∂L/∂b for each layer, moving backward.

   • The gradient at layer l depends on the gradient from layer l+1 and the local derivative of the activation function.

3. Weight Update: Use an optimizer (e.g., SGD, Adam) to update parameters: W ← W - η * ∂L/∂W, where η is the learning rate.

B. Optimization Algorithms

• Gradient Descent Variants:

  • Batch GD: Uses the entire training set to compute one gradient update. Slow, memory-intensive, but stable.

  • Stochastic GD (SGD): Uses one sample per update. Noisy updates, can escape shallow minima, but high variance.

  • Mini-batch GD: Uses a small random subset (mini-batch) of the training data. Standard practice—balances noise and computational efficiency.

• Key Concepts:

  • Learning Rate (η): Step size for weight updates. Too large → divergence; too small → slow convergence.

  • Epoch: One full pass through the entire training dataset.

  • Batch: Number of samples processed before a parameter update.

• Adaptive Learning Rate Optimizers: Adjust η per parameter based on historical gradients.

  • AdaGrad: Adapts learning rate by accumulating sum of squares of past gradients. θ ← θ - η * (g_t / (√(G_t) + ε)), where G_t = Σ_{i=1}^t g_i². Problem: Accumulated sum grows, causing aggressive, premature learning rate decay.

  • RMSProp: Fixes AdaGrad's decay by using an exponentially weighted moving average of squared gradients. G_t = β G_{t-1} + (1-β) g_t². Prevents aggressive decay.

  • Adam (Adaptive Moment Estimation): Most popular. Combines ideas from RMSProp and Momentum.

    1. Computes biased first-moment (mean) estimate: m_t = β₁ m_{t-1} + (1-β₁) g_t.

    2. Computes biased second-moment (uncentered variance) estimate: v_t = β₂ v_{t-1} + (1-β₂) g_t².

    3. Bias Correction (since m_0, v_0 initialized to 0): \hat{m}_t = m_t / (1 - β₁^t), \hat{v}_t = v_t / (1 - β₂^t).

    4. Update: θ ← θ - η * (\hat{m}_t / (√{\hat{v}_t} + ε)).

    \boxed{\theta_{t+1} = \theta_t - \eta \cdot \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}}

    [!TIP] Adam is a high-frequency exam topic. Know its components: Momentum (1st moment) + RMSProp (2nd moment) + Bias Correction.

C. Loss Functions & Model Evaluation

• Common Loss Functions:

  • Mean Squared Error (MSE): For regression. L = (1/N) Σ (y_i - ŷ_i)².

  • Cross-Entropy Loss: For classification.

    • Binary: L = -[y log(ŷ) + (1-y) log(1-ŷ)].

    • Multi-class: L = -Σ y_i log(ŷ_i).

• Evaluation Metrics:

  • Accuracy: (TP+TN)/Total.

  • Precision: TP/(TP+FP).

  • Recall: TP/(TP+FN).

  • F1-Score: Harmonic mean of Precision and Recall: F1 = 2 * (Precision*Recall)/(Precision+Recall).

• Overfitting vs. Underfitting:

  | | Underfitting | Overfitting | | :--- | :--- | :--- | | Definition | Model is too simple; fails to capture underlying pattern. | Model is too complex; memorizes training data noise. | | Training Error | High | Low | | Validation Error | High | High (gap between train & val is large) | | Diagnosis | Both train & val errors are high and close. | Low train error, high val error. | | Visual Cue | Model has high bias. | Model has high variance. |

[!TIP] Be able to diagnose from a learning curve plot (train/validation loss vs. epochs).

III. CHALLENGES IN TRAINING DEEP NETWORKS & REGULARIZATION

A. Vanishing & Exploding Gradient Problem

• Definition: During backpropagation, gradients can become extremely small (vanish) or extremely large (explode) as they are multiplied through many layers.

• Root Causes:

  • Deep Networks: Chain of many Jacobian matrices.

  • Activation Functions: Sigmoid/Tanh derivatives are ≤ 0.25. Multiplying many <1 values → vanishing.

  • Weight Initialization: Large initial weights → products >1 → exploding.

• Impact: Prevents effective learning in early layers (for vanishing) or causes unstable training/NaN loss (for exploding). Hinders learning long-range dependencies in RNNs.

• Mitigation Strategies:

  1. ReLU & variants: Derivative is 1 for positive inputs, mitigating vanishing gradient.

  2. Batch Normalization: Normalizes layer inputs, stabilizing distribution and reducing internal covariate shift, which helps gradient flow.

  3. Residual Connections (Skip Connections): In ResNet, output = F(x) + x. Gradient can flow directly through the identity connection, bypassing non-linearities.

  4. Proper Weight Initialization: e.g., He Initialization for ReLU (W ~ N(0, 2/n_in)), Xavier/Glorot for Tanh.

[!TIP] This is a classic 7-mark question. Structure answer: Definition → Cause (chain rule + activation) → Impact → Solutions (list & briefly explain 2-3).

B. Regularization for Generalization

• Dropout:

  • Mechanism: During training, randomly "drop" (set to zero) a fraction p of neurons in a layer for each forward/backward pass. The dropped neurons do not contribute to the output or gradient update.

  • How it Improves Generalization: Prevents co-adaptation of neurons. Effectively trains an ensemble of many sub-networks. At test time, all neurons are used, but outputs are scaled by (1-p) (or inverted dropout scales during training).

• Batch Normalization (Batch Norm):

  • Purpose: Stabilize and accelerate training by reducing internal covariate shift (change in layer input distributions during training).

  • How it Works (per mini-batch):

    1. Compute batch mean μ_B and variance σ_B² of activations x.

    2. Normalize: \hat{x} = (x - μ_B) / √(σ_B² + ε).

    3. Scale and shift: y = γ \hat{x} + β, where γ (scale) and β (shift) are learnable parameters.

  • Advantages:

    • Allows use of higher learning rates.

    • Reduces sensitivity to weight initialization.

    • Has a slight regularization effect (noise from batch statistics).

• L1/L2 Regularization (Weight Decay):

  • Add a penalty term to the loss: L_total = L_data + λ * Ω(W).

  • L2 (Ridge): Ω(W) = ||W||²_2 = Σ W_ij². Encourages small, distributed weights.

  • L1 (Lasso): Ω(W) = ||W||_1 = Σ |W_ij|. Encourages sparsity (some weights become exactly zero).

• Early Stopping: Monitor validation loss; stop training when it starts to increase (or stop improving) to prevent overfitting.

C. Data Preprocessing

• Normalization (Min-Max Scaling): Rescales features to a fixed range, typically [0, 1].

  \boxed{x_{\text{norm}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}}

  • When to use: When features have hard boundaries (e.g., pixel intensities). Sensitive to outliers.

• Standardization (Z-score Normalization): Transforms data to have zero mean and unit variance.

  \boxed{x_{\text{std}} = \frac{x - \mu}{\sigma}}

  • When to use: When data follows a Gaussian-like distribution. Less sensitive to outliers. Generally preferred for neural networks.

[!TIP] Know the formulas for both. Normalization bounds values; standardization centers data.

IV. CONVOLUTIONAL NEURAL NETWORKS (CNNs) FOR SPATIAL DATA

A. Core CNN Concepts & Architecture

• Convolutional Layer:

  • Filter/Kernel: A small matrix (e.g., 3x3, 5x5) that slides (convolves) over the input.

  • Feature Map/Activation Map: Output of a filter applied to the input. Detects specific features (edges, textures).

  • Stride (s): Number of pixels the filter moves each step. Larger stride → smaller output.

  • Padding (p): Adding zeros around the input border to control output spatial size. 'same' padding keeps size; 'valid' (no padding) reduces size.

  • Output Size Formula: (W - K + 2P)/S + 1, where W=input width, K=kernel size.

• Pooling Layer (Sub-sampling):

  • Purpose: Reduce spatial dimensions (width/height), increase receptive field, provide translation invariance.

  • Max Pooling: Takes maximum value in a window (most common). Preserves dominant features.

  • Average Pooling: Takes average value in a window. Smoothes features.

B. Data Formats for CNNs

[!TIP] "Develop a table" is a direct past question. Memorize the common shapes.

V. RECURRENT NEURAL NETWORKS (RNNs) FOR SEQUENTIAL DATA

A. Standard RNNs

• Architecture & Working: Has a recurrent connection (hidden state h_t) that acts as memory. At timestep t:

  \boxed{h_t = f(W_{xh} x_t + W_{hh} h_{t-1} + b_h)}, \quad \hat{y}t = g(W{hy} h_t + b_y)

  where f is typically tanh or ReLU, g is output activation (e.g., softmax).

• Comparison with FNN:

  • Handles Variable-Length Sequences: Same parameters applied across timesteps.

  • Captures Temporal Dependencies: Hidden state h_t summarizes past information.

  • Limitation: Short-term memory. Struggles with long-range dependencies due to vanishing/exploding gradients in Backpropagation Through Time (BPTT).

B. Advanced RNN Architectures: LSTM & GRU

• Long Short-Term Memory (LSTM):

  • Key Innovation: Cell State (C_t)—a "conveyor belt" running through the entire sequence, allowing gradients to flow easily.

  • Gates (using σ for sigmoid, tanh for candidate):

    1. Forget Gate: f_t = σ(W_f · [h_{t-1}, x_t] + b_f). Decides what to remove from C_{t-1}.

    2. Input Gate: i_t = σ(W_i · [h_{t-1}, x_t] + b_i). \tilde{C}_t = \tanh(W_C · [h_{t-1}, x_t] + b_C). Decides what new information to store.

    3. Cell State Update: C_t = f_t * C_{t-1} + i_t * \tilde{C}_t.

    4. Output Gate: o_t = σ(W_o · [h_{t-1}, x_t] + b_o). h_t = o_t * \tanh(C_t).

  • Advantages: Explicit memory cell, robust to vanishing gradients, excels at learning long-term dependencies.

• Gated Recurrent Unit (GRU):

  • Simpler variant with two gates (Reset r_t, Update z_t). Merges cell state and hidden state.

  • z_t = σ(W_z · [h_{t-1}, x_t]) (update gate), r_t = σ(W_r · [h_{t-1}, x_t]) (reset gate).

  • \tilde{h}_t = \tanh(W_h · [r_t * h_{t-1}, x_t]), h_t = (1-z_t)*h_{t-1} + z_t*\tilde{h}_t.

  • Often faster to train, performance comparable to LSTM.

C. Deep Recurrent Networks

• Stacked RNN/LSTM Architectures: Multiple recurrent layers stacked on top of each other. Lower layers learn short-term features, higher layers learn long-term abstractions.

• Challenges: Exacerbated vanishing/exploding gradients in BPTT over many layers and timesteps. Requires careful initialization, gradient clipping, and often layer normalization.

[!TIP] LSTM is a guaranteed question. Draw and explain the cell state and three gates (Forget, Input, Output). Contrast with GRU's simplicity.

VI. AUTOENCODERS & GENERATIVE MODELS

A. Autoencoders (AEs)

• Architecture: Encoder z = f(x) maps input x to a low-dimensional latent code z. Decoder ŷ = g(z) reconstructs x from z. Trained to minimize reconstruction loss L(x, g(f(x))).

• Purpose:

  • Dimensionality Reduction: Non-linear alternative to PCA.

  • Feature Learning: Latent space z learns meaningful representations.

  • Denoising: Train with corrupted input x', reconstruct clean x.

• Regularization in AEs:

  • Sparse Autoencoder: Adds a sparsity penalty (e.g., KL divergence) to the activation of hidden units, forcing only a few neurons to fire.

  • Contractive Autoencoder: Adds penalty on the Frobenius norm of the Jacobian of the encoder, making the encoder resistant to small input perturbations.

• When to Use AEs vs. PCA/SVD: Use AEs for complex, non-linear data (images, text). PCA/SVD are linear and fail to capture intricate structures.

B. Generative Models: GANs vs. VAEs

[!TIP] GANs vs. VAEs is a top-priority comparison. Use the table above. Key points: GAN = adversarial, sharp samples, unstable; VAE = probabilistic, stable, blurry but structured latent space.

VII. ADVANCED & SPECIALIZED TOPICS

A. Recursive Neural Networks (RecursiveNNs)

• Architecture: Tree-structured network. Same set of weights is applied recursively in a bottom-up manner over a hierarchical structure (e.g., a parse tree).

• Application: Process hierarchical data like sentences (parse trees), images (region trees), or logical expressions.

• Contrast with RNNs: RNNs process sequential data (linear chain). RecursiveNNs process tree-structured data, capturing compositional semantics. Computationally more complex.

B. Logistic Regression

• Detailed Working:

  1. Model: For binary classification, predicts probability P(y=1|x) = σ(z) = 1/(1+e^{-z}), where z = w^T x + b.

  2. Decision Boundary: w^T x + b = 0. Predict 1 if σ(z) ≥ 0.5 (i.e., z ≥ 0).

  3. Training: Use Maximum Likelihood Estimation (MLE).

     • Likelihood: L(w) = Π_{i=1}^N P(y_i|x_i)^{y_i} (1-P(y_i|x_i))^{1-y_i}.

     • Log-Likelihood: l(w) = Σ [ y_i log(ŷ_i) + (1-y_i) log(1-ŷ_i) ].

     • Loss: Binary Cross-Entropy: L(w) = -1/N Σ l(w). Minimize via Gradient Descent.

  4. Connection to NN: A single-layer neural network with a sigmoid activation in the output layer is exactly a logistic regression model.

C. Deep Dream & Feature Visualization

• Concept: An algorithm that amplifies the patterns a neural network "sees" in an input image. It performs gradient ascent on the input image itself to maximize the activation of a specific layer (or neuron) in the network.

• Process:

  1. Start with a random or input image.

  2. Forward pass to compute activation A of target layer.

  3. Compute gradient of A w.r.t. input image: ∂A/∂x.

  4. Update image: x ← x + η * (∂A/∂x) (often with some regularization/normalization).

  5. Repeat.

• Role: A tool for understanding what features a network has learned (interpretability). Generates surreal, dream-like images with repeating patterns (e.g., eyes, dogs) that the network is sensitive to.

D. Reinforcement Learning & Policy Optimization

• Markov Decision Process (MDP): Formalization of RL problem with states S, actions A, transition probabilities P(s'|s,a), reward function R(s,a,s'), discount factor γ.

• Value Iteration: Dynamic Programming method. Iteratively updates the state-value function V(s) until convergence: V_{k+1}(s) = max_a Σ_{s'} P(s'|s,a)[R(s,a,s') + γ V_k(s')]. Requires full model knowledge.

• Policy Iteration: Alternates between:

  1. Policy Evaluation: Compute V^π for current policy π.

  2. Policy Improvement: Update policy greedily: π'(s) = argmax_a Σ_{s'} P(s'|s,a)[R(s,a,s') + γ V^π(s')].

• Q-Learning & Deep Q-Networks (DQN):

  • Q-Learning: Learns action-value function Q(s,a) off-policy. Update: Q(s,a) ← Q(s,a) + α [r + γ max_{a'} Q(s',a') - Q(s,a)].

  • DQN: Uses a deep neural network to approximate Q(s,a; θ). Key innovations: Experience Replay (break correlations) and Fixed Q-Targets (stable targets).

• Advanced DQN Algorithms:

  • Double DQN: Decouples action selection and evaluation to reduce overestimation bias.

  • Dueling DQN: Separately estimates value V(s) and advantage A(s,a) of each action, then combines: Q(s,a) = V(s) + A(s,a) - mean_{a'} A(s,a').

  • Least Squares Policy Iteration (LSPI): A policy iteration method using least-squares regression to fit the Q-function from samples, avoiding sequential updates.