Optimization Techniques in Machine Leaning (AL-503 (C)) - Unit 4 Short Notes

UNIT 4: Optimization Techniques in Deep Learning – Short Notes

I. Foundations of Machine Learning and Neural Networks

Learning Paradigms

• Supervised Learning: Learn a mapping from inputs to outputs using labeled data.

  • Examples: Image classification (input: image, output: label), regression (predict house prices).

• Unsupervised Learning: Find hidden patterns in unlabeled data.

  • Examples: Clustering (customer segmentation), dimensionality reduction (PCA).

• Reinforcement Learning (RL): Agent learns by interacting with an environment to maximize cumulative reward.

  • Contrast with Supervised: No labeled dataset; uses reward signals; sequential decision-making.

Basic Models

• Logistic Regression: A linear model for binary classification using sigmoid activation. Outputs probability $$\displaystyle P(y=1|x) = \frac{1}{1+e^{-(w^Tx+b)}} $$. Can be viewed as a single-layer neural network.

• Feed-forward Neural Networks (MLP): Composed of input layer, hidden layers (fully connected), output layer. Each neuron applies: $$\displaystyle z = Wx + b $$, $$\displaystyle a = f(z) $$ where $f$ is activation function.

• Representation Power: MLPs with non-linear activations (e.g., sigmoid) are universal approximators—can approximate any continuous function given sufficient hidden units.

[!TIP] Exam often asks: "Explain logistic regression as a neural network." Emphasize the sigmoid output and binary cross-entropy loss.

II. Activation Functions and Network Initialization

Activation Functions

Weight Initialization

• Xavier (Glorot) Initialization: For tanh/sigmoid. $$\displaystyle W \sim \mathcal{U}\left(-\sqrt{\frac{6}{n_{in}+n_{out}}}, \sqrt{\frac{6}{n_{in}+n_{out}}}\right) $$ or normal with std $$\displaystyle \sqrt{\frac{2}{n_{in}+n_{out}}} $$. Keeps variance stable across layers.

• He Initialization: For ReLU. $$\displaystyle W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{in}}}\right) $$. Compensates for ReLU's half-rectification.

[!TIP] Common mistake: Using Xavier with ReLU leads to vanishing/exploding signals. Always match init to activation.

III. Training Neural Networks: Backpropagation and Optimization

Backpropagation Algorithm

1. Forward Pass: Compute output and loss $L$.

2. Backward Pass: Apply chain rule to compute $$\displaystyle \frac{\partial L}{\partial W} $$ for each layer.

   • For layer $l$: $$\displaystyle \delta^{(l)} = \frac{\partial L}{\partial z^{(l)}} = \left((W^{(l+1)})^T \delta^{(l+1)}\right) \odot f'(z^{(l)}) $$

   • Gradient: $$\displaystyle \frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T $$

3. Weight Update: $$\displaystyle W^{(l)} \leftarrow W^{(l)} - \eta \frac{\partial L}{\partial W^{(l)}} $$

Optimization Algorithms

Vanishing/Exploding Gradients

• Causes: Deep networks, sigmoid/tanh saturations (vanishing); large weights/loss (exploding).

• Mitigation:

  • Use ReLU (vanishing gradient only for -ve inputs).

  • Batch Normalization (stabilizes activations).

  • Gradient Clipping (exploding): set threshold $\theta$, if $$\displaystyle \|\nabla L\| > \theta $$, scale down.

  • Proper initialization (He/Xavier).

  • Residual Connections (ResNet) for vanishing.

Loss Functions

• Mean Squared Error (MSE): $$\displaystyle L = \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2 $$. For regression.

• Cross-Entropy:

  • Binary: $$\displaystyle L = -\frac{1}{N}\sum [y \log(\hat{y}) + (1-y)\log(1-\hat{y})] $$.

  • Multi-class: $$\displaystyle L = -\frac{1}{N}\sum_{i=1}^N \sum_{j=1}^C y_{ij} \log(\hat{y}_{ij}) $$.

IV. Regularization Techniques to Prevent Overfitting

Overfitting vs Underfitting

• Overfitting: Model learns noise, high training accuracy, low validation/test accuracy. High variance.

• Underfitting: Model too simple, fails to capture pattern, low training & validation accuracy. High bias.

Explicit Regularization

• L1 Regularization (Lasso): Adds $$\displaystyle \lambda \sum |w| $$ to loss. Promotes sparsity.

• L2 Regularization (Weight Decay): Adds $$\displaystyle \frac{\lambda}{2} \sum w^2 $$. Encourages small weights. Update becomes $$\displaystyle w \leftarrow w - \eta(\frac{\partial L}{\partial w} + \lambda w) $$.

• Dropout: Randomly "drop" (set to 0) neurons during training with probability $p$. At test time, use all neurons but scale weights by $(1-p)$. Forces network to learn redundant representations.

Implicit Regularization

• Batch Normalization (BN): Normalizes layer inputs to zero mean, unit variance per mini-batch. Then scales/shifts: $$\displaystyle y = \gamma \hat{x} + \beta $$.

  • Advantages: Reduces internal covariate shift, allows higher learning rates, acts as regularizer (noise from batch stats).

• Early Stopping: Monitor validation loss; stop training when it starts increasing.

Data Preprocessing

[!TIP] BN is applied before activation function in a layer. Dropout is not used with BN in some modern architectures (e.g., ResNet) due to redundant regularization.

V. Convolutional Neural Networks (CNNs)

Architecture Components

• Convolutional Layer: Applies filters/kernels to extract local features. Output size: $$\displaystyle O = \frac{W - K + 2P}{S} + 1 $$, where $W$=input width, $K$=kernel size, $P$=padding, $S$=stride.

• Pooling Layer (subsampling): Reduces spatial dimensions.

  • Max Pooling: Takes max in window. Preserves dominant features, translation invariant.

  • Average Pooling: Takes average. Smoothes features.

• Fully Connected (FC) Layer: At end for classification/regression.

Design Elements

• Padding: Adds zeros around input to control output size.

  • Same padding: Output size = input size (when stride=1).

  • Valid padding: No padding; output shrinks.

• Strides: Step size of filter. Larger stride → smaller output, less computation.

• Filter Design: Number of filters = depth of output volume; each filter learns different features (edges, textures).

ReLU in CNNs

• Significance: Introduces non-linearity, avoids vanishing gradient in deep CNNs, computationally cheap. Typically applied after conv layer (before pooling).

Data Formats Compatible with CNNs

[!TIP] For 1D CNNs (text/audio), kernel slides over sequence dimension. For 2D (images), over height & width.

VI. Recurrent Neural Networks (RNNs) and Sequence Modeling

Basic RNN Architecture

• Structure: Hidden state $$\displaystyle h_t = f(W_{hh} h_{t-1} + W_{xh} x_t + b_h) $$, output $$\displaystyle y_t = g(W_{hy} h_t + b_y) $$.

• Handles Sequential Data: Maintains memory via $$\displaystyle h_t $$; processes one timestep at a time, sharing weights across time.

Training RNNs: Backpropagation Through Time (BPTT)

• Unfold network through time steps, apply standard backpropagation.

• Challenges:

  • Vanishing/Exploding Gradients: Due to repeated multiplication of Jacobians (especially with sigmoid/tanh). Hinders learning long-range dependencies.

  • Mitigation: Use LSTM/GRU, gradient clipping, proper initialization, skip connections.

Advanced RNN Variants

• Long Short-Term Memory (LSTM):

  • Cell State $$\displaystyle c_t $$: "Highway" for long-term information.

  • Gates (sigmoid + tanh):

    1. Forget Gate $$\displaystyle f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) $$: What to discard from $$\displaystyle c_{t-1} $$.

    2. Input Gate $$\displaystyle i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) $$, $$\displaystyle \tilde{c}_t = \tanh(W_c \cdot [h_{t-1}, x_t] + b_c) $$: What to store.

    3. Update Cell State: $$\displaystyle c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t $$.

    4. Output Gate $$\displaystyle o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) $$, $$\displaystyle h_t = o_t \odot \tanh(c_t) $$.

  • Advantages: Explicit memory cell, gates control information flow → learns long dependencies.

• Gated Recurrent Unit (GRU): Simpler than LSTM. Combines forget & input gates into update gate $$\displaystyle z_t $$, and has reset gate $$\displaystyle r_t $$. No separate cell state; $$\displaystyle h_t $$ is hidden state. Computationally faster, often similar performance.

• Deep RNNs: Stack multiple RNN layers; each layer processes sequence at different abstraction levels.

Other Architectures

• Recursive Neural Networks (RvNN): Tree-structured, not sequential. Applied to NLP (syntax parsing), images (scene parsing).

Applications

• NLP: Machine translation, text generation.

• Time Series: Stock prediction, weather forecasting.

• Speech: Speech recognition, synthesis.

[!TIP] LSTM vs GRU: LSTM has 3 gates, GRU has 2. LSTM may be better for long sequences; GRU faster. Both solve vanishing gradient problem.

VII. Autoencoders and Dimensionality Reduction

Autoencoder Fundamentals

• Structure: Encoder $$\displaystyle h = f(x) $$ maps input to latent code; Decoder $$\displaystyle \hat{x} = g(h) $$ reconstructs input.

• Purpose: Learn efficient data representations (dimensionality reduction), denoising, generative pretraining.

• Trained to minimize reconstruction loss (e.g., MSE for continuous, cross-entropy for binary).

Types of Autoencoders

Comparison: Autoencoders vs PCA/SVD

[!TIP] VAE is not just an autoencoder; it's a generative model with probabilistic latent space. Standard autoencoder latent space may be discontinuous.

VIII. Generative Models: GANs and VAEs

Generative Adversarial Networks (GANs)

• Architecture:

  • Generator $G$: Takes random noise $$\displaystyle z \sim p_z(z) $$, generates fake data $G(z)$.

  • Discriminator $D$: Classifies real vs fake: $D(x)$ outputs probability real.

• Training: Minimax game: $$\displaystyle \min_G \max_D V(D,G) = \mathbb{E}_{x\sim p_{data}}[\log D(x)] + \mathbb{E}_{z\sim p_z}[\log(1-D(G(z)))] $$.

  • Alternate: Train $D$ to maximize $\log D(x) + \log(1-D(G(z)))$, then train $G$ to minimize $\log(1-D(G(z)))$ (or maximize $\log D(G(z))$).

• Challenges: Mode collapse, training instability, difficult to evaluate.

Variational Autoencoders (VAEs)

• Probabilistic Approach: Learns $p(x)$ by introducing latent variable $z$: $$\displaystyle p(x) = \int p(x|z)p(z)dz $$.

• Training: Maximize ELBO (Evidence Lower Bound): $$\displaystyle \mathcal{L} = \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}(q(z|x) \| p(z)) $$.

  • Encoder approximates posterior $q(z|x)$; decoder is $p(x|z)$.

  • Latent space is continuous and structured (by Gaussian prior).

• Advantage: Stable training, tractable likelihood, interpolatable latent space.

Comparison: GANs vs VAEs

Auto-regressive Models

• NADE (Neural Autoregressive Distribution Estimator): Models $$\displaystyle p(x) = \prod_{i=1}^D p(x_i | x_{<i}) $$ using a neural network with masked connections (ensures autoregressive property).

• MADE (Masked Autoencoder for Distribution Estimation): Extends NADE to deep networks by applying fixed masks to layers; efficient training.

Deep Belief Networks (DBNs)

• Architecture: Stack of Restricted Boltzmann Machines (RBMs). Each RBM trained greedily (unsupervised), then fine-tuned with backprop.

• Training: Contrastive Divergence for each RBM layer.

• Significance: Early deep learning model (pre-2010); showed deep networks could be trained layer-wise.

IX. Optimization in Reinforcement Learning

Markov Decision Processes (MDPs)

• Components:

  • States $s \in \mathcal{S}$, Actions $a \in \mathcal{A}$.

  • Transition probabilities $P(s'|s,a)$.

  • Reward function $R(s,a,s')$.

  • Discount factor $\gamma \in [0,1]$.

• Goal: Find policy $\pi(a|s)$ maximizing expected discounted return $$\displaystyle G_t = \sum_{k=0}^\infty \gamma^k r_{t+k+1} $$.

Dynamic Programming (DP) Methods (Assume known model $P, R$)

• Value Iteration:

  • Initialize $$\displaystyle V_0(s)=0 $$.

  • Iterate: $$\displaystyle V_{k+1}(s) = \max_a \sum_{s'} P(s'|s,a)[R(s,a,s') + \gamma V_k(s')] $$.

  • Converges to optimal $$\displaystyle V^* $$; policy $$\displaystyle \pi^*(s) = \arg\max_a \sum_{s'} P(s'|s,a)[R + \gamma V^*(s')] $$.

• Policy Iteration:

  1. Policy Evaluation: Compute $$\displaystyle V^\pi $$ for current $\pi$ (solve linear system or iterative).

  2. Policy Improvement: $$\displaystyle \pi' = \arg\max_a \sum_{s'} P(s'|s,\pi(s))[R + \gamma V^\pi(s')] $$.

  3. Repeat until $$\displaystyle \pi' = \pi $$.

  • Comparison: Policy iteration often converges in fewer iterations but each iteration costly (solving linear system). Value iteration simpler per iteration but may need more iterations.

Q-learning and Deep Variants

• Q-learning: Model-free. Learns action-value $Q(s,a)$ via: $$\displaystyle Q(s,a) \leftarrow Q(s,a) + \alpha [r + \gamma \max_{a'} Q(s',a') - Q(s,a)] $$. Off-policy.

• Deep Q-Network (DQN):

  • Uses deep NN to approximate $Q(s,a;\theta)$.

  • Stabilization tricks:

    • Experience Replay: Store transitions $(s,a,r,s')$ in buffer; sample mini-batches to break correlations.

    • Target Network: Separate target network $$\displaystyle Q'(s,a;\theta^-) $$ for computing TD target; update slowly: $$\displaystyle \theta^- \leftarrow \tau \theta + (1-\tau)\theta^- $$.

• Advanced Algorithms:

  • Double DQN: Decouples action selection and evaluation to reduce overestimation bias. Uses online net to select $$\displaystyle \arg\max_a Q(s',a) $$, target net to evaluate: $$\displaystyle y = r + \gamma Q'(s', \arg\max_a Q(s',a;\theta); \theta^-) $$.

  • Dueling DQN: Estimates $V(s)$ and $A(s,a)$ separately, then $$\displaystyle Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a') $$. Learns better value estimates.

Least Squares Methods

• Least Squares Policy Iteration (LSPI):

  • Uses linear function approximation for $$\displaystyle Q(s,a) = \phi(s,a)^T \theta $$.

  • Solves Bellman equation in least-squares sense: $$\displaystyle A\theta = b $$, where $$\displaystyle A = \Phi^T D (\Phi - \gamma \Phi') $$, $$\displaystyle b = \Phi^T D r $$.

  • $\Phi$: feature matrix for all state-action pairs; $D$: diagonal importance sampling weights.

  • Advantage: No need for step-size tuning; sample-efficient.

X. Advanced Optimization and Efficiency Techniques

Batch Normalization (Detailed)

• Working (during training):

  1. For each mini-batch, compute mean $$\displaystyle \mu_B = \frac{1}{m}\sum_{i=1}^m x_i $$ and variance $$\displaystyle \sigma_B^2 = \frac{1}{m}\sum (x_i - \mu_B)^2 $$.

  2. Normalize: $$\displaystyle \hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} $$.

  3. Scale and shift: $$\displaystyle y_i = \gamma \hat{x}_i + \beta $$ (learnable parameters).

• During Inference: Use running averages of mean/variance computed during training.

• Advantages Beyond Regularization:

  • Reduces internal covariate shift.

  • Allows higher learning rates.

  • Acts as a regularizer (noise from batch statistics).

  • Slightly reduces need for Dropout.

Unit Pruning

• Need: Network compression for deployment on edge devices (mobile, IoT); reduces memory, computation, energy.

• Methods:

  • Magnitude-based: Remove weights with small absolute values (below threshold).

  • Structured Pruning: Remove entire neurons/filters (units) based on importance (e.g., L1-norm of weights).

  • Iterative: Prune → fine-tune → repeat.

Representation Learning

• Concept: Automatically discover useful features/representations from raw data (e.g., pixels, words) without manual feature engineering.

• Importance: Core of deep learning; enables end-to-end learning; hierarchical features (low-level edges → high-level objects).

Deep Dream

• Working: Start with input image/noise; forward through network; choose a layer; modify image to maximize activations of certain filters/channels via gradient ascent on input (not weights). Repeats iteratively.

• Role: Visualization of what neurons "see"; generates surreal, artistic images; helps understand learned features.

Hardware and Computational Optimization

• GPU Implementation for Scalability:

  • Parallelize matrix operations (convolutions, matrix multiplies) across thousands of cores.

  • Randomized SVD: Approximates SVD using random projections; faster for large matrices; used in PCA, recommendation systems. GPU accelerates random matrix multiplication and QR decomposition steps.

Directed Graphical Models (Basics)

• Definition: Probabilistic model with directed acyclic graph (DAG); nodes = random variables, edges = conditional dependencies.

• Relation to Neural Networks:

  • Neural nets can be seen as directed graphical models with deterministic nodes (no explicit probability).

  • VAEs combine neural nets (encoder/decoder) with probabilistic graphical models (latent variable $z$).

  • Bayesian neural networks extend NNs with priors on weights → graphical model.

XI. Evaluation and Practical Considerations

Diagnosing Model Performance

• Overfitting: Training loss << validation loss. Solutions: more data, regularization (dropout, L2), reduce model capacity, early stopping.

• Underfitting: Both training & validation loss high. Solutions: increase model capacity, train longer, feature engineering, reduce regularization.

• Validation Strategies:

  • Hold-out validation set.

  • k-Fold Cross-Validation (for small datasets).

  • Stratified sampling for imbalanced data.

Hyperparameter Tuning

• Learning Rate: Most critical. Use schedules (step decay, cosine annealing) or adaptive optimizers (Adam). Typical range: $$\displaystyle 10^{-4} $$ to $$\displaystyle 10^{-1} $$.

• Batch Size: Larger → smoother gradients, but may generalize worse. Common: 32, 64, 128.

• Network Depth/Width: Start shallow, increase depth if underfitting; increase width if capacity insufficient.

• Methods: Grid search, random search, Bayesian optimization (Hyperopt, Optuna).

Domain Applications

• Image Recognition: CNNs (ResNet, EfficientNet).

• Natural Language Processing: RNNs/LSTMs, Transformers (BERT, GPT).

• Robotics: RL (DQN, PPO) for control; CNNs for perception.

• Time Series: RNNs, LSTMs, Temporal CNNs.

• Recommendation Systems: Autoencoders, matrix factorization (neural collaborative filtering).

Exam Focus Summary

• Backpropagation: Chain rule steps, weight update formula.

• Activation Functions: ReLU derivation, pros/cons; when to use which.

• Vanishing Gradient: Causes (deep nets, saturating activations), solutions (ReLU, BN, residual connections).

• Batch Normalization: Steps (normalize, scale, shift), advantages.

• Dropout: Mechanism (random dropout, scale at test), why it regularizes.

• RNNs/LSTMs: BPTT, LSTM gates (forget, input, output, cell state) with equations.

• CNNs: Padding (same/valid), pooling (max/average), stride effect; data formats table.

• Optimization: Adam update steps (momentum, RMSProp combo); compare SGD, AdaGrad, RMSProp, Adam.

• Autoencoders: VAE vs standard AE; VAE loss (reconstruction + KL).

• GANs vs VAEs: Key differences table (training stability, sample quality, latent space).

• RL: MDP components; value vs policy iteration; DQN (experience replay, target net); Double DQN idea.

• Regularization: L1 vs L2; early stopping; normalization vs standardization.

[!TIP] Always connect theory to practice: e.g., "Why ReLU in deep CNNs?" → avoids vanishing gradient, sparse activations, computationally efficient. "Why batch norm after conv but before activation?" → normalizes pre-activation, stabilizes distribution to activation function.

\boxed{\text{Focus on definitions, formulas, comparisons, and "why" explanations. Practice derivations (backprop, LSTM gates, Adam).}}