UNIT 5: Deep Learning – Exam-Focused Short Notes

I. Foundations & Core Concepts of Deep Learning

What is Deep Learning?

• Subset of machine learning using deep neural networks (multiple hidden layers) to learn hierarchical feature representations automatically.

• Relationship: AI ⊃ Machine Learning ⊃ Deep Learning.

• Historical Milestones:

  • 1943: McCulloch-Pitts neuron model.

  • 1958: Perceptron (Rosenblatt).

  • 1986: Backpropagation popularized (Rumelhart, Hinton, Williams).

  • 2006: Deep belief networks (Hinton) – breakthrough in training deep nets.

  • 2012: AlexNet wins ImageNet – triggers deep learning revolution (GPU + ReLU + Dropout).

Biological vs. Artificial Neural Networks

[!TIP] Exam Focus: Current DL models lack energy efficiency, lifelong learning, and causal reasoning compared to the brain.

Representation Power & Depth

• Depth refers to number of hidden layers.

• Why depth matters:

  • Hierarchical feature learning: Shallow layers learn edges/textures; deeper layers learn complex concepts (objects, semantics).

  • Representational efficiency: Deep nets can represent certain functions exponentially more efficiently than shallow nets (e.g., parity function).

  • Expressive power: With non-linear activations (ReLU), a deep network can approximate any continuous function (Universal Approximation Theorem holds for width-bounded deep nets too).

• Increasing layers:

  • Pros: Better accuracy on complex tasks (ImageNet, translation), better generalization with enough data.

  • Cons: Vanishing/exploding gradients, overfitting, computational cost.

  • Empirical observation: Beyond certain depth, accuracy saturates or degrades (degradation problem) – solved by ResNet.

[!TIP] Key Formula: For a function like parity on n bits, a shallow net needs exponential width, but a deep net needs only O(n) depth.

II. Fundamental Neural Network Architectures & Training

Single-Layer vs. Multilayer Perceptron (MLP)

• Single-Layer Perceptron:

  • Architecture: Input → Weighted sum → Step activation.

  • Limitation: Can only learn linearly separable patterns (fails on XOR).

• Multilayer Perceptron (MLP):

  • Architecture: Input layer → ≥1 hidden layers (non-linear activation) → Output layer.

  • Overcomes XOR: Hidden layer creates non-linear decision boundaries.

  • Feedforward Neural Network (FFN): Synonym for MLP; signals flow one direction.

  • Implementation: Forward pass: $$\displaystyle z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)} $$, $$\displaystyle a^{(l)} = f(z^{(l)}) $$.

Activation Functions

[!TIP] ReLU in CNNs: Default choice due to non-saturation for positive inputs, accelerating convergence.

Backpropagation Algorithm

Goal: Compute gradients $$\displaystyle \frac{\partial L}{\partial W^{(l)}} $$ for all layers to update weights via gradient descent. Steps:

1. Forward pass: Compute activations layer-by-layer, store $$\displaystyle z^{(l)}, a^{(l)} $$.

2. Compute loss: $$\displaystyle L = \mathcal{L}(\hat{y}, y) $$.

3. Backward pass:

   • Output layer: $$\displaystyle \delta^{(L)} = \nabla_a L \odot f'(z^{(L)}) $$.

   • Hidden layers: $$\displaystyle \delta^{(l)} = (W^{(l+1)})^T \delta^{(l+1)} \odot f'(z^{(l)}) $$.

4. Gradients: $$\displaystyle \frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T $$, $$\displaystyle \frac{\partial L}{\partial b^{(l)}} = \delta^{(l)} $$.

5. Update: $$\displaystyle W^{(l)} \leftarrow W^{(l)} - \eta \frac{\partial L}{\partial W^{(l)}} $$.

Backpropagation Through Time (BPTT):

• Unfolds RNN through time steps, applies backpropagation to the unfolded graph.

• Challenges: Vanishing/exploding gradients over long sequences, memory-intensive.

[!TIP] Common Pitfall: Forgetting to zero gradients before each backward pass in PyTorch/TensorFlow (optimizer.zero_grad()).

Weight Initialization

Why important? Poor init leads to vanishing/exploding gradients, slow convergence.

• Random (Uniform/Normal): Often too large/small.

• Xavier/Glorot (for tanh/sigmoid): $$\displaystyle W \sim \mathcal{U}\left(-\sqrt{\frac{6}{n_{in}+n_{out}}}, \sqrt{\frac{6}{n_{in}+n_{out}}}\right) $$ or $$\displaystyle \mathcal{N}\left(0, \sqrt{\frac{2}{n_{in}+n_{out}}}\right) $$.

• He Initialization (for ReLU): $$\displaystyle W \sim \mathcal{N}\left(0, \sqrt{\frac{2}{n_{in}}}\right) $$.

• Rule of thumb: Use He init with ReLU, Xavier with tanh.

[!TIP] Bias initialization: Often set to 0, but for ReLU, set to small positive (e.g., 0.01) to avoid dead neurons.

III. Optimization for Deep Networks

Gradient Descent Variants

Advanced Optimizers

[!TIP] Default choice: Adam ($$\displaystyle \eta=0.001 $$, $$\displaystyle \beta_1=0.9 $$, $$\displaystyle \beta_2=0.999 $$) works well for most tasks. Use SGD with momentum for CNNs if fine-tuning.

Normalization Techniques

• Batch Normalization (BN):

  • How: For each mini-batch, normalize layer inputs: $$\displaystyle \hat{x}^{(k)} = \frac{x^{(k)} - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} $$, then scale/shift: $$\displaystyle y^{(k)} = \gamma \hat{x}^{(k)} + \beta $$.

  • Why: Reduces internal covariate shift (distribution changes in layer inputs during training), allows higher learning rates, acts as regularizer.

  • Advantages: Faster convergence, improved gradient flow, slight regularization effect.

• Normalization vs. Standardization (data preprocessing):

  • Normalization: Rescale to [0,1] (min-max).

  • Standardization: Zero mean, unit variance ($$\displaystyle \frac{x - \mu}{\sigma} $$). BN is batch-wise standardization during training.

IV. Regularization & Mitigating Training Challenges

Overfitting & Underfitting

• Overfitting: Model memorizes training data, poor generalization (high train acc, low test acc).

• Underfitting: Model too simple, fails to capture pattern (low train & test acc).

Regularization Techniques

[!TIP] Dropout in practice: Use p=0.5 for fully connected layers, p=0.2-0.3 for CNNs.

Vanishing & Exploding Gradients

• Vanishing Gradient:

  • Cause: Sigmoid/tanh saturate (derivative →0), chain product of small numbers.

  • Impact: Early layers learn very slowly, network fails to learn long-range dependencies (RNNs).

• Exploding Gradient:

  • Cause: Large weights, deep nets, chain product of large numbers.

  • Impact: Unstable training, NaN weights.

• Mitigation Strategies:

  • ReLU family: No saturation for $$\displaystyle x>0 $$.

  • Batch Normalization: Stabilizes layer inputs.

  • Residual Connections (ResNet): Add identity mapping, gradients flow directly.

  • Proper initialization: He/Xavier.

  • Gradient clipping: Cap gradient norm (e.g., $\|\nabla L\| \leq 1$).

  • LSTM/GRU: For RNNs, additive cell state mitigates vanishing gradient.

[!TIP] LSTM solves vanishing gradient via cell state $$\displaystyle C_t $$: $$\displaystyle C_t = f_t \odot C_{t-1} + i_t \odot \tilde{C}_t $$. The additive update allows gradients to flow unchanged through time.

V. Convolutional Neural Networks (CNNs)

Core Concepts & Operations

• Convolution Operation:

  • Filter/kernel $K$ (size $k \times k$) slides over input $I$, computes dot product: $$\displaystyle (I * K)_{i,j} = \sum_m \sum_n I_{i+m, j+n} K_{m,n} $$.

  • Captures spatial features: Edges, textures, patterns via learned filters.

  • Stride: Step size of filter. Larger stride → smaller output.

  • Padding:

    • Valid: No padding, output shrinks.

    • Same: Pad so output size = input size (padding = $(k-1)/2$).

  • Dilation: Spacing between filter elements, increases receptive field without extra parameters.

• Number of parameters: For $$\displaystyle C_{in} $$ input channels, $$\displaystyle C_{out} $$ output channels, kernel size $k$: $$\displaystyle C_{in} \times C_{out} \times k^2 + C_{out} $$ (biases).

Pooling Layers

• Role: Downsampling, reduces spatial size, introduces translation invariance, reduces parameters.

• Max Pooling: Output = max value in window. Preserves dominant features.

• Average Pooling: Output = mean of window. Smoothes features.

• Common: $2 \times 2$ window, stride 2 → halves spatial dimensions.

CNN Architectures (Historical & Modern)

[!TIP] ResNet skip connection: Allows gradient to flow directly via identity mapping, mitigating vanishing gradient. $F(x)$ is residual mapping learned.

Applications & Specialized Uses

• Image Recognition: Classification (ImageNet), detection (R-CNN, YOLO), segmentation (FCN, U-Net).

• Structured Output:

  • Semantic Segmentation: Pixel-wise classification (FCN, SegNet).

  • Object Detection: Bounding boxes (Faster R-CNN).

  • Image Captioning: CNN encoder + RNN decoder.

• Data Formats Compatible with CNNs:

  • 2D Images: Standard (height × width × channels).

  • 1D Signals: Audio, text (as 1D sequences, use 1D convs).

  • 3D Volumes: Medical scans (CT/MRI), video (time as 3rd dimension).

  • Spectrograms: Time-frequency representations (2D).

VI. Recurrent Neural Networks (RNNs) & Sequence Modeling

Vanilla RNNs

• Architecture: Hidden state $$\displaystyle h_t = f(W_{xh} x_t + W_{hh} h_{t-1} + b_h) $$, output $$\displaystyle y_t = g(W_{hy} h_t + b_y) $$.

• For sequence data: Processes one token at a time, maintains memory via $$\displaystyle h_{t-1} $$.

• Comparison with FFN: FFN assumes independent inputs; RNN shares parameters across time, handles variable length.

• Challenges:

  • Vanishing/exploding gradients in BPTT.

  • Short-term memory: Struggles with long-range dependencies.

Gated Architectures

Long Short-Term Memory (LSTM)

• Cell State $$\displaystyle C_t $$: "Conveyor belt" for long-term memory.

• Gates (sigmoid activations, 0–1):

  • Forget gate $$\displaystyle f_t = \sigma(W_f \cdot [h_{t-1}, x_t] + b_f) $$: What to discard from $$\displaystyle C_{t-1} $$.

  • Input gate $$\displaystyle i_t = \sigma(W_i \cdot [h_{t-1}, x_t] + b_i) $$: What to store from candidate $$\displaystyle \tilde{C}_t = \tanh(W_C \cdot [h_{t-1}, x_t] + b_C) $$.

  • Output gate $$\displaystyle o_t = \sigma(W_o \cdot [h_{t-1}, x_t] + b_o) $$: What to output from $$\displaystyle C_t $$.

• Equations:

$$ \begin{aligned} C_t &= f_t \odot C_{t-1} + i_t \odot \tilde{C}_t \\ h_t &= o_t \odot \tanh(C_t) \end{aligned} $$

• Advantages: Solves vanishing gradient via additive state update, can learn long-range dependencies.

Gated Recurrent Unit (GRU)

• Simpler than LSTM: Update gate $$\displaystyle z_t $$ (combines forget/input), reset gate $$\displaystyle r_t $$.

• No separate cell state; hidden state $$\displaystyle h_t $$ is both memory and output.

• Equations:

$$ \begin{aligned} z_t &= \sigma(W_z \cdot [h_{t-1}, x_t]) \\ r_t &= \sigma(W_r \cdot [h_{t-1}, x_t]) \\ \tilde{h}_t &= \tanh(W_h \cdot [r_t \odot h_{t-1}, x_t]) \\ h_t &= (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t \end{aligned} $$

• Comparison: GRU fewer parameters, often similar performance; LSTM more expressive for long sequences.

[!TIP] When to use LSTM vs GRU? LSTM for very long sequences (e.g., speech), GRU for smaller datasets or faster training.

Encoding & Decoding in RNNs

• Encoding: RNN processes input sequence $$\displaystyle x_1, ..., x_T $$, final hidden state $$\displaystyle h_T $$ (or all states) as context vector representing input.

• Decoding: Another RNN (often with attention) generates output sequence $$\displaystyle y_1, ..., y_{T'} $$ conditioned on context.

• Challenges:

  • Fixed-length context vector bottleneck (especially for long inputs).

  • Solution: Attention mechanism (Bahdanau, 2015) – decoder attends to all encoder states dynamically.

Deep RNNs & Applications

• Deep RNN: Stack multiple RNN layers (e.g., 3 layers). Lower layers learn low-level temporal features, higher layers learn abstract patterns.

• Applications:

  • Image Processing: Image captioning (CNN encoder + RNN decoder), video classification (3D convs or frame-by-frame RNN).

  • Speech Recognition: Acoustic modeling (Bidirectional LSTM).

  • Machine Translation: Seq2seq with attention (Transformer now dominant).

• PixelRNN: Autoregressive model for image generation, generates pixels sequentially row-by-row (1D RNN) or in 2D (Diagonal BiLSTM). Captures spatial dependencies.

Backpropagation Through Time (BPTT)

• Unfold RNN for T time steps, treat as deep FFN with T layers, apply backpropagation.

• Challenges:

  • Vanishing/exploding gradients over many time steps.

  • Mitigation: LSTM/GRU, gradient clipping, proper initialization.

  • Computational cost: $O(T)$ memory for hidden states; use truncated BPTT (backprop only for last k steps).

VII. Autoencoders & Unsupervised Learning

Basic Autoencoder Architecture

• Encoder: $$\displaystyle z = f_\theta(x) $$ maps input $x$ to latent code $z$ (bottleneck).

• Decoder: $$\displaystyle \hat{x} = g_\phi(z) $$ reconstructs $x$ from $z$.

• Objective: Minimize reconstruction loss $\mathcal{L}(x, \hat{x})$ (MSE for continuous, cross-entropy for binary).

• Applications:

  • Dimensionality reduction: Non-linear alternative to PCA.

  • Denoising: Train on corrupted inputs, reconstruct clean.

  • Feature learning: Pre-training for supervised tasks.

  • Anomaly detection: High reconstruction error for anomalies.

Regularization in Autoencoders

• Why regularize? Without constraints, autoencoder learns identity function ($$\displaystyle z=x $$), useless.

• Regularization techniques:

  • Sparse autoencoder: Add sparsity penalty on latent activations (e.g., KL divergence to target sparsity $\rho$).

  • Contractive autoencoder: Penalize Jacobian norm $$\displaystyle \|\frac{\partial z}{\partial x}\|_F^2 $$ to make mapping locally stable.

  • Denoising autoencoder: Train to reconstruct clean input from corrupted version (e.g., add Gaussian noise, mask pixels).

  • Variational autoencoder (VAE): Probabilistic, imposes prior on latent space (see Generative Models).

When to use Autoencoders over PCA/SVD?

• Autoencoders:

  • Can learn non-linear manifolds (PCA linear).

  • Flexible architectures (CNNs for images, RNNs for sequences).

  • Can be regularized (sparse, denoising) for specific properties.

• PCA/SVD:

  • Linear, closed-form solution, computationally efficient.

  • Interpretable principal components.

  • Use autoencoders when: Data lies on non-linear manifold, need complex feature extraction, have large dataset and computational resources.

VIII. Generative Models

Variational Autoencoders (VAEs)

• Fundamental idea: Learn probabilistic latent variable model $$\displaystyle p_\theta(x) = \int p_\theta(x|z) p(z) dz $$.

• Encoder: Outputs parameters of approximate posterior $$\displaystyle q_\phi(z|x) $$ (usually Gaussian: mean $$\displaystyle \mu_\phi(x) $$, variance $$\displaystyle \sigma_\phi^2(x) $$).

• Reparameterization trick: Sample $$\displaystyle z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon $$, $\epsilon \sim \mathcal{N}(0,I)$. Allows gradient flow through sampling.

• Loss (ELBO):

$$ \mathcal{L} = \mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x|z)] - D_{KL}(q_\phi(z|x) \| p(z)) $$

Reconstruction term + KL regularization (forces latent space to match prior $\mathcal{N}(0,I)$).

• Comparison with standard autoencoder:

  • VAE: probabilistic, continuous latent space, can generate new samples by sampling $z \sim p(z)$.

  • Standard AE: deterministic, latent space may have holes, generation less reliable.

Generative Adversarial Networks (GANs)

• Architecture:

  • Generator $G(z)$: Takes random noise $$\displaystyle z \sim p_z $$, generates fake data $$\displaystyle \tilde{x} = G(z) $$.

  • Discriminator $D(x)$: Classifies real ($x$) vs fake ($\tilde{x}$).

• Training: Minmax game:

$$ \min_G \max_D V(D,G) = \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))] $$

• Comparison with VAEs:

  | Aspect | VAE | GAN | |------------------|----------------------------------|----------------------------------| | Training | Stable (likelihood-based) | Unstable (adversarial) | | Samples | Blurry but diverse | Sharp but may mode-collapse | | Likelihood | Explicit (approximate) | Not directly computable | | Inference | Encoder provides latent code | No encoder (inversion needed) | | Use case | Representation learning, interpolation | High-fidelity generation |

[!TIP] Choose VAE for disentangled representations, interpolation; choose GAN for photorealistic images (if stable training).

Hybrid & Advanced Generative Models

• VAE-GAN Hybrid:

  • Use VAE encoder-decoder, but replace pixel-wise reconstruction loss with GAN discriminator.

  • How: Decoder generates $\tilde{x}$, discriminator judges realism, VAE loss includes adversarial term.

  • Benefits: Combines VAE's structured latent space with GAN's sharp samples.

• Deep Dream:

  • Role: Generate surreal, hallucinatory images by maximizing activations of chosen layers.

  • Process: Gradient ascent on input image to increase target layer's activation (often with regularization to keep image natural).

  • Applications: Visualizing learned features, art generation.

• Autoregressive Models:

  • NADE (Neural Autoregressive Distribution Estimator): Factorizes $$\displaystyle p(x) = \prod_{i=1}^d p(x_i | x_{<i}) $$, each conditional modeled by MLP with masked weights.

  • MADE (Masked Autoencoder for Distribution Estimation): Applies random masks to autoencoder to ensure autoregressive property, efficient training.

Other Generative Frameworks

• Restricted Boltzmann Machines (RBMs):

  • Bipartite graph: visible layer $v$, hidden layer $h$, no intra-layer connections.

  • Energy-based model: $$\displaystyle E(v,h) = -b^T v - c^T h - v^T W h $$.

  • Training: Contrastive divergence (CD-k) approximation of gradient.

• Gibbs Sampling:

  • Used in RBMs for sampling: alternate between sampling $h$ given $v$ and $v$ given $h$.

  • Useful for persistent CD to improve samples.

• Deep Belief Networks (DBNs):

  • Stack of RBMs, trained greedily layer-by-layer (unsupervised pre-training).

  • Historical significance before backpropagation of deep nets was reliable.

IX. Reinforcement Learning with Deep Learning

Fundamentals

• Deep RL vs Supervised Learning:

  • Supervised: Fixed dataset, i.i.d., clear labels.

  • RL: Agent interacts with environment, delayed rewards, sequential decisions, exploration-exploitation trade-off.

• Markov Decision Process (MDP):

  • Components: States $S$, actions $A$, transition $P(s'|s,a)$, reward $R(s,a,s')$, discount $\gamma \in [0,1]$.

  • Policy $\pi(a|s)$: Agent's behavior.

  • Value function: $$\displaystyle V^\pi(s) = \mathbb{E}[\sum_{t=0}^\infty \gamma^t R_{t+1} | s_0=s, \pi] $$.

  • Q-function: $$\displaystyle Q^\pi(s,a) = \mathbb{E}[\sum_{t=0}^\infty \gamma^t R_{t+1} | s_0=s, a_0=a, \pi] $$.

• Policy Iteration vs Value Iteration:

  | Aspect | Policy Iteration | Value Iteration | |---------------------|-----------------------------------------------|----------------------------------------------| | Steps | Policy evaluation → Policy improvement (repeat) | Iterative Bellman update: $$\displaystyle V_{k+1}(s) = \max_a \sum_{s'} P(s'|s,a)[R + \gamma V_k(s')] $$ | | Convergence | Finite policies, finite MDP → finite steps | Contraction mapping → converges to $$\displaystyle V^* $$ | | Complexity | Policy eval costly (solve linear system) | Simpler per iteration, but may need many iterations | | Use case | Small MDPs | Large MDPs, dynamic programming |

Deep Q-Learning

• Q-learning: Model-free, learns $$\displaystyle Q^*(s,a) $$ via Bellman update:

$$ Q(s,a) \leftarrow Q(s,a) + \alpha \left[ r + \gamma \max_{a'} Q(s',a') - Q(s,a) \right] $$

• Deep Q-Network (DQN):

  • Use deep NN to approximate $Q(s,a; \theta)$.

  • Key tricks:

    • Experience replay: Store transitions $(s,a,r,s')$ in buffer, sample mini-batches to break correlations.

    • Target network: Separate network $Q'$ with parameters $$\displaystyle \theta^- $$ updated slowly ($$\displaystyle \theta^- \leftarrow \tau \theta + (1-\tau)\theta^- $$) to stabilize targets.

• Advanced Algorithms:

  • Double DQN: Decouple action selection and evaluation to reduce overestimation bias:

$$ y = r + \gamma Q(s', \arg\max_{a'} Q(s',a'; \theta); \theta^-) $$

• Dueling DQN: Separate streams for state value $V(s)$ and action advantages $A(s,a)$: $$\displaystyle Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a') $$.

Policy Gradient Methods

• Idea: Directly optimize policy $$\displaystyle \pi_\theta(a|s) $$ by gradient ascent on expected return $$\displaystyle J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[\sum_t \gamma^t r_t] $$.

• REINFORCE: $$\displaystyle \nabla J(\theta) = \mathbb{E}[\sum_t \nabla \log \pi_\theta(a_t|s_t) G_t] $$, where $$\displaystyle G_t $$ is return.

• Least Squares Policy Iteration (LSPI):

  • Uses linear function approximation for Q-function: $$\displaystyle Q(s,a) = \phi(s,a)^T w $$.

  • Fits $w$ via least squares on samples from experience replay, using Bellman residual minimization.

  • Advantage: Off-policy, data-efficient; limitation: linear approximation limits expressiveness.

X. Special Topics & Applications

Natural Language Processing (NLP) with Deep Learning

• Definition: NLP enables machines to understand, generate, and interact with human language.

• 4 Key Elements/Tasks:

  1. Syntax: Grammar, part-of-speech tagging, parsing.

  2. Semantics: Meaning, word sense disambiguation.

  3. Pragmatics: Context, intent, dialogue.

  4. Discourse: Cohesion, coherence across sentences.

• Role of RNNs/LSTMs/GRUs: Process sequences (words) one by one, capture contextual dependencies. Used in translation, sentiment analysis, named entity recognition.

• Transformers (implied): Now dominate NLP via self-attention (parallel, long-range dependencies).

Dimensionality Reduction & Linear Algebra

• PCA (Principal Component Analysis):

  • Goal: Find orthogonal axes (principal components) maximizing variance.

  • Steps: Center data $X$, compute covariance $$\displaystyle C = \frac{1}{n} X^T X $$, eigen-decomposition $$\displaystyle C = V \Lambda V^T $$. Top k eigenvectors give projection.

• SVD (Singular Value Decomposition):

  • Decompose matrix $$\displaystyle X = U \Sigma V^T $$, where $U, V$ orthogonal, $\Sigma$ diagonal (singular values).

• Relationship:

  • For centered data $X$, PCA eigenvectors = right singular vectors $V$ (columns of $V$ are principal directions).

  • SVD of $X$: $$\displaystyle X = U \Sigma V^T $$ → PCA projection: $$\displaystyle X V_k = U_k \Sigma_k $$ (top k components).

• How SVD reduces dimensionality: Keep top k singular values/vectors: $$\displaystyle X \approx U_k \Sigma_k V_k^T $$, reduces storage and noise.

Pruning & Model Compression

• Unit/Neuron Pruning:

  • Need: Deploy deep nets on edge devices (mobile, IoT) with limited compute/memory.

  • Method: Remove entire neurons/filters with small importance scores (e.g., L1 norm of weights, activation magnitude).

  • Process: Train network, prune unimportant units, fine-tune. Iterate.

  • Benefits: Reduces model size, inference time, energy consumption.

Graphical Models

• Directed Graphical Models (Bayesian Networks):

  • Nodes = random variables, directed edges = causal dependencies.

  • Factorization: $$\displaystyle p(x) = \prod_i p(x_i | \text{parents}(x_i)) $$.

  • Example: Naïve Bayes (class → features).

• Markov Networks (Undirected Graphical Models):

  • Undirected edges, factors over cliques: $$\displaystyle p(x) = \frac{1}{Z} \prod_C \phi_C(x_C) $$.

  • Example: Ising model, CRFs for sequence labeling.

  • Inference: Often harder than Bayesian nets (no topological order), but can model symmetric relationships.

Final Exam Strategy:

✅ Memorize key formulas: Backpropagation, Adam, LSTM gates, VAE loss, Q-learning update.

✅ Draw diagrams: LSTM cell, ResNet block, CNN architecture flow, VAE/GAN frameworks.

✅ Compare/contrast: LSTM vs GRU, VAE vs GAN, Batch GD vs SGD, PCA vs SVD.

✅ Applications: Link architectures to tasks (CNN→images, RNN→sequences, Autoencoder→denoising).

✅ Historical context: Know AlexNet (2012), ResNet (2015), Transformer (2017 – implied in NLP).

[!TIP] Past Paper Pattern: 7-mark questions often ask "Explain...", "Compare...", "Discuss...". Structure answers: Definition → Architecture/Mechanism → Advantages/Disadvantages → Applications. Use diagrams where possible.