Deep Learning (AL-503 (B)) - Unit 3 Short Notes

UNIT 3: Deep Learning – Comprehensive Short Notes

I. Foundations and Core Concepts

A. Historical Progression of Deep Learning

[!TIP] Exam Focus: Key milestones like perceptron (1957), backpropagation (1986), AlexNet (2012), ResNet (2015).

• 1950s-60s: Perceptron model, initial excitement.

• 1970s-80s: AI winter; Minsky & Papert highlight XOR limitation.

• 1986: Backpropagation popularized (Rumelhart, Hinton, Williams).

• 2006: Deep Belief Networks (Hinton) – pre-training breakthrough.

• 2012: AlexNet wins ImageNet; GPU training, ReLU, Dropout.

• 2015: ResNet (He et al.) – residual connections enable >100 layers.

• 2017: Transformer architecture (Vaswani et al.) – revolutionizes NLP.

B. AI vs. ML vs. Deep Learning

C. Biological vs. Artificial Neural Networks

D. Learning Paradigms

• Supervised: Labeled data; e.g., classification, regression.

• Unsupervised: Unlabeled data; e.g., clustering, dimensionality reduction, generative modeling.

• Reinforcement: Agent learns via rewards/penalties; e.g., game playing, robotics.

E. Single-Layer Perceptron & XOR Problem

• Model: $$\displaystyle y = f\left(\sum_{i=1}^n w_i x_i + b\right) $$, where $f$ is step function.

• Limitation: XOR is not linearly separable → perceptron cannot learn it.

• Implication: Necessitates multilayer networks and non-linear activation.

F. Multilayer Perceptron (MLP)

1. Architecture: Input layer → [Hidden layers] → Output layer. Fully connected.

2. Universal Approximation Theorem: A single hidden layer with sufficient neurons can approximate any continuous function.

3. Implementation Steps:

   • Define layers (input, hidden, output sizes).

   • Choose activation functions.

   • Forward pass: Compute layer-by-layer outputs.

   • Backward pass: Compute gradients via backpropagation.

   • Update weights using optimizer.

G. Activation Functions

[!TIP] Common Pitfall: Using sigmoid/tanh in deep networks → vanishing gradients. Prefer ReLU variants.

H. Logistic Regression as Linear Classifier

• Model: $$\displaystyle P(y=1|\mathbf{x}) = \frac{1}{1 + e^{-(\mathbf{w}^T\mathbf{x} + b)}} $$.

• Loss: Binary cross-entropy: $$\displaystyle \mathcal{L} = -\frac{1}{N}\sum_{i=1}^N [y_i \log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i)] $$.

• Role: Building block for binary classification; output layer of MLP for binary tasks.

II. Training Neural Networks: Optimization and Regularization

A. Backpropagation Algorithm

1. Concept: Compute gradient of loss w.r.t. each weight using chain rule.

2. Computational Graph: Forward pass computes and stores intermediate values; backward pass computes gradients.

3. Step-by-Step:

   • Forward pass: compute loss $\mathcal{L}$.

   • Backward pass: $$\displaystyle \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(l)}} = \delta^{(l)} (\mathbf{a}^{(l-1)})^T $$, where $$\displaystyle \delta^{(l)} = (\mathbf{W}^{(l+1)})^T \delta^{(l+1)} \odot f'(\mathbf{z}^{(l)}) $$.

   • Update: $$\displaystyle \mathbf{W}^{(l)} \leftarrow \mathbf{W}^{(l)} - \eta \frac{\partial \mathcal{L}}{\partial \mathbf{W}^{(l)}} $$.

4. Applications: Training MLPs, CNNs, RNNs (via BPTT).

B. Weight Initialization

C. Optimization Algorithms

[!TIP] Exam Tip: Adam is default choice; SGD with momentum often generalizes better with tuning.

D. Loss Functions

• Regression: Mean Squared Error (MSE): $$\displaystyle \frac{1}{N}\sum_{i=1}^N (y_i - \hat{y}_i)^2 $$.

• Classification: Cross-Entropy:

  • Binary: $$\displaystyle -\frac{1}{N}\sum [y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)] $$.

  • Multi-class: $$\displaystyle -\frac{1}{N}\sum_{i=1}^N \sum_{j=1}^C y_{ij} \log \hat{y}_{ij} $$.

E. Overfitting & Underfitting

• Overfitting: Low train error, high test error. Remedies: More data, regularization (L1/L2, Dropout, BatchNorm), early stopping, data augmentation.

• Underfitting: High train/test error. Remedies: Larger model, more features, reduce regularization, longer training.

F. Regularization Techniques

1. L1/L2 (Weight Decay): Add $$\displaystyle \lambda \|\mathbf{W}\|_1 $$ or $$\displaystyle \lambda \|\mathbf{W}\|_2^2 $$ to loss. L1 sparsifies, L2 shrinks.

2. Dropout: Randomly set fraction $p$ of neurons to 0 during training. Ensemble effect; prevents co-adaptation.

3. Early Stopping: Monitor validation loss; stop when it increases.

4. Data Augmentation: Transform training data (rotate, flip, crop) to increase effective dataset size.

G. Batch Normalization

1. Mechanism: For each mini-batch $\mathcal{B}$:

$$\hat{x}^{(k)} = \frac{x^{(k)} - \mu_\mathcal{B}}{\sqrt{\sigma_\mathcal{B}^2 + \epsilon}}, \quad y^{(k)} = \gamma \hat{x}^{(k)} + \beta$$

where $$\displaystyle \mu_\mathcal{B}, \sigma_\mathcal{B}^2 $$ are batch stats; $\gamma, \beta$ are learnable.

2. Advantages:

   • Reduces internal covariate shift.

   • Allows higher learning rates.

   • Regularization effect (noise from batch stats).

   • Mildly reduces need for Dropout.

H. Vanishing & Exploding Gradients

• Causes: Deep networks; repeated multiplication of gradients <1 (vanish) or >1 (explode); especially in RNNs (BPTT).

• Mitigation:

  • ReLU activations.

  • BatchNorm.

  • Residual connections (ResNet).

  • Gradient clipping (set max norm).

  • Proper initialization (He, Xavier).

I. Learning Rate Scheduling

• Strategies: Step decay, exponential decay, cosine annealing.

• Adaptive LR: Adam, RMSProp adjust per-parameter LR.

• Warm-up: Start with small LR, increase gradually (common in Transformers).

III. Convolutional Neural Networks (CNNs)

A. Convolution Operation

• Filter/Kernel: Learnable weight matrix $$\displaystyle \mathbf{W} \in \mathbb{R}^{k \times k \times C_{in} \times C_{out}} $$.

• Stride ($s$): Step size; larger stride → smaller output.

• Padding:

  • Valid: No padding; output size: $$\displaystyle \left\lfloor \frac{W - k}{s} \right\rfloor + 1 $$.

  • Same: Pad to maintain spatial size; output size = input size if $$\displaystyle s=1 $$.

• Output Size Formula:

$$H_{out} = \left\lfloor \frac{H_{in} + 2P - K}{S} \right\rfloor + 1, \quad W_{out} = \left\lfloor \frac{W_{in} + 2P - K}{S} \right\rfloor + 1$$

• Hierarchical Features: Early layers → edges/textures; deeper layers → object parts/objects.

B. Pooling Layers

C. CNN Architecture Design

• Typical Sequence: CONV → Activation (ReLU) → POOL → [repeat] → FC → Output.

• Design Choices:

  • Number of filters: Increases with depth (e.g., 64 → 128 → 256).

  • Filter size: Often 3×3 or 1×1 (inception); smaller filters reduce parameters, increase depth.

  • Depth: More layers → higher capacity, but need residual connections for training.

D. Landmark CNN Architectures

E. Structured Output in CNNs

• Semantic Segmentation: Pixel-wise classification (e.g., U-Net, FCN).

• Object Detection: Bounding box + class (e.g., Faster R-CNN, YOLO).

• Approach: Replace final FC layers with convolutional/transpose layers to produce spatial output.

F. CNN Data Formats

G. Applications

• Image classification, object detection, medical image analysis (tumor detection), face recognition, autonomous driving.

IV. Recurrent Neural Networks (RNNs) and Sequence Modeling

A. RNN Fundamentals

• Architecture: Hidden state $$\displaystyle \mathbf{h}_t = f(\mathbf{h}_{t-1}, \mathbf{x}_t) $$; parameters shared across time.

• Variable-Length Sequences: Process each time step until end-of-sequence token.

• vs. Feedforward: Temporal dynamics; same weights applied sequentially.

• Applications: NLP (translation), time series forecasting, speech recognition.

B. Backpropagation Through Time (BPTT)

1. Unfolding: Expand RNN for $T$ time steps → deep feedforward network.

2. Gradient Computation: Chain rule through time; $$\displaystyle \frac{\partial \mathcal{L}}{\partial \mathbf{W}} = \sum_{t=1}^T \frac{\partial \mathcal{L}}{\partial \mathbf{h}_t} \frac{\partial \mathbf{h}_t}{\partial \mathbf{W}} $$.

3. Challenges: Vanishing/exploding gradients over long sequences; computational cost $O(T)$.

C. Long Short-Term Memory (LSTM)

1. Cell Structure:

   • Cell state $$\displaystyle \mathbf{c}_t $$: "Highway" for long-term memory.

   • Forget gate $$\displaystyle \mathbf{f}_t = \sigma(\mathbf{W}_f [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f) $$: What to discard from $$\displaystyle \mathbf{c}_{t-1} $$.

   • Input gate $$\displaystyle \mathbf{i}_t = \sigma(\mathbf{W}_i [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i) $$: What new info to store.

   • Candidate cell $$\displaystyle \tilde{\mathbf{c}}_t = \tanh(\mathbf{W}_c [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_c) $$.

   • Update cell state: $$\displaystyle \mathbf{c}_t = \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tilde{\mathbf{c}}_t $$.

   • Output gate $$\displaystyle \mathbf{o}_t = \sigma(\mathbf{W}_o [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o) $$.

   • Hidden state: $$\displaystyle \mathbf{h}_t = \mathbf{o}_t \odot \tanh(\mathbf{c}_t) $$.

2. Mitigates Vanishing Gradients: Additive update of $$\displaystyle \mathbf{c}_t $$ → constant error flow.

3. Variants: Peephole connections (gates see cell state).

D. Gated Recurrent Unit (GRU)

1. Architecture:

   • Update gate $$\displaystyle \mathbf{z}_t = \sigma(\mathbf{W}_z [\mathbf{h}_{t-1}, \mathbf{x}_t]) $$: Blend old/new state.

   • Reset gate $$\displaystyle \mathbf{r}_t = \sigma(\mathbf{W}_r [\mathbf{h}_{t-1}, \mathbf{x}_t]) $$: Control candidate state.

   • Candidate hidden: $$\displaystyle \tilde{\mathbf{h}}_t = \tanh(\mathbf{W}_h [\mathbf{r}_t \odot \mathbf{h}_{t-1}, \mathbf{x}_t]) $$.

   • Hidden state: $$\displaystyle \mathbf{h}_t = (1-\mathbf{z}_t) \odot \mathbf{h}_{t-1} + \mathbf{z}_t \odot \tilde{\mathbf{h}}_t $$.

2. vs. LSTM: Fewer parameters (no cell state, 2 gates vs. 3); similar performance; faster.

E. Deep RNNs

• Stacking: Multiple RNN layers; lower layers capture short-term patterns, higher layers capture long-term.

• Challenge: Gradient flow through both depth and time → harder to train.

F. Encoding & Decoding with RNNs (Seq2Seq)

1. Encoder: Processes input sequence $$\displaystyle \mathbf{x}_1,...,\mathbf{x}_T $$ → final hidden state $$\displaystyle \mathbf{h}_T $$ (context vector).

2. Decoder: Initialized with $$\displaystyle \mathbf{h}_T $$; generates output sequence $$\displaystyle \mathbf{y}_1,...,\mathbf{y}_{T'} $$ autoregressively.

3. Applications: Machine translation, text summarization, image captioning.

G. Advanced RNN Variants

• Bidirectional RNNs (BiRNNs): Forward + backward passes; each $$\displaystyle \mathbf{h}_t $$ sees past and future.

• Attention Mechanism: Allow decoder to "attend" to all encoder states, not just final context; crucial for long sequences.

H. Applications of Deep RNNs

• NLP: Language modeling, sentiment analysis, named entity recognition.

• Image Processing: PixelRNN – generates images pixel-by-pixel (autoregressive).

• Time Series: Forecasting, anomaly detection, stock prediction.

V. Generative Models

A. Autoencoders

1. Basic Architecture:

   • Encoder: $$\displaystyle \mathbf{z} = f_\phi(\mathbf{x}) $$ (maps input to latent code).

   • Decoder: $$\displaystyle \hat{\mathbf{x}} = g_\theta(\mathbf{z}) $$ (reconstructs from latent).

   • Loss: Reconstruction loss (MSE for images, cross-entropy for binary).

2. Types:

   • Sparse Autoencoder: Add sparsity penalty (e.g., KL divergence to target activation) on $\mathbf{z}$.

   • Contractive Autoencoder: Penalize Jacobian norm $$\displaystyle \|\frac{\partial \mathbf{z}}{\partial \mathbf{x}}\|_F^2 $$ → robust features.

   • Denoising Autoencoder: Corrupt input $\tilde{\mathbf{x}}$ (add noise), reconstruct $\mathbf{x}$; learns invariant features.

   • Variational Autoencoder (VAE):

     • Probabilistic: Encoder outputs distribution $$\displaystyle q_\phi(\mathbf{z}|\mathbf{x}) = \mathcal{N}(\boldsymbol{\mu}, \boldsymbol{\sigma}^2) $$.

     • Reparameterization Trick: $$\displaystyle \mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot \boldsymbol{\epsilon}, \boldsymbol{\epsilon} \sim \mathcal{N}(0,\mathbf{I}) $$.

     • Loss: $$\displaystyle \mathcal{L} = \text{ReconLoss} + \text{KL}(q_\phi(\mathbf{z}|\mathbf{x}) \| p(\mathbf{z})) $$.

3. Regularization: Prevents trivial identity mapping; encourages meaningful latent space (smooth, continuous).

B. Generative Adversarial Networks (GANs)

1. Architecture:

   • Generator $G$: Takes noise $$\displaystyle \mathbf{z} \sim p_\mathbf{z} $$ → fake data $G(\mathbf{z})$.

   • Discriminator $D$: Classifies real vs. fake; outputs probability.

2. Adversarial Training (Min-Max Game):

$$\min_G \max_D V(D,G) = \mathbb{E}_{\mathbf{x}\sim p_{data}}[\log D(\mathbf{x})] + \mathbb{E}_{\mathbf{z}\sim p_\mathbf{z}}[\log(1 - D(G(\mathbf{z})))]$$

3. Challenges: Mode collapse, training instability, evaluation difficulty (use Inception Score, FID).

C. GANs vs. VAEs

D. Hybrid Models: VAE-GAN

• Combine VAE's reconstruction loss with GAN's adversarial loss.

• Encoder-decoder (VAE) + discriminator (GAN).

• Benefits: Sharp samples (from GAN) + structured latent (from VAE).

E. Restricted Boltzmann Machines (RBMs)

1. Energy-Based Model: Bipartite graph (visible $\mathbf{v}$, hidden $\mathbf{h}$).

$$E(\mathbf{v},\mathbf{h}) = -\mathbf{v}^T \mathbf{W} \mathbf{h} - \mathbf{b}^T \mathbf{v} - \mathbf{c}^T \mathbf{h}$$

Probability: $$\displaystyle p(\mathbf{v},\mathbf{h}) = \frac{e^{-E(\mathbf{v},\mathbf{h})}}{Z} $$.

2. Training via Contrastive Divergence (CD-k):

   • Positive phase: $$\displaystyle \langle \mathbf{v}\mathbf{h}^T \rangle_{data} $$.

   • Negative phase: Run Gibbs sampling (alternating $\mathbf{v} \sim p(\mathbf{v}|\mathbf{h})$, $\mathbf{h} \sim p(\mathbf{h}|\mathbf{v})$) for $k$ steps.

   • Update: $$\displaystyle \Delta \mathbf{W} \propto \langle \mathbf{v}\mathbf{h}^T \rangle_{data} - \langle \mathbf{v}\mathbf{h}^T \rangle_{recon} $$.

F. Deep Belief Networks (DBNs)

• Stacked RBMs; trained greedily layer-wise (unsupervised pre-training).

• Each RBM trained sequentially; weights initialize next layer.

• Historical importance (pre-2012) for deep network initialization.

G. Autoregressive Models

• Idea: Factorize joint distribution $$\displaystyle p(\mathbf{x}) = \prod_{i=1}^d p(x_i | \mathbf{x}_{<i}) $$.

• MADE (Masked Autoencoder): Autoencoder with masks to enforce autoregressive ordering; efficient sampling.

• NADE: Neural Autoregressive Distribution Estimator; similar, with normalized exponential.

H. Deep Dream

• Process: Gradient ascent on input to maximize activations of specific layers/filters.

• Result: Surreal, dream-like images highlighting learned features.

• Use: Feature visualization, understanding network internals.

I. Applications of Deep Generative Models

• Image synthesis (StyleGAN), data augmentation, anomaly detection (unsupervised), drug discovery (molecule generation), art generation.

VI. Deep Reinforcement Learning

A. Reinforcement Learning Fundamentals

• Agent: Learner/decision-maker.

• Environment: World agent interacts with.

• State $s$: Current situation.

• Action $a$: Agent's move.

• Reward $r$: Immediate feedback.

• Policy $\pi(a|s)$: Agent's strategy (probability of action in state).

• Markov Decision Process (MDP): $(S, A, P, R, \gamma)$ where $P(s'|s,a)$ transition, $R(s,a)$ reward, $\gamma \in [0,1]$ discount.

B. Dynamic Programming for MDPs

1. Value Iteration:

   • Initialize $$\displaystyle V_0(s)=0 $$.

   • Iterate: $$\displaystyle V_{k+1}(s) = \max_a \sum_{s'} P(s'|s,a) [R(s,a) + \gamma V_k(s')] $$.

   • Converges to optimal $$\displaystyle V^* $$; extract policy $$\displaystyle \pi^*(s) = \arg\max_a \sum_{s'} P(s'|s,a)[R(s,a)+\gamma V^*(s')] $$.

2. Policy Iteration:

   • Policy Evaluation: Solve $$\displaystyle V^\pi(s) = \sum_{s'} P(s'|s,\pi(s))[R(s,\pi(s)) + \gamma V^\pi(s')] $$ (linear system).

   • Policy Improvement: $$\displaystyle \pi_{new}(s) = \arg\max_a \sum_{s'} P(s'|s,a)[R(s,a) + \gamma V^\pi(s')] $$.

   • Comparison: Policy iteration often faster convergence per iteration but costly evaluation; value iteration simpler per step.

C. Q-Learning & Deep Q-Networks (DQN)

1. Q-Learning: Learn action-value $Q(s,a)$.

   • Update: $$\displaystyle Q(s,a) \leftarrow Q(s,a) + \alpha [r + \gamma \max_{a'} Q(s',a') - Q(s,a)] $$.

   • Off-policy; converges to optimal $$\displaystyle Q^* $$.

2. DQN: Use neural network to approximate $Q(s,a;\theta)$.

   • Experience Replay: Store transitions $(s,a,r,s')$ in buffer; sample mini-batches to break correlations.

   • Target Network: Separate network with parameters $$\displaystyle \theta^- $$ for target $$\displaystyle y = r + \gamma \max_{a'} Q(s',a';\theta^-) $$; updated slowly → stabilizes training.

   • Loss: $$\displaystyle \mathcal{L} = \mathbb{E}_{(s,a,r,s')\sim \mathcal{D}} \left[ \left( y - Q(s,a;\theta) \right)^2 \right] $$.

D. Advanced DQN Algorithms

1. Double DQN: Decouple action selection and evaluation to reduce overestimation bias.

   • $$\displaystyle y = r + \gamma Q(s', \arg\max_{a'} Q(s',a';\theta); \theta^-) $$.

2. Dueling DQN: Separate value stream $V(s)$ and advantage stream $A(s,a)$; combine: $$\displaystyle Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a') $$.

   • Better generalization, especially when action values similar.

E. Least Squares Policy Iteration (LSPI)

• Linear function approximation: $$\displaystyle Q(s,a) = \phi(s,a)^T \mathbf{w} $$.

• Policy Evaluation: Solve linear system $$\displaystyle (\Phi^\top D \Phi) \mathbf{w} = \Phi^\top D \mathbf{r} $$, where $D$ is state-action distribution matrix, $\Phi$ feature matrix.

• Policy Improvement: Greedy w.r.t. current $Q$.

• Advantage: No step-size tuning; uses least squares.

F. Applications

• Game playing (Atari DQN, AlphaGo), robotics control, autonomous navigation, resource management.

VII. Dimensionality Reduction and Linear Algebra Foundations

A. Principal Component Analysis (PCA)

1. Objective: Find orthogonal axes (principal components) maximizing variance.

2. Algorithm:

   • Mean-center data: $$\displaystyle \mathbf{X}_{centered} = \mathbf{X} - \boldsymbol{\mu} $$.

   • Compute covariance matrix: $$\displaystyle \mathbf{C} = \frac{1}{N-1} \mathbf{X}_{centered}^T \mathbf{X}_{centered} $$.

   • Eigenvalue decomposition: $$\displaystyle \mathbf{C} = \mathbf{V} \mathbf{\Lambda} \mathbf{V}^T $$; $\mathbf{V}$ columns = eigenvectors (PCs), $\mathbf{\Lambda}$ diagonal = eigenvalues (variance explained).

   • Project: $$\displaystyle \mathbf{Z} = \mathbf{X}_{centered} \mathbf{V}_k $$ (top $k$ eigenvectors).

3. Interpretation: PCs are directions of maximum variance; projection retains most information.

B. Singular Value Decomposition (SVD)

1. Matrix Factorization: $$\displaystyle \mathbf{A} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^T $$.

   • $\mathbf{U}$: left singular vectors (columns orthonormal).

   • $\mathbf{\Sigma}$: diagonal matrix of singular values $$\displaystyle \sigma_1 \ge \sigma_2 \ge ... \ge 0 $$.

   • $\mathbf{V}$: right singular vectors (columns orthonormal).

2. Relationship with PCA: If $\mathbf{A}$ is centered data matrix, then $\mathbf{V}$ = eigenvectors of $$\displaystyle \mathbf{A}^T\mathbf{A} $$ (covariance matrix up to scale); $$\displaystyle \sigma_i^2/(N-1) $$ = eigenvalues.

3. Truncated SVD: Keep top $k$ singular values/vectors → low-rank approximation $$\displaystyle \mathbf{A}_k = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^T $$; used for dimensionality reduction (e.g., LSA).

C. PCA vs. SVD vs. Autoencoders

D. Representation Learning

• Idea: Learn hierarchical features from raw data; lower layers capture simple patterns, higher layers capture abstract concepts.

• Benefits: Reduces need for manual feature engineering; enables transfer learning (pre-trained networks); improves performance with less labeled data.

VIII. Advanced Topics and Model Efficiency

A. Probabilistic Graphical Models

1. Directed (Bayesian Networks): DAG; nodes = random variables; edges = conditional dependencies.

   • Conditional Independence: $X \perp Y | Z$ means $$\displaystyle p(X,Y|Z) = p(X|Z)p(Y|Z) $$.

   • d-separation: Algorithm to determine independence from graph structure.

2. Undirected (Markov Networks): Undirected graph; factors $$\displaystyle \phi_C(\mathbf{X}_C) $$ defined on cliques $C$.

   • Energy: $$\displaystyle E(\mathbf{x}) = -\sum_C \log \phi_C(\mathbf{x}_C) $$.

   • Probability: $$\displaystyle p(\mathbf{x}) = \frac{1}{Z} e^{-E(\mathbf{x})} $$.

B. Model Compression & Optimization

1. Unit Pruning: Remove entire neurons/filters with small norm or low activation.

   • Impact: Increases sparsity, reduces computation; may require fine-tuning.

2. Weight Pruning: Set small weights to zero (e.g., magnitude-based).

3. Quantization: Reduce precision of weights/activations (e.g., 32-bit → 8-bit).

C. Data Preprocessing

D. Natural Language Processing (NLP) with Deep Learning

1. Four Key Elements:

   • Lexical Analysis: Tokenization, stemming, stop-word removal.

   • Syntactic Analysis: Parsing, POS tagging.

   • Semantic Analysis: Meaning, word sense disambiguation.

   • Pragmatic Analysis: Context, intent, sarcasm.

2. Deep Learning Applications:

   • Word Embeddings: Word2Vec (skip-gram, CBOW), GloVe – dense vector representations.

   • RNNs/Transformers: Language modeling, translation, sentiment analysis.

E. Recursive Neural Networks (RecursiveNNs)

1. Tree-Structured: Apply same weights recursively over tree structure (e.g., parse tree).

2. vs. RNN: RNNs handle sequences (linear); RecursiveNNs handle hierarchical structures (trees).

3. Architecture: Each node computes representation from children: $$\displaystyle \mathbf{h}_\text{parent} = f(\mathbf{h}_{\text{left child}}, \mathbf{h}_{\text{right child}}) $$.

4. Applications: Sentence meaning composition, sentiment analysis (parse tree), program analysis.

IX. Cross-Cutting Themes and Critical Analysis

A. Significance of Depth

• Hierarchical Feature Learning: Low-level → high-level abstractions.

• Increased Model Capacity: Exponential number of regions representable (for ReLU networks).

• Parameter Efficiency: Deeper networks often need fewer parameters than wide shallow ones for same performance.

• Empirical Performance: Depth correlates strongly with accuracy on large datasets (ImageNet).

B. Limitations vs. Human Brain

C. Emerging Trends & Future Directions

• Explainable AI (XAI): Techniques (LIME, SHAP, Grad-CAM) to interpret decisions.

• Few-shot Learning: Learn from few examples (meta-learning, prototypical networks).

• Neuromorphic Computing: Brain-inspired hardware (spiking NN, memristors).

• Self-supervised Learning: Learn representations from unlabeled data (contrastive learning, masked modeling).

• Foundation Models: Large pre-trained models (LLMs, diffusion models) adapted to many tasks.

Final Exam Strategy:

1. Definitions: Memorize key terms (e.g., backpropagation, ReLU, residual connection).

2. Formulas: Boxed ones are critical (softmax, cross-entropy, convolution output size).

3. Comparisons: Know differences (GAN vs VAE, LSTM vs GRU, SGD vs Adam).

4. Architectures: Recall key innovations of AlexNet, ResNet, LSTM.

5. Diagrams: Sketch computational graph of LSTM, CNN layer, seq2seq.

6. Applications: Link models to real-world uses (CNN→image classification, RNN→NLP).