UNIT 3: OPTIMIZATION TECHNIQUES IN MACHINE LEARNING – SHORT NOTES

I. INTRODUCTION TO OPTIMIZATION IN MACHINE LEARNING

• Role of Optimization: The core of training ML models is finding parameters (weights) $\theta$ that minimize a loss function $J(\theta)$ over training data. It's the mathematical engine that turns data into a predictive model.

• Supervised vs. Unsupervised (Optimization View):

  • Supervised: Minimize error between predictions $f(x;\theta)$ and known labels $y$. (e.g., minimize MSE for regression, cross-entropy for classification).

  • Unsupervised: Optimize a measure of data structure (e.g., maximize likelihood in clustering, minimize reconstruction error in autoencoders).

• Historical Milestones: Key breakthroughs include backpropagation (1986), ReLU & dropout (2012, AlexNet), ResNet (2015) for very deep networks, and Transformer architecture (2017) for sequence modeling.

• Representation Learning: The process where a model automatically discovers the most informative features/representations from raw data (e.g., CNN layers learning edges to objects). Optimization drives this discovery.

II. NEURAL NETWORK FUNDAMENTALS

• Perceptron & MLP:

  • Perceptron: Single neuron: $$\displaystyle y = f(\mathbf{w}^T\mathbf{x} + b) $$. Linear classifier.

  • MLP: Stacked perceptrons with non-linear activations. Universal approximation theorem: A sufficiently large MLP can approximate any continuous function.

• Activation Functions (Introduce Non-linearity):

  | Function | Formula | Pros | Cons / Notes | | :--- | :--- | :--- | :--- | | Sigmoid | $$\displaystyle \sigma(z) = \frac{1}{1+e^{-z}} $$ | Output in (0,1), smooth | Vanishing gradients, not zero-centered | | Tanh | $$\displaystyle \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} $$ | Zero-centered, steeper than sigmoid | Vanishing gradients | | ReLU | $$\displaystyle \text{ReLU}(z) = \max(0, z) $$ | Computationally cheap, mitigates vanishing gradient for +ve inputs | "Dying ReLU" problem (neurons stuck at 0) | | Leaky ReLU | $\max(\alpha z, z)$, $\alpha \approx 0.01$ | Fixes "dying ReLU" | May still have inconsistent gradient for -ve inputs | | Softmax | $$\displaystyle \sigma(\mathbf{z})_j = \frac{e^{z_j}}{\sum_{k=1}^K e^{z_k}} $$ | Converts logits to probability distribution | Used only in output layer for multi-class classification |

  Exam Tip: ReLU is the default for hidden layers in deep networks/CNNs due to its effectiveness in mitigating vanishing gradients and computational efficiency.

• Forward Propagation: Sequential computation of layer outputs: $$\displaystyle \mathbf{a}^{[l]} = g^{[l]}(\mathbf{z}^{[l]}) $$, where $$\displaystyle \mathbf{z}^{[l]} = \mathbf{W}^{[l]}\mathbf{a}^{[l-1]} + \mathbf{b}^{[l]} $$.

• Loss Functions:

  • Mean Squared Error (MSE): $$\displaystyle J = \frac{1}{m}\sum_{i=1}^m (y^{(i)} - \hat{y}^{(i)})^2 $$. Used for regression.

  • Cross-Entropy (CE):

    • Binary: $$\displaystyle J = -\frac{1}{m}\sum [y^{(i)}\log(\hat{p}^{(i)}) + (1-y^{(i)})\log(1-\hat{p}^{(i)})] $$

    • Multi-class: $$\displaystyle J = -\frac{1}{m}\sum_{i=1}^m \sum_{j=1}^K y_j^{(i)} \log(\hat{p}_j^{(i)}) $$. Used for classification.

III. TRAINING NEURAL NETWORKS: BACKPROPAGATION & OPTIMIZATION

• Backpropagation (BP):

  1. Forward Pass: Compute loss $J$.

  2. Backward Pass: Apply chain rule to compute gradients $$\displaystyle \frac{\partial J}{\partial \mathbf{W}^{[l]}} $$, $$\displaystyle \frac{\partial J}{\partial \mathbf{b}^{[l]}} $$ for each layer, starting from output layer $L$ to input layer $1$.

  3. Update: $$\displaystyle \mathbf{W}^{[l]} := \mathbf{W}^{[l]} - \alpha \frac{\partial J}{\partial \mathbf{W}^{[l]}} $$, $$\displaystyle \mathbf{b}^{[l]} := \mathbf{b}^{[l]} - \alpha \frac{\partial J}{\partial \mathbf{b}^{[l]}} $$.

  Core Principle: Efficiently computes gradients for all parameters in a single forward+backward pass.

• Gradient Descent Variants:

  | Variant | Batch Size | Pros | Cons | | :--- | :--- | :--- | :--- | | Batch GD | Entire dataset ($m$) | Stable convergence, exact gradient | Slow, memory-intensive | | Stochastic GD (SGD) | 1 sample | Fast updates, can escape shallow minima | Noisy, unstable | | Mini-batch GD | $n$ samples (e.g., 32, 128) | Best of both: Balance of speed & stability. Standard practice. |

• Advanced Optimization Algorithms:

  • Momentum: $$\displaystyle \mathbf{v} = \beta \mathbf{v} + (1-\beta) \nabla J $$, $$\displaystyle \theta := \theta - \alpha \mathbf{v} $$. Accelerates convergence, dampens oscillations.

  • AdaGrad: Adapts learning rate per parameter. $$\displaystyle G_t = G_{t-1} + (\nabla J)^2 $$, $$\displaystyle \theta := \theta - \frac{\alpha}{\sqrt{G_t + \epsilon}} \nabla J $$. Accumulates squared gradients, causing aggressive lr decay.

  • RMSProp: Fixes AdaGrad's aggressive decay using moving average. $$\displaystyle G_t = \gamma G_{t-1} + (1-\gamma)(\nabla J)^2 $$. Better for non-convex.

  • Adam (Adaptive Moment Estimation): Most popular. Combines Momentum (1st moment) & RMSProp (2nd moment) with bias correction.

$$\mathbf{m}_t = \beta_1 \mathbf{m}_{t-1} + (1-\beta_1) \nabla J_t$$

$$\mathbf{v}_t = \beta_2 \mathbf{v}_{t-1} + (1-\beta_2) (\nabla J_t)^2$$

$$\hat{\mathbf{m}}_t = \frac{\mathbf{m}_t}{1-\beta_1^t}, \hat{\mathbf{v}}_t = \frac{\mathbf{v}_t}{1-\beta_2^t}$$

$$\theta_t := \theta_{t-1} - \alpha \frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t + \epsilon}}$$

    \boxed{\text{Adam: Adaptive per-parameter learning with momentum. Default: } \alpha=0.001, \beta_1=0.9, \beta_2=0.999}

• Vanishing/Exploding Gradients:

  • Problem: In deep networks or BPTT for RNNs, gradients $\prod \sigma'(z)$ can shrink to ~0 (vanish) or explode to $\infty$ during backpropagation.

  • Impact: Prevents early layers from learning (vanishing) or causes unstable training (exploding).

  • Mitigation:

    1. Weight Initialization: He initialization (for ReLU), Xavier/Glorot (for Tanh).

    2. Batch Normalization: Normalizes layer inputs, stabilizing distribution.

    3. Residual Connections (ResNet): $$\displaystyle \mathbf{y} = \mathbf{F}(\mathbf{x}) + \mathbf{x} $$. Creates identity shortcut, gradient flows directly.

    4. Gradient Clipping: Cap gradient norm (common in RNNs).

    5. Use ReLU family activations.

• Value Iteration vs. Policy Iteration (for MDPs):

  | Aspect | Value Iteration | Policy Iteration | | :--- | :--- | :--- | | Process | Iteratively apply Bellman expectation update to value function $V(s)$ until convergence. | Alternate: Policy Evaluation (compute $$\displaystyle V^\pi $$) -> Policy Improvement (greedy update of $\pi$). | | Convergence | Slower, guaranteed. | Faster per iteration, guaranteed. | | Complexity | Each iteration: $$\displaystyle O(|\mathcal{S}|^2|\mathcal{A}|) $$ | Each iteration (eval+improve): $$\displaystyle O(|\mathcal{S}|^2|\mathcal{A}|) $$ but fewer iterations. |

  Key Difference: Policy iteration often converges in fewer iterations but each iteration is more expensive (requires full policy evaluation).

IV. REGULARIZATION TECHNIQUES (Prevent Overfitting)

• Overfitting vs. Underfitting:

  • Overfitting: Model learns noise, high training accuracy, low test accuracy. High variance.

  • Underfitting: Model fails to capture pattern, low training & test accuracy. High bias.

• L1 & L2 Regularization (Weight Decay):

  • Add penalty to loss: $$\displaystyle J_{\text{reg}} = J + \lambda \Omega(\theta) $$.

  • L2 (Ridge): $$\displaystyle \Omega(\theta) = \|\theta\|_2^2 = \sum \theta_i^2 $$. Shrinks weights uniformly. Most common.

  • L1 (Lasso): $$\displaystyle \Omega(\theta) = \|\theta\|_1 = \sum |\theta_i| $$. Produces sparse weights (some become exactly zero). Feature selection.

  • Update Rule (for L2): $$\displaystyle \theta := \theta - \alpha (\nabla J + \lambda \theta) $$. Equivalent to weight decay.

• Dropout:

  • Mechanism: During training, randomly "drop" (set to 0) a fraction $p$ of neurons in a layer for each forward pass. At test time, use all neurons but scale activations by $(1-p)$ (or inverted dropout: scale during training).

  • Impact: Prevents co-adaptation of neurons, acts like ensemble of many thinned networks. Greatly improves generalization.

• Batch Normalization (BN):

  • How: For each mini-batch, normalize layer inputs: $$\displaystyle \hat{x}^{(k)} = \frac{x^{(k)} - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} $$, then scale & shift: $$\displaystyle y^{(k)} = \gamma \hat{x}^{(k)} + \beta $$.

  • Advantages:

    1. Reduces internal covariate shift.

    2. Allows higher learning rates.

    3. Has slight regularization effect (noise from batch stats).

    4. Reduces need for careful weight initialization.

• Early Stopping: Monitor validation loss; stop training when it starts increasing. Simple, effective regularization.

• Data Augmentation: Artificially increase training data by applying transformations (rotate, flip, crop images). Forces model to learn invariances.

• Regularization in Autoencoders: Sparsity constraints (sparse autoencoder), denoising (corrupt input, reconstruct clean), contractive (penalize Jacobian norm).

V. CONVOLUTIONAL NEURAL NETWORKS (CNNs)

• Core Operations:

  • Convolution: $$\displaystyle \mathbf{S}(i,j) = (\mathbf{I} * \mathbf{K})(i,j) = \sum_m \sum_n \mathbf{I}(i+m, j+n) \mathbf{K}(m,n) $$. Extracts local features (feature maps).

  • Padding: Adds zeros to border of input to control output size. 'same' vs 'valid'.

  • Stride: Step size of filter movement. Larger stride = smaller output.

  • Pooling (Downsampling): Max pooling (most common), Average pooling. Provides translation invariance, reduces spatial size.

• Key Architectures:

  • LeNet-5 (1998): First successful CNN (handwritten digits).

  • AlexNet (2012): ReLU, dropout, GPU training. Deep learning breakthrough.

  • VGG (2014): Very deep, uniform 3x3 conv layers. Simple, effective.

  • ResNet (2015): Residual blocks with skip connections. Enabled training of 100+ layer networks.

• Applications Beyond Images:

  | Data Format | How CNN Adapts | Example | | :--- | :--- | :--- | | 1D Signals (text, time series) | 1D convolution over sequence | Text classification, sensor data analysis | | Video | 3D convolution (spatial + temporal) or frame-by-frame + RNN | Action recognition | | 3D Data (medical CT/MRI) | 3D convolution kernels | Tumor segmentation |

• Optimization Advantages:

  • Parameter Sharing: Same filter used across spatial locations. Drastically reduces parameters.

  • Sparse Connectivity: Each output neuron connected only to local input region. Exploits spatial locality.

VI. RECURRENT NEURAL NETWORKS (RNNs) & VARIANTS

• Basic RNN:

  • Architecture: Hidden state $$\displaystyle \mathbf{h}_t = f(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}_h) $$, output $$\displaystyle \mathbf{y}_t = g(\mathbf{W}_{hy}\mathbf{h}_t + \mathbf{b}_y) $$.

  • BPTT: Unroll network through time, apply standard backpropagation. Gradients can vanish/explode over long sequences.

• Long Short-Term Memory (LSTM):

  • Purpose: Solve vanishing gradient problem in RNNs for long-range dependencies.

  • Cell State ($$\displaystyle \mathbf{C}_t $$): The "memory conveyor belt". Information can flow with minimal linear transformation.

  • Gates (Sigmoid + Tanh):

    1. Forget Gate ($$\displaystyle f_t $$): What to remove from cell state? $$\displaystyle \sigma(\mathbf{W}_f \cdot [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_f) $$

    2. Input Gate ($$\displaystyle i_t $$): What new info to store? $$\displaystyle \sigma(\mathbf{W}_i \cdot [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_i) $$

    3. Candidate Cell State ($$\displaystyle \tilde{\mathbf{C}}_t $$): New candidate values. $$\displaystyle \tanh(\mathbf{W}_C \cdot [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_C) $$

    4. Update Cell State: $$\displaystyle \mathbf{C}_t = f_t \odot \mathbf{C}_{t-1} + i_t \odot \tilde{\mathbf{C}}_t $$

    5. Output Gate ($$\displaystyle o_t $$): What to output from cell state? $$\displaystyle \sigma(\mathbf{W}_o \cdot [\mathbf{h}_{t-1}, \mathbf{x}_t] + \mathbf{b}_o) $$

    6. Hidden State: $$\displaystyle \mathbf{h}_t = o_t \odot \tanh(\mathbf{C}_t) $$

  Advantage over Simple RNN: Explicit cell state path with additive interactions ($$\displaystyle \mathbf{C}_t = f_t \mathbf{C}_{t-1} + ... $$) allows gradients to flow stably over many time steps.

• Gated Recurrent Unit (GRU): LSTM simplification. Combines forget & input gates into a single update gate ($$\displaystyle z_t $$). Has a reset gate ($$\displaystyle r_t $$). No separate cell state; $$\displaystyle \mathbf{h}_t $$ is the main state. Often faster, similar performance.

• Deep RNNs: Stack multiple RNN/LSTM/GRU layers. Each layer learns representations at different temporal abstractions.

• Applications: NLP (translation, sentiment), time series forecasting, speech recognition.

VII. AUTOENCODERS & DIMENSIONALITY REDUCTION

• Basic Autoencoder: Encoder $$\displaystyle z = f_\theta(x) $$ compresses input to latent code $z$. Decoder $$\displaystyle \hat{x} = g_\phi(z) $$ reconstructs input. Trained to minimize reconstruction loss $$\displaystyle L(x, g_\phi(f_\theta(x))) $$.

• Types:

  • Sparse Autoencoder: Adds sparsity penalty (e.g., KL divergence) to hidden activations. Forces learning of meaningful features.

  • Contractive Autoencoder: Adds penalty on Frobenius norm of Jacobian $$\displaystyle \|\frac{\partial f_\theta(x)}{\partial x}\|_F^2 $$. Makes encoder locally contractive, robust to input noise.

  • Denoising Autoencoder (DAE): Train to reconstruct clean input from corrupted version (e.g., add Gaussian noise, drop pixels). Learns robust features.

  • Variational Autoencoder (VAE): Probabilistic. Encoder outputs distribution parameters $(\mu, \sigma)$ for latent code $z$. Uses reparameterization trick: $$\displaystyle z = \mu + \sigma \odot \epsilon $$, $\epsilon \sim \mathcal{N}(0,I)$. Loss = Reconstruction loss + KL divergence between $q(z|x)$ and $\mathcal{N}(0,I)$. Enables generative sampling.

• Autoencoders vs. PCA/SVD:

  | Feature | PCA/SVD (Linear) | Autoencoders (Non-linear) | | :--- | :--- | :--- | | Model | Linear transformation | Deep neural network (non-linear) | | Capacity | Limited to linear subspace | Can learn complex, non-linear manifolds | | When to Use Autoencoder | Data lies on/near a linear subspace. | Data has non-linear structure (e.g., images, manifold data). |

• Auto-regressive Models (NADE, MADE): Generate data by factorizing joint probability $$\displaystyle p(\mathbf{x}) = \prod_{i=1}^d p(x_i | \mathbf{x}_{<i}) $$. MADE (Masked Autoencoder for Distribution Estimation) uses masked layers to enforce auto-regressive property within a feedforward network.

VIII. GENERATIVE MODELS

• Generative Adversarial Networks (GANs):

  • Architecture: Two networks: Generator $G$ (noise $z \to$ fake data $\hat{x}$) and Discriminator $D$ (data $\to$ probability real).

  • Adversarial Training: Min-max game: $$\displaystyle \min_G \max_D V(D,G) = \mathbb{E}_{x\sim p_{data}}[\log D(x)] + \mathbb{E}_{z\sim p_z}[\log(1-D(G(z)))] $$.

  • Training: Alternate: 1) Fix $G$, train $D$ to distinguish real/fake. 2) Fix $D$, train $G$ to fool $D$.

• Variational Autoencoders (VAEs):

  • Latent Variable Model: $$\displaystyle p(x) = \int p(x|z)p(z)dz $$. Intractable.

  • Solution: Use inference network $$\displaystyle q_\phi(z|x) $$ (encoder). Optimize Evidence Lower Bound (ELBO):

$$\log p(x) \geq \mathbb{E}_{z\sim q_\phi}[\log p_\theta(x|z)] - D_{KL}(q_\phi(z|x) \| p(z))$$

*   **Reparameterization Trick**: Makes gradients flow through sampling: $$\displaystyle z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon $$.

• GANs vs. VAEs:

  | Aspect | GANs | VAEs | | :--- | :--- | :--- | | Training | Adversarial, unstable, mode collapse risk. | Variational, stable, but can produce blurry samples. | | Sample Quality | Often sharper, more realistic. | Typically blurrier. | | Latent Space | Unstructured, not necessarily continuous. | Structured, continuous, disentangled (by design). | | Use Case | Choose GANs for highest-fidelity image synthesis (art, photo-realistic). Choose VAEs for structured latent space (interpolation, downstream tasks), when training stability is critical. |

• Deep Dream & Neural Style Transfer:

  • Optimization of Inputs: Start with noise/input image, optimize input pixels (not weights) to maximize:

    • Deep Dream: Activations of specific layer(s) (amplify patterns network "sees").

    • Style Transfer: Match content (content layer activations) + style (Gram matrix of feature correlations) of a style image.

IX. REINFORCEMENT LEARNING WITH DEEP LEARNING

• Markov Decision Process (MDP): $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$. State transition $P(s'|s,a)$, reward $R(s,a,s')$, discount $\gamma$.

• Policy Iteration vs. Value Iteration:

  • Policy Iteration: $$\displaystyle \pi_0 \to $$ Policy Evaluation (solve for $$\displaystyle V^{\pi} $$) $\to$ Policy Improvement ($$\displaystyle \pi' = \text{greedy}(V^{\pi}) $$) $\to$ repeat. Faster convergence, expensive per step.

  • Value Iteration: Directly apply Bellman optimality update: $$\displaystyle V_{k+1}(s) = \max_a \sum_{s'} P(s'|s,a)[R(s,a,s') + \gamma V_k(s')] $$. Simpler, single loop.

• Q-learning & Deep Q-Networks (DQN):

  • Q-learning: Learn action-value function $Q(s,a)$. Update: $$\displaystyle Q(s,a) \leftarrow Q(s,a) + \alpha [r + \gamma \max_{a'} Q(s',a') - Q(s,a)] $$.

  • DQN: Use deep network to approximate $Q(s,a;\theta)$. Key Innovations:

    1. Experience Replay: Store transitions $(s,a,r,s')$ in buffer, sample mini-batches. Breaks correlation.

    2. Target Network: Use separate network with frozen parameters $$\displaystyle \theta^- $$ to compute target $$\displaystyle y = r + \gamma \max_{a'} Q(s',a';\theta^-) $$. Stabilizes target.

• Advanced DQN:

  • Double DQN: Decouples action selection and evaluation to reduce overestimation bias.

$$y = r + \gamma Q(s', \arg\max_{a'} Q(s',a';\theta); \theta^-)$$

*   **Dueling DQN**: Estimates $V(s)$ and $A(s,a)$ separately, then combines: $$\displaystyle Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'} A(s,a') $$. Learns state-value independently of specific actions.

• Least Squares Policy Iteration (LSPI): Model-based RL. Uses linear function approximation for Q-values. Policy iteration where policy evaluation step is solved via least-squares regression ( LSTD(λ) algorithm). Sample efficient.

X. TRADITIONAL ML ALGORITHMS WITH OPTIMIZATION

• Logistic Regression:

  • Model: $$\displaystyle P(y=1|x) = \sigma(\mathbf{w}^T\mathbf{x} + b) $$.

  • Cost Function (Binary Cross-Entropy): $$\displaystyle J(\mathbf{w}) = -\frac{1}{m}\sum_{i=1}^m [y^{(i)}\log(p^{(i)}) + (1-y^{(i)})\log(1-p^{(i)})] $$.

  • Optimization: Gradient descent on $J(\mathbf{w})$. $$\displaystyle \frac{\partial J}{\partial w_j} = \frac{1}{m}\sum_{i=1}^m (p^{(i)} - y^{(i)})x_j^{(i)} $$.

• Naive Bayes Classification:

  • Principle: Apply Bayes' theorem with naive assumption of feature independence given class $y$: $$\displaystyle P(x|y) = \prod_i P(x_i|y) $$.

  • MLE: Estimate $P(y)$ and $$\displaystyle P(x_i|y) $$ from training counts (for discrete) or Gaussian (for continuous).

  • Limitation: Independence assumption often violated. Can be overly confident in probability estimates.

• k-Means Clustering:

  • Objective: Minimize within-cluster sum of squares (WCSS): $$\displaystyle \min_{C_1,...,C_k} \sum_{j=1}^k \sum_{x \in C_j} \|x - \mu_j\|^2 $$.

  • Optimization: Alternating minimization (Lloyd's algorithm):

    1. Initialize centroids $$\displaystyle \mu_j $$.

    2. Assignment Step: Assign each point to nearest centroid.

    3. Update Step: Recompute $$\displaystyle \mu_j $$ as mean of assigned points.

    4. Repeat 2-3 until convergence.

  • Choosing K:

    • Elbow Method: Plot WCSS vs. K; look for "elbow" (diminishing returns).

    • Silhouette Score: Measures cluster cohesion & separation. Range [-1,1], higher is better.

• PCA & SVD (Optimization View):

  • PCA: Find orthogonal directions (principal components) that maximize variance. Optimization: Solve eigenvalue problem for covariance matrix $$\displaystyle \mathbf{C} = \frac{1}{m}\mathbf{X}^T\mathbf{X} $$. Top $k$ eigenvectors are solution.

  • SVD: Decompose $$\displaystyle \mathbf{X} = \mathbf{U}\mathbf{\Sigma}\mathbf{V}^T $$. PCA on $\mathbf{X}$ is SVD on centered $\mathbf{X}$; right singular vectors $\mathbf{V}$ are principal directions.

  • Randomized SVD (GPU): Approximates top $k$ SVD using random projections. Complexity: $O(mnk \log k)$ vs. deterministic $$\displaystyle O(m^2n) $$. Applications: Large-scale PCA, latent semantic analysis, recommender systems (matrix completion).

XI. ADVANCED TOPICS IN DEEP LEARNING OPTIMIZATION

• Deep Belief Networks (DBNs):

  • Structure: Stack of Restricted Boltzmann Machines (RBMs).

  • Pre-training: Greedy, layer-wise unsupervised pre-training using Contrastive Divergence (CD-k) on each RBM. Then fine-tune with backpropagation. Was crucial before ReLU/Adam era.

• Directed Graphical Models: Bayesian networks. Represent conditional dependencies via directed acyclic graph. Learning often via MLE/MAP with gradient-based or EM algorithm.

• Representation Learning: Automatic discovery of features. Principles: disentangling factors of variation, learning hierarchical features, manifold learning.

• Model Pruning & Compression:

  • Unit/Neuron Pruning: Remove entire neurons/filters with small weights or activations. Reduces model size/inference cost.

  • Need: Deploy models on edge devices (mobile, IoT), reduce latency, energy.

  • Techniques: Magnitude-based pruning (remove smallest $|w|$), regularization (L1), sensitivity analysis.

• Weight Decay (L2 Regularization) Benefits:

  • Controls model complexity, prevents overfitting.

  • Encourages small, distributed weights.

  • Improves generalization by reducing network's capacity to fit noise.

  • Implementation: Add $$\displaystyle \frac{\lambda}{2}\|\theta\|_2^2 $$ to loss, or directly decay weights: $$\displaystyle \theta := \theta - \alpha\lambda\theta $$ during update.

XII. DATA PREPROCESSING FOR OPTIMIZATION

• Normalization (Min-Max Scaling):

$$x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$

Scales to $[0, 1]$. Sensitive to outliers.

• Standardization (Z-score Normalization):

$$x' = \frac{x - \mu}{\sigma}$$

Scales to mean=0, std=1. Robust to outliers. **Generally preferred for gradient-based optimization** as it keeps gradients in a stable range.

• Impact on Optimization: Proper scaling ensures all features contribute equally to loss gradient, prevents some weights from updating too fast/slow. Crucial for algorithms using distance-based updates (SGD, Adam).

XIII. OTHER ARCHITECTURES AND CONCEPTS

• Recursive Neural Networks (RecursiveNNs): Tree-structured, not sequential. Apply same set of weights in a bottom-up (or top-down) manner over a parse tree. Captures hierarchical structure (e.g., in NLP parse trees, computer vision scene graphs).

• Feedforward Neural Networks (FNNs): Baseline architecture. No cycles, information flows only forward. MLP is a type of FNN.

• Unsupervised Learning Examples:

  • Clustering: k-means, hierarchical.

  • Dimensionality Reduction: PCA, Autoencoders.

  • Generative Models: GANs, VAEs, RBMs.

• Content-based Recommendation Systems:

  • Optimization of Similarity Metrics: Learn user and item feature vectors $$\displaystyle \mathbf{u}_i, \mathbf{v}_j $$ such that dot product $$\displaystyle \mathbf{u}_i^T\mathbf{v}_j $$ matches user-item interaction (rating, click). Loss: $$\displaystyle J = \sum_{(i,j)} (r_{ij} - \mathbf{u}_i^T\mathbf{v}_j)^2 + \lambda(\|\mathbf{U}\|_F^2 + \|\mathbf{V}\|_F^2) $$.

  • Optimized via gradient descent (ALS or SGD). Similar to matrix factorization.