Information Retrieval (AL-503 (A)) - Unit 2 Short Notes

I. INFORMATION RETRIEVAL FUNDAMENTALS

IR Process and System Components

The IR process follows a pipeline:

1. Query Processing: Parsing, tokenization, stop-word removal, stemming.

2. Matching: Comparing the processed query against the index.

3. Ranking: Scoring and ordering matched documents by relevance.

4. Presentation: Displaying ranked results, often with snippets.

Key System Components:

• Crawler: Discovers and fetches documents from the web.

• Indexer: Processes documents and builds an inverted index.

• Retriever/Ranker: Matches queries to the index and computes scores.

• User Interface: Accepts queries and displays results.

[!TIP] Exam Focus: Be prepared to draw and label this pipeline. The indexer and retriever are the core computational components.

Role of Artificial Intelligence in IR

AI techniques enhance modern IR systems:

• Query Understanding: Using NLP for query expansion, spelling correction, and intent detection.

• Ranking: Learning-to-rank models (e.g., using neural networks) that go beyond simple term matching.

• Personalization: Adapting results based on user history, context, and profile.

• Clustering/Classification: Automatically organizing documents or categorizing queries.

Traditional IR Systems vs. Web Search

Document Preprocessing

A sequence of steps to normalize text for indexing:

1. Tokenization: Splitting text into individual terms (words, n-grams).

2. Stop-word Removal: Eliminating high-frequency, low-meaning words (e.g., "the", "is").

3. Stemming/Lemmatization: Reducing words to their root form (e.g., "running" -> "run"). Stemming is crude; lemmatization uses vocabulary/dictionary.

Indexing and Compression

• Inverted Index: The core data structure. Maps each term to a postings list of documents containing it, often with positions and frequencies.

  
  Term: "information"
  
  Postings: [(docID1, freq1, [pos1, pos2]), (docID2, freq2, [pos1]), ...]
  
  

• Index Compression: Vital for web-scale IR. Reduces disk space and I/O time.

  • Methods: Gap encoding (store differences between sorted docIDs), variable-byte encoding, gamma encoding.

  • Example: Postings [10, 20, 35, 40] become gaps [10, 10, 15, 5], which are encoded more compactly.

  • Benefit: Can reduce index size by 50-80% with minimal CPU overhead during decoding.

TF-IDF Term Weighting

A classic scheme to weight term importance in a document relative to a collection.

• TF (Term Frequency): Frequency of term t in document d. Often log-scaled: tf(t,d) = 1 + log(freq(t,d)) (if freq>0).

• IDF (Inverse Document Frequency): Measures term rarity across collection D. Downweights common terms.

$$\text{idf}(t, D) = \log \frac{N}{\text{df}(t)}$$

where `N` = total documents, `df(t)` = documents containing *t*.

• TF-IDF Weight: w(t,d) = tf(t,d) * idf(t, D)

• Rationale: A term is important if it appears frequently in a document but rarely across the whole collection.

• Common Variants: tf can be binary (0/1), augmented (0.5 + 0.5*tf/max_tf), or log-scaled. idf can use log(1 + N/df) or log(N/(1+df)) to avoid division by zero.

Retrieval Models and Ranking

• Vector Space Model: Represents documents and queries as vectors in term space. Similarity (often cosine similarity) is the ranking score.

$$\text{sim}(d,q) = \frac{\vec{d} \cdot \vec{q}}{||\vec{d}|| \cdot ||\vec{q}||}$$

• Probabilistic Models (BM25): A probabilistic retrieval function. A key component is BM25:

$$\text{score}(D,Q) = \sum_{t \in Q} \text{IDF}(t) \cdot \frac{f(t,D) \cdot (k_1 + 1)}{f(t,D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)}$$

where `f(t,D)` is term frequency, `|D|` is document length, `avgdl` is average doc length, `k1` & `b` are free parameters (typically `k1`≈1.2-2.0, `b`≈0.75).

• Probabilistic Relevance Feedback (Rocchio): Updates query vector based on relevant/non-relevant documents.

$$\vec{q}_{\text{new}} = \alpha \vec{q}_{\text{old}} + \beta \frac{1}{|D_r|} \sum_{\vec{d}_j \in D_r} \vec{d}_j - \gamma \frac{1}{|D_{nr}|} \sum_{\vec{d}_j \in D_{nr}} \vec{d}_j$$

• Latent Semantic Indexing (LSI): Uses Singular Value Decomposition (SVD) to map documents/queries into a lower-dimensional "concept space," capturing synonymy and polysemy.

  • Example: Terms "car" and "automobile" might map to the same latent dimension. Reduces noise and improves recall at potential cost to precision.

  • Steps: 1) Build term-document matrix A. 2) Compute SVD: A ≈ U_k Σ_k V_k^T. 3) Represent docs/queries in k-dimensional space (rows of V_k scaled by Σ_k).

Web Information Retrieval

• Crawlers (Spiders):

  • Advantages: Ensures freshness, achieves broad coverage, can discover new/isolated pages.

  • Needs: Must handle scale (billions of pages), adhere to politeness (robots.txt, delay between requests), and manage duplicate content.

• Focused Crawling: A topical crawler that prioritizes fetching pages relevant to a predefined subject.

  • Mechanism: Uses a classifier to predict relevance of a URL based on its context (anchor text, page content, parent page). Maintains a priority queue of URLs.

• XML Information Retrieval System Requirements:

  • Must be structure-aware: Index and query both content and XML tags/elements.

  • Requires a query language supporting structural constraints (e.g., XPath, XQuery).

  • Returns fragments or entire documents, not just whole-document scores.

• MapReduce & Hadoop for IR:

  • Use Cases: Distributed crawling, parallel index construction (inverted index merge), batch processing of query logs.

  • Scalability Analysis: Provides linear scalability for embarrassingly parallel tasks (like indexing). Overhead comes from network shuffling during sort/merge phases. Ideal for batch processing, not low-latency querying.

Clustering and Classification in IR

• K-Nearest Neighbors (KNN):

  • Choosing K: Use cross-validation on a labeled validation set to find K that maximizes performance (e.g., F1-score). Also consider domain knowledge (e.g., number of natural classes) and dataset size (larger K for noisy data).

• Agglomerative Clustering (Hierarchical):

  • Bottom-up approach. Each document starts as its own cluster.

  • Linkage Criteria determines cluster distance:

    • Single Link: Min distance between clusters (chaining effect).

    • Complete Link: Max distance (compact clusters).

    • Average/Group Link: Average distance (balanced).

• Naive Bayesian Classification:

  • Working: Applies Bayes' theorem with strong independence assumption (features/words are conditionally independent given the class).

  • Formula: P(C|D) ∝ P(C) * ∏ P(w_i|C) for words w_i in document D.

  • Advantages: Simple, fast, works well with high-dimensional text.

  • Limitations:

    1. Independence Assumption: Rarely holds in natural language (e.g., "New York").

    2. Zero-Frequency Problem: If a word never appeared in training for a class, P(w|C)=0 kills the product. Solution: Laplace (add-one) smoothing.

• Snippet Generation: Creating short, query-biased excerpts from a document.

  • Query-biased: Include query terms, often with surrounding context.

  • Length Control: Fixed number of words/characters or until a sentence boundary.

• Content-based Recommendation Systems:

  • Builds item profiles (e.g., for movies: genre, director, actors; for documents: TF-IDF vector).

  • Computes similarity (cosine similarity for vectors) between a user's profile (based on items they liked) and candidate items.

  • Limitation: Overspecialization (no serendipity), cannot recommend items without a profile.

II. MACHINE LEARNING FOR INFORMATION RETRIEVAL

Supervised vs. Unsupervised Learning

Reinforcement Learning (RL)

• Difference from Supervised: Learns from reward signals (scalar feedback) rather than explicit (input, target) pairs. Involves sequential decisions where actions affect future states.

• Deep Learning in RL: Deep Q-Networks (DQN) use a deep neural network to approximate the Q-value function Q(s,a). This allows RL to handle high-dimensional state spaces (e.g., raw pixels, complex user state representations) for tasks like ranking optimization or personalized search session management.

III. DEEP LEARNING FOUNDATIONS

Neural Network Basics

• Architecture: Input Layer (features) → Hidden Layer(s) (non-linear transformations) → Output Layer (predictions).

• Activation Functions: Introduce non-linearity, enabling networks to learn complex functions.

  • Sigmoid: σ(x) = 1/(1+e^{-x}). Outputs (0,1). Problems: Vanishing gradients, not zero-centered.

  • Tanh: tanh(x) = (e^x - e^{-x})/(e^x + e^{-x}). Outputs (-1,1). Zero-centered, but still vanishing gradients.

  • ReLU (Rectified Linear Unit): f(x) = max(0, x). Significance in CNNs: Computationally cheap, induces sparsity (some neurons inactive), mitigates vanishing gradient for positive inputs.

  • Leaky ReLU: f(x) = max(αx, x) (α small, e.g., 0.01). Fixes "dying ReLU" problem (neurons stuck at 0).

  • Softmax: Used in output layer for multi-class classification. Converts logits to probability distribution: σ(z)_j = e^{z_j} / Σ_{k=1}^K e^{z_k}.

• Feed Forward Neural Networks (FFNNs): Data flows strictly forward. Universal Approximation Theorem: A single hidden layer FFNN with enough neurons can approximate any continuous function.

• Backpropagation Algorithm: The core training algorithm.

  1. Forward Pass: Compute output and loss L for a batch.

  2. Backward Pass: Apply chain rule to compute gradient ∂L/∂w for each weight w.

  3. Weight Update: w := w - η * ∂L/∂w (using an optimizer like SGD).

  • Role: Efficiently computes gradients for all parameters in a deep network by reusing intermediate computations.

Representation Learning

• The automatic discovery of meaningful features from raw data (e.g., pixels, words) through neural network layers.

• Hierarchical Representations: Early layers learn simple features (edges, n-grams); deeper layers learn complex, abstract concepts (object parts, semantic phrases).

Representation Power

• Multilayer Perceptrons (MLPs): Deep MLPs have high capacity to model highly non-linear, complex relationships in data, far surpassing linear models.

• Sigmoid Neurons: Historically used. Their saturation (outputs near 0 or 1 for large |x|) causes vanishing gradients during backpropagation, hindering deep network training—a key reason for the shift to ReLU.

Historical Progression of Deep Learning

• 1940s-50s: Perceptron (single neuron).

• 1980s: Backpropagation algorithm popularized.

• 1990s-2000s: SVM dominance; neural networks seen as inferior.

• 2006: "Deep Learning" coined (Hinton et al.)—successful training of deep belief networks (DBNs).

• 2012: AlexNet (CNN) wins ImageNet, proving deep CNNs' power.

• 2010s: RNNs/LSTMs for sequences; GANs (2014); Transformers (2017) revolutionize NLP.

• 2020s: Massive scale pre-training (GPT, BERT), diffusion models.

IV. OPTIMIZATION, REGULARIZATION & PREPROCESSING

Optimization Algorithms (Adaptive Learning Rates)

Overfitting and Underfitting

• Overfitting: Model learns noise/training data specifics; high training accuracy, low validation/test accuracy. High variance.

• Underfitting: Model fails to capture underlying pattern; low training and validation accuracy. High bias.

• Prevention Techniques for Overfitting:

  1. Dropout: Randomly "drop" (set to 0) a fraction of neurons during training. Forces network to learn redundant representations.

  2. Weight Decay (L2 Regularization): Add penalty λ||W||^2 to loss. Encourages small weights.

  3. Early Stopping: Monitor validation loss; stop training when it starts to increase.

  4. Data Augmentation: Artificially increase training data size (e.g., rotate/crop images, synonym replacement in text).

  5. Reducing Model Capacity: Fewer layers/neurons.

Vanishing and Exploding Gradients

• Vanishing Gradient Problem: Gradients become extremely small as they backpropagate through many layers (especially with saturating activations like sigmoid/tanh). Effect: Early layers learn very slowly or not at all. Mitigation:

  • Use ReLU activations.

  • Batch Normalization (stabilizes activations).

  • Residual Connections (ResNet) allow gradient flow via identity mappings.

• Exploding Gradients: Gradients become excessively large, causing unstable training (NaNs). Common in BPTT for RNNs with long sequences. Mitigation: Gradient clipping (set a max threshold for gradient norm).

Regularization Techniques

• Dropout: During training, each neuron has probability p of being temporarily removed. At test time, all neurons are used, but outputs are scaled by (1-p). Improves generalization by preventing co-adaptation of neurons.

• Batch Normalization: Normalizes layer inputs to have zero mean and unit variance per mini-batch. Then applies a learned scale (γ) and shift (β).

  • Purpose: Reduces internal covariate shift (distribution changes of layer inputs during training). Allows higher learning rates, acts as a regularizer (adds noise via batch statistics), and speeds up convergence.

• Weight Decay (L2 Regularization): Loss becomes L_original + λ Σ w_{ij}^2. The λ parameter controls strength. Benefits: Penalizes large weights, leads to smoother models, often improves generalization.

Data Preprocessing

• Normalization (Min-Max Scaling): Rescales feature to a fixed range, usually [0,1].

$$x_{\text{norm}} = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$$

*   **When to use:** When algorithm assumes bounded input (e.g., neural networks with sigmoid output), or for image pixel values.

• Standardization (Z-score Normalization): Transforms data to have mean=0, std=1.

$$x_{\text{std}} = \frac{x - \mu}{\sigma}$$

*   **When to use:** When data follows a Gaussian-like distribution, or for algorithms assuming centered data (e.g., PCA, SVM). More robust to outliers than min-max.

V. DEEP LEARNING ARCHITECTURES

Convolutional Neural Networks (CNNs)

• Architecture: Designed for grid-like data (images). Core layers:

  • Convolutional Layer: Applies filters/kernels to extract local features (edges, textures). Produces feature maps.

  • Pooling Layer (Downsampling): Reduces spatial size. Max pooling (most common) takes max value in a window; average pooling takes average.

  • Fully Connected (FC) Layer: At the end, for classification/regression.

• Padding:

  • Valid ('V'): No padding. Output size shrinks.

  • Same ('S'): Padding added so output size equals input size (for stride=1).

• Data Formats for CNNs:

Recurrent Neural Networks (RNNs)

• For Sequence Data: Process one element at a time, maintaining a hidden state h_t that acts as memory of previous inputs.

$$h_t = f(W_{xh} x_t + W_{hh} h_{t-1} + b_h)$$

• Comparison with Feed-Forward:

  • RNN: Parameter sharing across time steps (same W_{hh}), handles variable-length sequences, captures temporal dependencies.

  • FFNN: No memory, treats each input independently, fixed input size.

• Deep RNN Architectures:

  • Stacked RNNs: Multiple RNN layers. Output of layer l at time t is input to layer l+1 at time t.

  • Bidirectional RNNs (BiRNN): Two RNNs: one forward (past→future), one backward (future→past). Concatenate outputs to capture context from both directions.

• Challenges in BPTT: Unrolling RNN through time creates a very deep network. Suffers severely from vanishing/exploding gradients, making it hard to learn long-range dependencies (e.g., 100+ steps).

• Encoding/Decoding Sequential Data (Seq2Seq): Framework for tasks like translation, summarization.

  • Encoder RNN compresses input sequence into a fixed-length context vector (final hidden state).

  • Decoder RNN generates output sequence autoregressively, conditioned on the context vector and its own previous outputs.

Long Short-Term Memory (LSTM)

• Designed to solve vanishing gradient problem in standard RNNs. Has a cell state C_t (the "memory highway") and gates to regulate information flow.

• Detailed Working (Equations):

  1. Forget Gate: f_t = σ(W_f · [h_{t-1}, x_t] + b_f) Decides what to remove from C_{t-1}.

  2. Input Gate & Candidate Cell State:

     • i_t = σ(W_i · [h_{t-1}, x_t] + b_i)

     • Ĉ_t = tanh(W_C · [h_{t-1}, x_t] + b_C) (new candidate values)

  3. Update Cell State: C_t = f_t ⊙ C_{t-1} + i_t ⊙ Ĉ_t (⊙ = element-wise multiply). Additive update preserves gradients.

  4. Output Gate & Hidden State:

     • o_t = σ(W_o · [h_{t-1}, x_t] + b_o)

     • h_t = o_t ⊙ tanh(C_t)

• Advantages over RNNs: Can maintain information for very long periods (memory persistence), gradient flows effectively through the additive cell state update.

Recursive Neural Networks

• Tree-Structured: Unlike sequential RNNs, they process hierarchical structures (parse trees, sentences).

• Architecture: Each node's representation is a composition function (e.g., a neural network) applied to its children's representations.

  p(node) = f( W · [p(left_child), p(right_child)] + b )

• Applications: Syntactic parsing (building parse trees), sentiment analysis (capturing phrase-level compositionality).

Autoencoders

• Goal: Learn efficient data encoding (dimensionality reduction) by training to reconstruct its input.

• Structure: Encoder z = f(x) maps input to latent code z. Decoder x̂ = g(z) reconstructs from z. Trained to minimize reconstruction loss L(x, g(f(x))).

• Sparse Autoencoders: Add a sparsity penalty (e.g., KL divergence) to the loss to force the latent code z to have few active units. Learns disentangled, meaningful features.

• Contractive Autoencoders: Add penalty on the Jacobian of the encoder: λ ||∂f(x)/∂x||_F^2. Penalizes sensitivity of latent code to input changes, leading to robust, locally invariant features.

• Autoencoders vs. PCA/SVD:

  • PCA/SVD: Linear transformation.

  • Autoencoders: Can learn non-linear manifolds. More powerful representation, but requires more data and careful training.

Generative Models

• Generative Adversarial Networks (GANs):

  • Architecture: Two networks compete:

    • Generator G: Creates fake samples from noise z.

    • Discriminator D: Classifies real vs. fake.

  • Training: Min-max game: min_G max_D V(D,G) = E[log D(x)] + E[log(1-D(G(z)))].

  • Strengths: Generates sharp, high-fidelity samples (images, audio).

  • Weaknesses: Training unstable (mode collapse), no explicit likelihood, difficult to evaluate.

• Variational Autoencoders (VAEs):

  • Architecture: Encoder outputs parameters (μ, σ) of a latent distribution (usually Gaussian). Sample z ~ N(μ, σ). Decoder reconstructs.

  • Loss: Reconstruction Loss (e.g., MSE) + KL Divergence (regularizes latent space to be close to prior N(0,I)).

  • Strengths: Stable training, structured latent space (interpolation possible), provides likelihood.

  • Weaknesses: Generated samples often blurry compared to GANs.

• GANs vs. VAEs: When to Choose?

  | Criterion | GAN | VAE | | :--- | :--- | :--- | | Sample Quality | High (sharp) | Lower (blurry) | | Latent Structure | Poorly structured | Well-structured, interpolatable | | Training Stability | Unstable | Stable | | Likelihood | Not available | Available | | Use Case | Photo-realistic generation, art | Representation learning, controlled generation, semi-supervised learning |

Deep Belief Networks (DBNs)

• Stacked Restricted Boltzmann Machines (RBMs).

• Pretraining: Train each RBM layer greedily (unsupervised) to initialize deep network weights. This was crucial for early deep learning before ReLU/Adam.

Autoregressive Models

• Model the joint distribution p(x) as a product of conditionals: p(x) = Π p(x_i | x_{<i}).

• NADE (Neural Autoregressive Distribution Estimator): Uses a neural network with masked connections to ensure autoregressive property. Each output depends only on previous inputs.

• MADE (Masked Autoencoder for Distribution Estimation): Applies a fixed binary mask to an autoencoder's weights to enforce the autoregressive property. Efficiently samples and computes exact likelihood.

VI. ADVANCED TOPICS & APPLICATIONS

Reinforcement Learning (Advanced)

• Markov Decision Processes (MDPs): Formal framework (S, A, P, R, γ).

  • S: States, A: Actions, P(s'|s,a): Transition probability, R(s,a,s'): Reward, γ: Discount factor.

  • Goal: Find policy π(a|s) that maximizes expected discounted return.

• Value Iteration (Dynamic Programming): Iteratively updates state-value function V(s) using Bellman Expectation Equation until convergence.

$$V_{k+1}(s) = \max_a \sum_{s'} P(s'|s,a) [R(s,a,s') + γ V_k(s')]$$

• Policy Iteration vs. Value Iteration:

  • Policy Iteration: Alternate between policy evaluation (compute V^π) and policy improvement (greedy update). Often faster convergence but each step is more expensive (full policy evaluation).

  • Value Iteration: Combines evaluation and improvement into one update. Simpler per step, but may require more iterations. Computationally cheaper per iteration.

• Q-Learning Algorithms (Model-Free):

  • Learns action-value function Q(s,a) directly from experience.

  • Double DQN: Addresses overestimation bias in standard DQN. Uses two networks: one to select best action (argmax_a Q(s',a; θ)), another to evaluate it (Q(s', argmax_a Q(s',a; θ); θ⁻)). Reduces positive bias.

  • Dueling DQN: Decomposes Q(s,a) into value stream V(s) and advantage stream A(s,a). Q(s,a) = V(s) + (A(s,a) - mean(A)). Learns value of state independent of specific actions, improving learning efficiency.

  • Least Squares Policy Iteration (LSPI): A batch, model-free RL algorithm using linear function approximation (e.g., Q(s,a) = θ^T φ(s,a)). Solves a least-squares problem to find θ that best fits the Bellman residual. More data-efficient than Q-learning.

Directed Graphical Models (Bayesian Networks)

• Definition: Directed acyclic graph (DAG) where nodes are random variables, edges represent conditional dependencies.

• Joint Distribution: Factorizes according to graph structure: P(X1,...,Xn) = Π_i P(X_i | Parents(X_i)).

• Use: Compact representation of probability distributions, reasoning under uncertainty, causal inference.

Unit Pruning

• Definition: Removing entire neurons (or filters in CNNs) from a trained network.

• Need:

  1. Efficiency: Reduces model size, memory footprint, and inference time.

  2. Regularization: Can act as a form of model selection, removing redundant units.

  3. Interpretability: Smaller networks may be easier to analyze.

• Methods: Based on weight magnitude, activation statistics, or contribution to loss.

Deep Dream

• A visualization technique to understand what neurons/layers "see".

• Process: Start with an input image (or noise). Gradient ascent on the input to maximize the activation of a chosen neuron/layer. The input is repeatedly modified, amplifying patterns the network recognizes.

• Result: Surreal, dream-like images revealing the network's learned features and biases.

GPU Implementation for Efficiency

• Randomized SVD on GPU:

  • Algorithm: Approximates SVD of a large matrix A by:

    1. Random projection: Y = A * Ω (Ω random Gaussian matrix).

    2. Orthonormalize Y (QR decomposition) to get Q.

    3. Form smaller matrix B = Q^T * A.

    4. Compute SVD of B: B = U_B Σ V^T.

    5. Final approximation: U ≈ Q * U_B.

  • Acceleration: The large matrix-matrix multiply A*Ω is highly parallelizable on GPU. Steps 2-4 are on much smaller matrices.

  • Applications in IR: Fast PCA/LSI on massive term-document matrices, latent factor analysis, preprocessing for neural IR models.

Logistic Regression (Detailed Working)

• Model: For binary classification, predicts probability P(y=1|x).

$$\hat{y} = σ(z) = σ(w^T x + b) = \frac{1}{1 + e^{-(w^T x + b)}}$$

• Loss Function: Binary Cross-Entropy (Log Loss).

$$L(w,b) = -\frac{1}{N} \sum_{i=1}^N \left[ y_i \log(\hat{y}_i) + (1-y_i) \log(1-\hat{y}_i) \right]$$

• Interpretation: w_j represents the change in log-odds of y=1 for a one-unit increase in x_j, holding other features constant.

• Training: Minimize loss using gradient descent (or variants like Adam). Simple, interpretable, and forms the basis for the output layer of binary classifiers in neural networks.

VII. CROSS-CUTTING THEMES (Exam-Focused Integration)

Backpropagation (Revisited)

• Universal Role: The algorithm that enables training in all architectures with differentiable components (FFNN, CNN, RNN/LSTM).

• Mechanism: Computes gradient of loss w.r.t. all parameters via chain rule, regardless of network structure. For CNNs, gradients flow through convolution operations. For RNNs/LSTMs, it's BPTT (unrolling through time).

• Key Insight: It's an efficient application of the chain rule to a computational graph.

BPTT (Backpropagation Through Time) for RNNs

• Process: The RNN is unrolled for T time steps, creating a deep feed-forward graph. Standard backpropagation is then applied to this unrolled graph.

• Challenges: The unrolled graph can be very deep (T large), leading to vanishing/exploding gradients. This is the fundamental reason standard RNNs fail on long sequences.

• Solution in LSTMs: The additive cell state update (C_t = f_t ⊙ C_{t-1} + ...) creates a nearly linear gradient flow path, mitigating the problem.

Normalization Techniques for Stable Training

• Batch Normalization: Normalizes per mini-batch during training. Uses batch statistics. Crucial for deep CNNs/MLPs.

• Layer Normalization: Normalizes across all features for a single sample. No batch dependency. Crucial for RNNs/LSTMs and transformers, as sequence lengths vary and batch statistics can be noisy for short sequences.

• Purpose: Both reduce internal covariate shift, allow higher learning rates, and act as regularizers.

Evaluation Considerations (IR vs. DL)

• IR Metrics: Precision@k, Recall@k, NDCG, MAP. Measure ranking quality and relevance of top results. Non-differentiable.

• DL Loss Functions: Cross-entropy, MSE, contrastive loss. Differentiable, used for training.

• Integration: In learning-to-rank for IR, a differentiable surrogate loss (e.g., softmax cross-entropy with pairwise or listwise approaches) is used during training. The final model is then evaluated using IR metrics (NDCG) on a validation set.