UNIT 4: Advanced Information Retrieval and Deep Learning

1. Fundamentals of Information Retrieval

1.1 Information Retrieval Process and Key Components

• Process: Query → Index lookup → Ranking → Results presentation.

• Key Components:

  • Document Collection: Corpus of text/data.

  • Index: Inverted index for fast lookup.

  • Query Processor: Parses and transforms user queries.

  • Ranking Algorithm: Scores documents (e.g., TF-IDF, BM25).

  • User Interface: Displays results with snippets.

[!TIP] Exam focus: Diagram the IR pipeline and explain each component's role.

1.2 Traditional IR Systems vs Web Search

1.3 Role of Artificial Intelligence in IR

• Query Understanding: NLP for intent detection, expansion.

• Ranking: Learning to Rank (LTR) models (e.g., LambdaRank).

• User Modeling: Personalization via collaborative filtering.

• Example: BERT for semantic matching between query and document.

2. Document and Query Processing

2.1 Document Preprocessing Procedures

1. Tokenization: Split text into words/terms.

2. Stop Word Removal: Eliminate common words (e.g., "the", "is").

3. Stemming/Lemmatization: Reduce to root form (e.g., "running" → "run").

4. Normalization: Case folding, accent removal.

2.2 Term Weighting: TF-IDF

• Term Frequency (TF): Frequency of term in document.

$$ \text{tf}(t,d) = \frac{f_{t,d}}{\sum_{t' \in d} f_{t',d}} $$

• Inverse Document Frequency (IDF): Downweight frequent terms.

$$ \text{idf}(t) = \log\left(\frac{N}{\text{df}(t)}\right) $$

• TF-IDF Score:

$$ \text{TF-IDF}(t,d) = \text{tf}(t,d) \times \text{idf}(t) \boxed{} $$

[!TIP] IDF reduces impact of common words; higher score = more important term in document.

2.3 Index Compression (Benefits with Examples)

• Need: Reduce storage, improve I/O speed.

• Techniques:

  • Gamma Encoding: For gaps between document IDs.

  • Variable-Byte Encoding: Byte-aligned compression.

• Example: Store gaps [5, 12, 3] instead of IDs [10, 22, 25].

• Benefit: 50-70% space reduction; faster disk reads.

2.4 Snippet Generation

• Static Snippets: Precomputed from document (e.g., first 100 words).

• Dynamic Snippets: Query-dependent; extract sentences containing query terms.

• Algorithm: Identify query term positions, return surrounding context (e.g., 20 words before/after).

[!TIP] Dynamic snippets improve relevance but increase computation.

3. Retrieval Models and Techniques

3.1 Vector Space Model and Ranking Procedures

• Documents and queries as vectors in term space.

• Ranking: Cosine similarity between query vector $\vec{q}$ and document vector $\vec{d}$:

$$ \text{score}(q,d) = \frac{\vec{q} \cdot \vec{d}}{||\vec{q}|| \cdot ||\vec{d}||} $$

• Weighting: Typically TF-IDF.

3.2 Probabilistic Relevance Feedback

• Rocchio Algorithm: Update query vector using relevant/non-relevant docs.

$$ \vec{q}_{\text{new}} = \alpha \vec{q} + \beta \frac{1}{|D_r|} \sum_{\vec{d} \in D_r} \vec{d} - \gamma \frac{1}{|D_{nr}|} \sum_{\vec{d} \in D_{nr}} \vec{d} $$

where $$\displaystyle D_r $$ = relevant docs, $$\displaystyle D_{nr} $$ = non-relevant.

• Goal: Move query toward relevant documents in vector space.

3.3 Latent Semantic Indexing (LSI) with Examples

• Idea: Use SVD to find latent topics (lower-dimensional space).

• Steps:

  1. Build term-document matrix $A$.

  2. Compute SVD: $$\displaystyle A = U \Sigma V^T $$.

  3. Truncate to $k$ dimensions: $$\displaystyle A_k = U_k \Sigma_k V_k^T $$.

• Example: Words "car", "automobile" map to same latent topic.

• Benefit: Handles synonymy, polysemy.

3.4 Retrieval Using Ranking and Relative Scores

• BM25: Probabilistic model with term frequency saturation and document length normalization.

$$ \text{score}(D,Q) = \sum_{t \in Q} \text{idf}(t) \cdot \frac{f(t,D) \cdot (k_1 + 1)}{f(t,D) + k_1 \cdot \left(1 - b + b \cdot \frac{|D|}{\text{avgdl}}\right)} $$

• Relative Scores: Combine multiple signals (e.g., BM25 + PageRank) via linear combination or learning to rank.

4. Web Information Retrieval

4.1 Web Crawlers: Advantages, Needs, and Focused Crawling

• Needs:

  • Scale: Discover billions of pages.

  • Freshness: Update changed content.

  • Politeness: Respect robots.txt, avoid overload.

• Advantages: Build search engine index, monitor web changes.

• Focused Crawling: Target specific topics (e.g., "machine learning").

  • Strategy: Use classifier to predict relevance of links; prioritize promising URLs.

  • Example: Use BFS with priority queue based on page relevance score.

4.2 XML Information Retrieval System Requirements

• Structural Queries: Support XPath/XQuery for nested elements.

• Element Retrieval: Return specific XML fragments (not whole docs).

• Ranking: Combine text relevance with structural similarity (e.g., tag weights).

• Indexing: Inverted index on element text + structural index (e.g., region encoding).

4.3 Large-Scale IR: MapReduce and Hadoop Implementation

• Map Phase: Process document chunks → emit (term, docID) pairs.

• Shuffle/Sort: Group by term.

• Reduce Phase: For each term, merge docIDs → postings list.

• Example: Inverted index construction:

  • Map: Input split → (term, 1) for each occurrence.

  • Reduce: Sum counts → (term, [(docID, freq)]).

• Benefit: Parallelizable, fault-tolerant; handles petabytes.

5. Machine Learning for Information Retrieval

5.1 Supervised vs Unsupervised Learning (with Examples)

5.2 Classification Techniques

5.2.1 Naive Bayesian Classification and Limitations

• Bayes' Theorem: $$\displaystyle P(C|D) \propto P(C) \prod_{t \in D} P(t|C) $$.

• Assumption: Features (words) conditionally independent.

• Limitations:

  • Independence assumption often violated.

  • Zero probability problem (solved by Laplace smoothing).

  • Poor with correlated features.

5.2.2 Logistic Regression

• Model: $$\displaystyle P(y=1|x) = \frac{1}{1 + e^{-(w^T x + b)}} $$.

• Training: Maximize log-likelihood via gradient descent.

• Use: Binary classification (e.g., relevant/not relevant).

• Advantage: Probabilistic output, interpretable weights.

5.2.3 k-Nearest Neighbors (Choosing K)

• Algorithm: For a query, find K most similar training instances; majority vote.

• Choosing K:

  • Elbow Method: Plot accuracy vs K; choose point of diminishing returns.

  • Cross-Validation: Test different K on validation set.

  • Rule of Thumb: $K \approx \sqrt{N}$ (N = samples).

• Distance Metric: Euclidean, cosine similarity.

5.3 Clustering Techniques

5.3.1 Agglomer Clustering

• Bottom-up: Start each doc as cluster; merge closest pairs.

• Linkage Criteria:

  • Single: min distance between clusters.

  • Complete: max distance.

  • Average: average pairwise distance.

• Stopping: Predefined number of clusters or distance threshold.

5.3.2 k-Means Clustering (Choosing K)

• Algorithm:

  1. Initialize K centroids.

  2. Assign points to nearest centroid.

  3. Update centroids as mean of assigned points.

  4. Repeat until convergence.

• Objective:

$$ J = \sum_{i=1}^{k} \sum_{x \in C_i} ||x - \mu_i||^2 \boxed{} $$

• Choosing K: Elbow method (plot J vs K), silhouette score.

5.4 Dimensionality Reduction

5.4.1 PCA/SVD vs Autoencoders

5.4.2 Autoencoders: Sparse, Contractive, Variational

• Sparse Autoencoder: Add sparsity constraint on hidden units (e.g., L1 penalty).

• Contractive Autoencoder: Penalize Jacobian norm → robust features.

• Variational Autoencoder (VAE): Probabilistic; learns latent distribution (see Section 7.6).

5.5 Recommendation Systems

5.5.1 Content-Based Recommendation Systems

• Idea: Recommend items similar to user's past likes based on item features.

• Steps:

  1. Build user profile from liked items (e.g., TF-IDF vector of item descriptions).

  2. Compute similarity between user profile and candidate items.

  3. Rank and recommend top-N.

• Advantage: No cold-start for new items; transparent.

• Limitation: Overspecialization, limited by item features.

6. Deep Learning Fundamentals

6.1 Historical Progression and Key Milestones

• 1940s-50s: Perceptron (Rosenblatt).

• 1980s: Backpropagation (Rumelhart, Hinton).

• 2006: Deep Belief Networks (Hinton) → deep learning renaissance.

• 2012: AlexNet (ReLU, dropout, GPU) wins ImageNet.

• 2014-: GANs, VAEs, transformers.

6.2 Neural Network Basics

6.2.1 Architecture of Simple and Deep Feedforward Networks

• Simple: Input layer → Hidden layer(s) → Output layer.

• Deep: Multiple hidden layers (≥2).

• Fully Connected: Each neuron connected to all in next layer.

• Example: MLP for classification: Input (features) → Hidden (ReLU) → Output (softmax).

6.2.2 Activation Functions: Importance and Types

[!TIP] ReLU most common in hidden layers; softmax for multi-class output.

6.2.3 Representation Power of MLPs and Sigmoid Neurons

• Universal Approximation Theorem: MLP with one hidden layer (sigmoid/tanh) can approximate any continuous function given enough neurons.

• Deep vs Shallow: Deep networks learn hierarchical features more efficiently (fewer parameters).

6.3 Training Neural Networks

6.3.1 Backpropagation Algorithm

1. Forward Pass: Compute output and loss.

2. Backward Pass:

   • Compute gradient of loss w.r.t. each weight via chain rule.

   • Update weights: $$\displaystyle w \leftarrow w - \eta \frac{\partial \mathcal{L}}{\partial w} $$.

[!TIP] Key: Compute local gradients at each layer; store activations during forward pass.

6.3.2 Optimization Algorithms

6.3.3 Challenges: Vanishing and Exploding Gradients (BPTT)

• Vanishing: Gradients → 0 in early layers (sigmoid/tanh); weights stop updating.

• Exploding: Gradients → large; unstable training.

• Mitigation:

  • ReLU and variants.

  • Batch Normalization.

  • Residual Connections (skip connections).

  • Gradient Clipping (for exploding).

[!TIP] Common in deep networks and RNNs (BPTT); use LSTM/GRU for RNNs.

6.4 Regularization Techniques

6.4.1 Overfitting and Underfitting

• Overfitting: Model fits noise; high train accuracy, low test accuracy.

• Underfitting: Model too simple; low train/test accuracy.

• Diagnosis: Learning curves (train vs validation loss).

6.4.2 Dropout

• Mechanism: Randomly set fraction $p$ of hidden units to 0 during training.

• Inference: Scale weights by $(1-p)$ or use inverted dropout.

• Effect: Prevents co-adaptation; ensemble effect.

• Typical $p$: 0.5 for hidden layers, 0.2 for input.

6.4.3 Weight Decay

• L2 Regularization: Add $$\displaystyle \frac{\lambda}{2} ||w||^2 $$ to loss.

• Update: $$\displaystyle w \leftarrow w - \eta (\nabla \mathcal{L} + \lambda w) $$.

• Effect: Penalizes large weights; smoother function.

6.4.4 Batch Normalization (How it Works and Advantages)

• How:

  1. For mini-batch, compute mean $$\displaystyle \mu_B $$ and variance $$\displaystyle \sigma_B^2 $$.

  2. Normalize: $$\displaystyle \hat{x}^{(i)} = \frac{x^{(i)} - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}} $$.

  3. Scale and shift: $$\displaystyle y^{(i)} = \gamma \hat{x}^{(i)} + \beta $$.

• Advantages:

  • Reduces internal covariate shift.

  • Allows higher learning rates.

  • Acts as regularizer (noise from batch statistics).

[!TIP] Applied before activation; use learned $\gamma,\beta$.

6.5 Data Preprocessing

6.5.1 Normalization vs Standardization

7. Deep Learning Architectures

7.1 Convolutional Neural Networks (CNNs)

7.1.1 Architecture and Role in Image Recognition

• Layers:

  • Convolutional: Apply filters to extract features (edges, textures).

  • Pooling: Downsample (max/average pooling).

  • Fully Connected: At end for classification.

• Role: Translation invariance, parameter sharing (efficient for images).

7.1.2 Key Terms: Padding and Pooling

• Padding: Add zeros around input to control output size.

  • Same Padding: Output size = input size.

  • Valid Padding: No padding; output shrinks.

• Pooling: Reduce spatial dimensions, increase receptive field.

  • Max Pooling: Take maximum in window (common).

  • Average Pooling: Average values.

[!TIP] Pooling provides translation invariance; stride controls downsampling.

7.1.3 Data Formats for CNNs (Examples)

[!TIP] CNNs can handle 1D (text), 2D (images), 3D (video) data.

7.2 Recurrent Neural Networks (RNNs)

7.2.1 Architecture and Handling Sequential Data

• Architecture: Hidden state $$\displaystyle h_t $$ depends on previous state $$\displaystyle h_{t-1} $$ and input $$\displaystyle x_t $$:

$$ h_t = f(W_{hh} h_{t-1} + W_{xh} x_t + b_h) $$

• Handles Sequences: Processes one time step at a time; maintains memory.

• Use: Language modeling, time series.

7.2.2 Deep RNNs and Architectures

• Deep RNNs: Stack multiple RNN layers.

  • Layer 1: $$\displaystyle h_t^{(1)} = f(W^{(1)} [h_{t-1}^{(1)}, x_t] + b^{(1)}) $$

  • Layer 2: $$\displaystyle h_t^{(2)} = f(W^{(2)} [h_{t-1}^{(2)}, h_t^{(1)}] + b^{(2)}) $$

• Bidirectional RNNs: Process sequence forward and backward; concatenate hidden states.

• Architectures: Many-to-one (sentiment), one-to-many (captioning), many-to-many (machine translation).

7.2.3 Backpropagation Through Time (BPTT) Challenges

• Unfolding: RNN unrolled for T time steps.

• Challenges:

  • Vanishing/Exploding Gradients: Gradients multiplied by same weight matrix repeatedly.

  • Long-Term Dependencies: Gradients from distant time steps vanish.

• Mitigation: LSTM/GRU, gradient clipping, proper initialization.

7.3 Long Short-Term Memory (LSTM)

7.3.1 Detailed Working Mechanism

• Gates:

  • Forget Gate: Decides what to discard from cell state.

  • Input Gate: Updates cell state with new information.

  • Output Gate: Produces hidden state from cell state.

• Cell State $$\displaystyle C_t $$: Linear path; mitigates vanishing gradient.

7.3.2 Advantages over Standard RNNs

• Handles long-term dependencies (cell state acts as memory).

• Solves vanishing gradient problem (additive updates).

• Flexible gating allows learning when to forget/remember.

7.4 Recursive Neural Networks

• Architecture: Tree-structured; same neural net applied in a recursive manner.

• Process: Leaf nodes (words) → combine via parent node → root.

• Use: Parsing, sentiment analysis (captures hierarchical structure).

• Difference from RNN: RNNs sequential; recursive neural networks tree-based.

[!TIP] Requires parse tree; less common than RNNs/LSTMs for sequence data.

7.5 Autoencoders

7.5.1 Types: Sparse, Contractive, Variational

• Sparse: Hidden layer activations penalized (L1 loss) → few active neurons.

• Contractive: Penalize Jacobian norm $$\displaystyle \left\| \frac{\partial h}{\partial x} \right\|^2 $$ → robust to input perturbations.

• Variational (VAE): See Section 7.6.2.

7.5.2 Regularization in Autoencoders

• Why: Prevent trivial identity mapping; force learning meaningful representations.

• Methods:

  • Sparsity constraint: Encourage few active units.

  • Denoising autoencoder: Train to reconstruct clean input from corrupted version.

  • Contractive penalty: As above.

7.5.3 Applications in Dimensionality Reduction

• Process: Encoder maps high-dim input to low-dim latent code; decoder reconstructs.

• Vs PCA: Non-linear; can learn complex manifolds.

• Use: Pre-training for deep networks, anomaly detection, generative modeling (VAE).

7.6 Generative Models

7.6.1 Generative Adversarial Networks (GANs)

• Architecture:

  • Generator $G$: Takes noise $z$ → generates fake data $G(z)$.

  • Discriminator $D$: Classifies real vs fake.

• Training: Minimax game:

$$ \min_G \max_D V(D,G) = \mathbb{E}_{x\sim p_{data}}[\log D(x)] + \mathbb{E}_{z\sim p_z}[\log(1-D(G(z)))] \boxed{} $$

• Use: Image generation, style transfer.

7.6.2 Variational Autoencoders (VAEs)

• Idea: Learn probabilistic latent variable model.

• Encoder: Outputs parameters $(\mu, \sigma)$ of latent distribution $q(z|x)$.

• Decoder: Reconstructs $x$ from sampled $z$.

• Loss (ELBO):

$$ \mathcal{L} = \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}(q(z|x) || p(z)) \boxed{} $$

• Use: Smooth latent space, interpolation.

7.6.3 Comparison and Use Cases

8. Advanced Deep Learning Concepts

8.1 Representation Learning

• Goal: Learn features automatically from raw data (instead of manual feature engineering).

• Methods: Autoencoders, deep networks, contrastive learning.

• Benefit: Captures hierarchical patterns; improves performance on downstream tasks.

8.2 Deep Belief Networks

• Structure: Stack of Restricted Boltzmann Machines (RBMs).

• Training: Greedy layer-wise pretraining (unsupervised) → fine-tune with backprop.

• Use: Early deep learning for initialization; now largely superseded by better initialization (He, Xavier) and ReLU.

8.3 Auto-regressive Models: NADE and MADE

• Idea: Model density $p(x)$ as product of conditionals: $$\displaystyle p(x) = \prod_{i=1}^d p(x_i | x_{<i}) $$.

• NADE (Neural Autoregressive Density Estimator): Single hidden layer with masked connections to ensure autoregressive property.

• MADE (Masked Autoencoder for Distribution Estimation): Extends NADE to deep networks; masks enforce dependency order.

• Use: Density estimation, generative modeling.

8.4 Model Compression and Pruning

8.4.1 Unit Pruning

• Idea: Remove entire neurons/filters with small impact on output.

• Method:

  1. Train large network.

  2. Evaluate importance (e.g., magnitude of weights, activation frequency).

  3. Prune least important units.

  4. Fine-tune remaining network.

• Benefit: Reduces parameters, inference speed; minimal accuracy loss.

[!TIP] Structured pruning (whole filters) more efficient for hardware than unstructured.

8.5 Visualization Techniques

8.5.1 Deep Dream

• Process: Modify input image to maximize activations of specific layers/filters.

• Algorithm:

  1. Forward pass; compute gradient of loss (activation) w.r.t. input.

  2. Update input in gradient ascent direction.

  3. Repeat.

• Result: Surreal, enhanced patterns; reveals what filters "see".

• Use: Interpretability, art.

8.6 Graphical Models

8.6.1 Directed Graphical Models

• Definition: Bayesian networks; nodes = random variables, edges = conditional dependencies.

• Factorization: Joint probability $$\displaystyle p(x_1,...,x_n) = \prod_{i=1}^n p(x_i | \text{parents}(x_i)) $$.

• Example: Naive Bayes is a directed graph (class → features).

• Use: Probabilistic reasoning, causal inference.

9. Deep Learning for Sequential Data and Decision Making

9.1 Advanced RNNs for Long-Range Dependencies

• GRUs: Gated Recurrent Units; simpler than LSTM (reset and update gates).

• Attention Mechanism: Allow decoder to attend to all encoder states (Transformer).

• Layer Normalization: Stabilize RNN training.

• Use: Machine translation, speech recognition.

9.2 Reinforcement Learning Foundations

9.2.1 Markov Decision Processes (MDPs)

• Components:

  • States $S$, Actions $A$, Transition $P(s'|s,a)$, Reward $R(s,a,s')$, Discount $\gamma$.

  • Policy $\pi(a|s)$: Probability of action in state.

• Goal: Maximize expected discounted return $$\displaystyle G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1} $$.

9.2.2 Value Iteration and Policy Iteration (Comparison)

• Value Iteration:

  • Update value function until convergence: $$\displaystyle V_{k+1}(s) = \max_a \sum_{s'} P(s'|s,a) [R(s,a,s') + \gamma V_k(s')] $$.

  • Pros: Simple, no policy evaluation step.

  • Cons: Slower per iteration (full backup).

• Policy Iteration:

  1. Policy Evaluation: Compute $$\displaystyle V^\pi $$ for current policy $\pi$ (solve linear system).

  2. Policy Improvement: Update policy greedily: $$\displaystyle \pi'(s) = \arg\max_a \sum_{s'} P(s'|s,a)[R + \gamma V^\pi(s')] $$.

  • Pros: Faster convergence per iteration.

  • Cons: Policy evaluation expensive for large state spaces.

[!TIP] Value iteration used in model-based RL; policy iteration in dynamic programming.

9.3 Deep Reinforcement Learning

9.3.1 Q-Learning and Deep Q-Networks (DQN)

• Q-Learning: Learn action-value function $Q(s,a)$.

  • Update: $$\displaystyle Q(s,a) \leftarrow Q(s,a) + \alpha [r + \gamma \max_{a'} Q(s',a') - Q(s,a)] $$.

• DQN:

  • Use deep network to approximate $Q(s,a;\theta)$.

  • Key tricks:

    • Experience Replay: Store transitions $(s,a,r,s')$; sample mini-batches.

    • Target Network: Separate network for target $$\displaystyle y = r + \gamma \max_{a'} Q(s',a';\theta^-) $$; update slowly.

  • Loss:

$$ L(\theta) = \mathbb{E}_{(s,a,r,s') \sim U(D)} \left[ \left( r + \gamma \max_{a'} Q(s',a';\theta^-) - Q(s,a;\theta) \right)^2 \right] \boxed{} $$

9.3.2 Advanced DQN Variants

• Double DQN: Decouple action selection and evaluation to reduce overestimation.

$$ y = r + \gamma Q(s', \arg\max_{a'} Q(s',a';\theta); \theta^-) $$

• Dueling DQN: Separate streams for value $V(s)$ and advantage $A(s,a)$:

$$ Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|} \sum_{a'} A(s,a') $$

9.3.3 Least Squares Methods: LSPI

• LSPI (Least Squares Policy Iteration):

  • Use least-squares to fit $Q(s,a)$ from samples (instead of gradient descent).

  • Steps:

    1. Collect dataset with current policy.

    2. Solve linear system for $Q$ weights (using basis functions).

    3. Improve policy greedily.

  • Advantage: Sample efficiency; converges in few iterations.

  • Disadvantage: Requires matrix inversion; costly for large problems.

9.3.4 Deep RL Applications

• Games: AlphaGo (policy + value networks), DQN for Atari.

• Robotics: Policy gradient methods (PPO) for control.

• Recommendation: Multi-armed bandits for personalization.

10. Integration of Deep Learning in IR Systems

10.1 Neural Networks for Ranking (Learning to Rank)

• Pointwise: Treat ranking as regression/classification on individual documents (e.g., $\text{score}(q,d)$).

• Pairwise: Learn to compare document pairs (e.g., RankNet: logistic loss on $$\displaystyle P(d_i \succ d_j|q) $$).

• Listwise: Optimize ranking metric directly (e.g., LambdaRank, ListNet).

[!TIP] LTR features: Traditional (TF-IDF) + deep (BERT embeddings).

10.2 CNNs for Image and Multimodal Retrieval

• Image Retrieval: CNN (e.g., ResNet) extracts feature vectors; compare via cosine similarity.

• Multimodal: Joint embedding space (e.g., CNN for image, LSTM for text); train with contrastive loss.

• Example: Cross-modal retrieval: query text → retrieve images.

10.3 RNNs/LSTMs for Text Retrieval and Query Understanding

• Query Understanding: LSTMs encode query into vector; capture intent, context.

• Session Modeling: RNNs model user's sequential interactions for next-query prediction.

• Use in IR: Re-ranking with semantic similarity (e.g., match query and doc vectors).

10.4 Deep Learning in Recommendation Systems

• Neural Collaborative Filtering (NCF): Replace matrix factorization with neural network on user/item embeddings.

• Sequence-aware: Use RNNs/Transformers to model user interaction history (e.g., SASRec).

• Multimodal: Combine text (BERT), images (CNN) for item representation.

10.5 Challenges and Future Directions

• Challenges:

  • Data Sparsity: Few user-item interactions.

  • Scalability: Real-time inference for millions of items.

  • Interpretability: Black-box models.

  • Cold Start: New users/items.

• Future Directions:

  • Self-supervised learning for pre-training.

  • Causal inference for recommendations.

  • Federated learning for privacy.

  • Efficient transformers for long sequences.