Deep & Reinforcement Learning (CS-702 (B)) Unit 1 Notes

How unit 1 is examined

Neurons and history, gradient-descent optimisers, RNNs with BPTT, gating (LSTM/GRU) and attention; marks sit on gradient descent with its optimisers, vanishing gradients, and the 7-mark neuron and RNN questions.

History of Deep Learning

<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Medium weight</span>

Definition. <mark>Deep learning is machine learning with neural networks that have many hidden layers, which learn features directly from raw data instead of using hand-made features.</mark>

Key points.

A shallow network has one hidden layer or none, while a deep network has two or more hidden layers stacked between input and output.
History: McCulloch-Pitts neuron (1943), Rosenblatt's perceptron (1958), the first AI winter after Minsky and Papert showed a perceptron cannot learn XOR (1969), backpropagation (1986), LeNet CNN (1989-98), and the deep revival with AlexNet winning ImageNet on GPUs (2012).
The revival happened because of big datasets, GPU compute, better activations (ReLU) and better optimisers, with frameworks such as TensorFlow and PyTorch making it practical.
Uses: vision (classification, detection), NLP (translation, chatbots), speech, and reinforcement learning (game playing, robotics).
A single layer network connects the input layer directly to the output neuron(s); each input $x_i$ has a weight $w_i$, the net input is $net=\sum w_ix_i+b$ and the output is $y=f(net)$.
Suitable activations are step (binary decision), sigmoid (probability of one class) and softmax (multi-class); it only solves linearly separable problems and is trained by the perceptron or delta rule.

Diagram. <figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u1-01" viewBox="0 0 467 338" width="467" height="338" role="img" aria-label="Single layer network: inputs x1..x3 with weights, summation plus bias, activation f, output y"><style>#dsfig-u1-01 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u1-01 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u1-01 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u1-01 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u1-01 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u1-01 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u1-01 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u1-01 .t{fill:#16181D;font-weight:500}#dsfig-u1-01 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u1-01 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u1-01 .dot{fill:#16181D}#dsfig-u1-01 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u1-01 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u1-01 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u1-01 .ah{fill:#454C5A}#dsfig-u1-01 .ah.hi{fill:#2340B8}#dsfig-u1-01 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u1-01 .wl .t{font-size:12px;font-weight:700}#dsfig-u1-01 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u1-01 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u1-01 .e{stroke:#B1B7C3}html.dark #dsfig-u1-01 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u1-01 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u1-01 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u1-01 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u1-01 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u1-01 .t{fill:#E6E8ED}html.dark #dsfig-u1-01 .t.inv{fill:#0F1115}html.dark #dsfig-u1-01 .kd{stroke:#E6E8ED}html.dark #dsfig-u1-01 .dot{fill:#E6E8ED}html.dark #dsfig-u1-01 .ann{fill:#8FA3FF}html.dark #dsfig-u1-01 .lbl{fill:#858D9C}html.dark #dsfig-u1-01 .ptr{fill:#8FA3FF}html.dark #dsfig-u1-01 .ah{fill:#B1B7C3}html.dark #dsfig-u1-01 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u1-01 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u1-01 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u1-01 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah1" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh1" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M56.3,49.8 L237,158.2" marker-end="url(#ah1)"/><path class="e" d="M59,169 L234,169" marker-end="url(#ah1)"/><path class="e" d="M56.3,288.2 L237,179.8" marker-end="url(#ah1)"/><path class="e" d="M274,169 L406,169" marker-end="url(#ah1)"/><g class="wl"><rect x="134.3" y="95.5" width="26.4" height="18" rx="9"/><text class="t" x="147.5" y="104.5" dy=".35em" text-anchor="middle">w1</text></g><g class="wl"><rect x="134.3" y="160" width="26.4" height="18" rx="9"/><text class="t" x="147.5" y="169" dy=".35em" text-anchor="middle">w2</text></g><g class="wl"><rect x="134.3" y="224.5" width="26.4" height="18" rx="9"/><text class="t" x="147.5" y="233.5" dy=".35em" text-anchor="middle">w3</text></g><g class="wl"><rect x="331.4" y="160" width="19.2" height="18" rx="9"/><text class="t" x="341" y="169" dy=".35em" text-anchor="middle">f</text></g><circle class="n" cx="40" cy="40" r="18"/><text class="t" x="40" y="40" dy=".35em" text-anchor="middle">x1</text><circle class="n" cx="40" cy="169" r="18"/><text class="t" x="40" y="169" dy=".35em" text-anchor="middle">x2</text><circle class="n" cx="40" cy="298" r="18"/><text class="t" x="40" y="298" dy=".35em" text-anchor="middle">x3</text><circle class="n" cx="255" cy="169" r="18"/><text class="t" x="255" y="169" dy=".35em" text-anchor="middle">Sum</text><circle class="n" cx="427" cy="169" r="18"/><text class="t" x="427" y="169" dy=".35em" text-anchor="middle">y</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">Single layer network: inputs x1..x3 with weights, summation plus bias, activation f, output y</figcaption></figure>

Feature	Shallow network	Deep network
Hidden layers	0 or 1	2 or more
Features	Hand-crafted	Learned hierarchically
Capacity	Low, needs many neurons	High with fewer parameters per layer
Data need	Small	Large
Training	Easy, few gradient problems	Costly, vanishing gradients
Example	Perceptron, one-hidden-layer MLP	CNN, LSTM, ResNet

Answer frame. Open with the definition of deep learning; draw the single layer network for Q6 or write the table for Q7; for Q8 give history in date order, then uses, then data and compute drivers; close by saying depth gives automatic feature learning at the cost of data and compute.

Asked: [7 marks] (Dec 2020) Explain the single layer neural network architecture with suitable activation function. Asked: [7 marks] (Nov 2023) Difference between deep and shallow network. Asked: [7 marks] (Nov 2023) What is deep learning? Explain its uses and application and history.

McCulloch Pitts Neuron

Definition. <mark>The McCulloch-Pitts neuron is the first artificial neuron: it takes binary inputs, sums them with weights, and outputs 1 if the sum reaches a threshold $\theta$, otherwise 0.</mark>

Key points.

Inputs $x_i\in\{0,1\}$ are excitatory (weight $+1$) or inhibitory (weight $-1$, and any active inhibitory input forces output 0 in the original model).
The summation unit computes $g(x)=\sum_{i=1}^{n} w_ix_i$.
The firing rule is $y=1$ if $g(x)\ge\theta$, else $y=0$, so the activation is a hard step.
AND of two inputs uses $w=1,1$ and $\theta=2$; OR uses $\theta=1$; NOT uses one inhibitory weight $-1$ with $\theta=0$.
Capabilities: any Boolean function can be built from networks of such neurons.
Limitations: weights and thresholds are fixed by hand (no learning), inputs are only binary, and a single neuron cannot compute XOR.

Diagram. <figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u1-02" viewBox="0 0 553 338" width="553" height="338" role="img" aria-label="McCulloch-Pitts neuron: binary inputs, weights, summation S, threshold T, binary output y"><style>#dsfig-u1-02 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u1-02 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u1-02 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u1-02 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u1-02 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u1-02 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u1-02 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u1-02 .t{fill:#16181D;font-weight:500}#dsfig-u1-02 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u1-02 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u1-02 .dot{fill:#16181D}#dsfig-u1-02 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u1-02 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u1-02 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u1-02 .ah{fill:#454C5A}#dsfig-u1-02 .ah.hi{fill:#2340B8}#dsfig-u1-02 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u1-02 .wl .t{font-size:12px;font-weight:700}#dsfig-u1-02 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u1-02 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u1-02 .e{stroke:#B1B7C3}html.dark #dsfig-u1-02 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u1-02 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u1-02 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u1-02 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u1-02 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u1-02 .t{fill:#E6E8ED}html.dark #dsfig-u1-02 .t.inv{fill:#0F1115}html.dark #dsfig-u1-02 .kd{stroke:#E6E8ED}html.dark #dsfig-u1-02 .dot{fill:#E6E8ED}html.dark #dsfig-u1-02 .ann{fill:#8FA3FF}html.dark #dsfig-u1-02 .lbl{fill:#858D9C}html.dark #dsfig-u1-02 .ptr{fill:#8FA3FF}html.dark #dsfig-u1-02 .ah{fill:#B1B7C3}html.dark #dsfig-u1-02 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u1-02 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u1-02 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u1-02 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh2" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M56.3,49.8 L237,158.2" marker-end="url(#ah2)"/><path class="e" d="M59,169 L234,169" marker-end="url(#ah2)"/><path class="e" d="M56.3,288.2 L237,179.8" marker-end="url(#ah2)"/><path class="e" d="M274,169 L363,169" marker-end="url(#ah2)"/><path class="e" d="M403,169 L492,169" marker-end="url(#ah2)"/><g class="wl"><rect x="134.3" y="95.5" width="26.4" height="18" rx="9"/><text class="t" x="147.5" y="104.5" dy=".35em" text-anchor="middle">w1</text></g><g class="wl"><rect x="134.3" y="160" width="26.4" height="18" rx="9"/><text class="t" x="147.5" y="169" dy=".35em" text-anchor="middle">w2</text></g><g class="wl"><rect x="134.3" y="224.5" width="26.4" height="18" rx="9"/><text class="t" x="147.5" y="233.5" dy=".35em" text-anchor="middle">wn</text></g><circle class="n" cx="40" cy="40" r="18"/><text class="t" x="40" y="40" dy=".35em" text-anchor="middle">x1</text><circle class="n" cx="40" cy="169" r="18"/><text class="t" x="40" y="169" dy=".35em" text-anchor="middle">x2</text><circle class="n" cx="40" cy="298" r="18"/><text class="t" x="40" y="298" dy=".35em" text-anchor="middle">xn</text><circle class="n" cx="255" cy="169" r="18"/><text class="t" x="255" y="169" dy=".35em" text-anchor="middle">S</text><circle class="n" cx="384" cy="169" r="18"/><text class="t" x="384" y="169" dy=".35em" text-anchor="middle">T</text><circle class="n" cx="513" cy="169" r="18"/><text class="t" x="513" y="169" dy=".35em" text-anchor="middle">y</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">McCulloch-Pitts neuron: binary inputs, weights, summation S, threshold T, binary output y</figcaption></figure>

Answer frame. Open with the definition; draw the diagram; then the equation and firing rule, the AND/OR gate example, and limitations; close by noting the perceptron later added learnable weights.

Asked: [7 marks] (Dec 2020, Nov 2023) Draw and explain McCulloch Pitts neuron model.

Thresholding Logic

Definition. Threshold logic implements a Boolean function by comparing a weighted sum of inputs with a threshold.

Key points.

The output is $y=1$ if $\sum w_ix_i\ge\theta$, else 0.
Choosing weights and $\theta$ realises AND, OR and NOT gates.
Functions that are not linearly separable, such as XOR, cannot be realised by one threshold unit.

Activation functions

Definition. <mark>An activation function is a function applied to a neuron's net input to produce its output, and it must be non-linear for depth to add power.</mark>

Key points.

Sigmoid: $\sigma(z)=1/(1+e^{-z})$ gives outputs in (0,1); tanh gives (-1,1); ReLU is $\max(0,z)$.
Proof that linear hidden units collapse: layer one gives $h_1=W_1x+b_1$ and layer two gives $h_2=W_2h_1+b_2=W_2W_1x+W_2b_1+b_2=W'x+b'$ with $W'=W_2W_1$, $b'=W_2b_1+b_2$.
By induction, if $k$ layers collapse to $W_kx+b_k$ form, one more linear layer again gives $W'x+b'$, so any depth equals one matrix.
Hence the MLP is equivalent to a single layer perceptron with weights $W'$, and it loses all non-linear power.

Asked: [7 marks] (Dec 2020) If the activation function of all hidden unit is linear, show that a MLP is equivalent to a single layer perceptron.

Gradient Descent (GD)

Definition. <mark>Gradient descent minimises the loss $L(\theta)$ by repeatedly moving the parameters a small step opposite to the gradient: $\theta \leftarrow \theta-\eta\nabla_\theta L$.</mark>

Key points.

The gradient of the loss with respect to every weight is computed by backpropagation, and GD uses it to reduce the loss.
$\eta$ is the learning rate: too small gives slow convergence, too large makes the loss oscillate or diverge.
Significance: it is the only practical way to train networks with millions of parameters, since the loss is non-convex and has no closed-form solution.
It scales to deep networks because each step needs only gradients, and it converges to a local minimum or good plateau for a suitable $\eta$.
Batch GD uses all data per step (stable but slow); its drawbacks are slow steps on flat regions, zig-zag in ravines and one $\eta$ for all parameters.
AdaGrad, RMSProp and Adam (sections below) fix the single-learning-rate problem by adapting the step per parameter.

Steps.

Step 1: Initialise weights theta randomly.
Step 2: Forward pass, compute loss L.
Step 3: Backpropagate to get the gradient of L.
Step 4: Update theta = theta - eta * gradient.
Step 5: Repeat until the loss stops decreasing.

Example. For $L=\theta^2$, $\theta_0=4$, $\eta=0.1$: gradient $2\theta=8$, so $\theta_1=4-0.8=3.2$, then $\theta_2=3.2-0.64=2.56$, moving towards 0.

Answer frame. Open with the definition and update rule; then significance points 1-4; then the optimisers in order Momentum, AdaGrad, RMSProp, Adam with their formulas; close by saying Adam is the usual default.

Asked: [14 marks] (Jun 2025) What is the significance of gradient descent in training deep learning models? Discuss various optimization algorithms including AdaGrad, RMSProp, and Adam.

Momentum Based GD

Definition. Momentum GD adds a velocity that accumulates past gradients: $v_t=\gamma v_{t-1}+\eta\nabla L(\theta)$, $\theta\leftarrow\theta-v_t$.

Key points.

It behaves like a heavy ball, so steps in a consistent direction grow larger.
It damps oscillation across ravines and speeds up flat regions.
$\gamma$ is typically 0.9.

Nesterov Accelerated GD

Definition. NAG computes the gradient at the look-ahead point: $v_t=\gamma v_{t-1}+\eta\nabla L(\theta-\gamma v_{t-1})$, $\theta\leftarrow\theta-v_t$.

Key points.

It first jumps by the momentum and then corrects using the gradient there.
This look-ahead reduces overshooting compared with plain momentum.
It converges faster near minima.

Stochastic GD

Definition. SGD updates the parameters using the gradient of one example (or a mini-batch) instead of the whole dataset.

Key points.

Each update is cheap, so learning starts immediately on huge datasets.
The gradient is noisy, so the loss fluctuates, which can help escape shallow local minima.
Mini-batch (32-256 samples) balances speed and stability and is the standard.

AdaGrad

Definition. AdaGrad gives each parameter its own learning rate: $G_t=G_{t-1}+g_t^2$, $\theta\leftarrow\theta-\dfrac{\eta}{\sqrt{G_t}+\epsilon}g_t$.

Key points.

Parameters with rare or small gradients get large steps, so it suits sparse data.
It needs little tuning of $\eta$.
Drawback: $G_t$ only grows, so the learning rate shrinks towards zero and learning stops early.

RMSProp

Definition. RMSProp replaces the AdaGrad sum by an exponential moving average: $E[g^2]_t=\beta E[g^2]_{t-1}+(1-\beta)g_t^2$, $\theta\leftarrow\theta-\dfrac{\eta}{\sqrt{E[g^2]_t}+\epsilon}g_t$.

Key points.

The decaying average forgets old gradients, so the rate does not vanish.
Typical $\beta=0.9$.
It works well for non-stationary problems and RNNs.

Adam

Definition. Adam combines momentum and RMSProp: $m_t=\beta_1m_{t-1}+(1-\beta_1)g_t$, $v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2$, $\theta\leftarrow\theta-\eta\dfrac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}$.

Key points.

Bias correction $\hat m_t=m_t/(1-\beta_1^t)$ and $\hat v_t=v_t/(1-\beta_2^t)$ fixes the start-up bias towards zero.
Defaults are $\beta_1=0.9$, $\beta_2=0.999$, $\epsilon=10^{-8}$, $\eta=0.001$.
It is fast, robust and the most widely used default optimiser.

Eigenvalue Decomposition

Definition. ==Eigenvalue decomposition writes a square matrix as $A=Q\Lambda Q^{-1}$, where columns of $Q$ are eigenvectors satisfying $Av=\lambda v$ and $\Lambda$ holds the eigenvalues.==

Key points.

PCA is unsupervised dimensionality reduction: it finds orthogonal directions (principal components) of maximum variance.
Steps: centre the data, compute the covariance matrix $C$, find its eigenvalues and eigenvectors, keep the top $k$ eigenvectors, and project $Z=XW_k$.
An RNN is a supervised sequence model whose hidden state is fed back at each time step, so it handles sequential data.
So PCA reduces features without labels, while an RNN learns a mapping over sequences from labelled data.

Asked: [7 marks] (Nov 2023) What is PCA (Principle Component Analysis) and RNN?

Recurrent Neural Networks

Definition. <mark>A recurrent neural network is a network with a feedback loop in which the hidden state at time $t$ depends on the current input and the previous hidden state, so it processes sequences.</mark>

Key points.

The equations are $h_t=\tanh(W_{hh}h_{t-1}+W_{xh}x_t+b)$ and $y_t=W_{hy}h_t$.
Unfolded in time, it is one copy of the same cell per step with shared weights.
The hidden state acts as memory of earlier inputs.
Applications are language modelling, translation, speech recognition and time-series prediction.
Plain RNNs suffer from vanishing gradients, so they forget long-term dependencies.

Asked: [7 marks] (Nov 2023) What are Recurrent Neural Networks?

Backpropagation through time (BPTT)

Definition. <mark>BPTT unfolds the RNN over $T$ steps and applies backpropagation to the unfolded network, summing the gradients of the shared weights over all time steps.</mark>

Key points.

Loss is $L=\sum_t L_t$, with $h_t=\tanh(W_{hh}h_{t-1}+W_{xh}x_t)$ and $y_t=W_{hy}h_t$.
Output weights: $\dfrac{\partial L}{\partial W_{hy}}=\sum_t\dfrac{\partial L_t}{\partial y_t}h_t^\top$.
Recurrent weights, by the chain rule: $\dfrac{\partial L}{\partial W_{hh}}=\sum_t\sum_{k\le t}\dfrac{\partial L_t}{\partial y_t}\dfrac{\partial y_t}{\partial h_t}\Big(\prod_{j=k+1}^{t}\dfrac{\partial h_j}{\partial h_{j-1}}\Big)\dfrac{\partial h_k}{\partial W_{hh}}$.
Input weights $W_{xh}$ get the same form; the gradients from all steps are added because the weights are shared.
Steps: forward pass over the sequence, compute the loss, backward pass from $t=T$ to 1, sum, update.
The product of Jacobians causes vanishing gradients.

Asked: [7 marks] (Dec 2020) Derive the Back Propagation Through Time (BPTT) algorithm used to train the recurrent neural network.

Vanishing and Exploding Gradients

Definition. <mark>In an RNN, gradients are repeated products of the Jacobian $\partial h_j/\partial h_{j-1}$; they shrink to zero (vanish) or grow without bound (explode) as the sequence gets longer.</mark>

Key points.

In BPTT, $\dfrac{\partial h_t}{\partial h_k}=\prod_{j=k+1}^{t}W_{hh}^\top\,\mathrm{diag}(1-h_j^2)$, so the gradient is multiplied $t-k$ times.
If the largest singular value of $W_{hh}$ times the activation slope is below 1, the product tends to 0 (vanishing); if it is above 1, it blows up (exploding).
Effect of vanishing: early time steps get almost no update, so the network cannot learn long-term dependencies.
Effect of exploding: weights change hugely, the loss becomes NaN and training is unstable.
Exploding gradients are fixed by gradient clipping; vanishing needs architectural change.
LSTM has a cell state updated additively, $c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$, so gradients flow along it, and the forget gate stays near 1 to keep memory.
GRU uses an update gate $z_t$ that blends old and new state, $h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t$, giving a similar shortcut path.

Aspect	Vanishing	Exploding
Cause	Jacobian norm below 1	Jacobian norm above 1
Effect	Cannot learn long dependencies	Unstable, NaN loss
Remedy	LSTM, GRU, ReLU, good init	Gradient clipping

Answer frame. Open by stating that BPTT multiplies gradients repeatedly; explain causes and effects with the table; then LSTM gates (input, forget, output, cell state), then GRU gates; close by saying gating gives an additive path that preserves long-term dependencies.

Asked: [14 marks] (Jun 2025) Describe the challenges of vanishing and exploding gradients in Recurrent Neural Networks (RNNs). How do GRUs and LSTMs mitigate these issues?

Truncated BPTT

Definition. Truncated BPTT backpropagates only through the last $k$ time steps instead of the whole sequence.

Key points.

The hidden state is still carried forward across chunks, but gradients stop at the chunk boundary.
It cuts memory and computation per update.
It limits vanishing or exploding effects but cannot learn dependencies longer than $k$.

GRU

Definition. A Gated Recurrent Unit is a simplified LSTM with two gates and no separate cell state.

Key points.

Update gate $z_t=\sigma(W_z[h_{t-1},x_t])$ decides how much old state to keep, and reset gate $r_t=\sigma(W_r[h_{t-1},x_t])$ decides how much past to use.
Candidate $\tilde h_t=\tanh(W[r_t\odot h_{t-1},x_t])$ and $h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t$.
It has fewer parameters than LSTM and trains faster.

LSTMs

Definition. <mark>Long Short-Term Memory is a recurrent cell with a cell state and three sigmoid gates that control what to forget, write and output, designed to learn long-term dependencies.</mark>

Key points.

Forget gate $f_t=\sigma(W_f[h_{t-1},x_t]+b_f)$ decides what to erase from the cell state.
Input gate $i_t=\sigma(W_i[h_{t-1},x_t]+b_i)$ with candidate $\tilde c_t=\tanh(W_c[h_{t-1},x_t]+b_c)$ decides what to add.
Cell state $c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$; output gate $o_t=\sigma(W_o[h_{t-1},x_t]+b_o)$ and $h_t=o_t\odot\tanh(c_t)$.
Advantage over vanilla RNN: the additive cell path avoids vanishing gradients.

<figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u1-03" viewBox="0 0 510 209" width="510" height="209" role="img" aria-label="LSTM cell: cell state C (top line), forget gate F, input gate I with tanh candidate G, output gate O giving hidden state H"><style>#dsfig-u1-03 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u1-03 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u1-03 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u1-03 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u1-03 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u1-03 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u1-03 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u1-03 .t{fill:#16181D;font-weight:500}#dsfig-u1-03 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u1-03 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u1-03 .dot{fill:#16181D}#dsfig-u1-03 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u1-03 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u1-03 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u1-03 .ah{fill:#454C5A}#dsfig-u1-03 .ah.hi{fill:#2340B8}#dsfig-u1-03 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u1-03 .wl .t{font-size:12px;font-weight:700}#dsfig-u1-03 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u1-03 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u1-03 .e{stroke:#B1B7C3}html.dark #dsfig-u1-03 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u1-03 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u1-03 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u1-03 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u1-03 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u1-03 .t{fill:#E6E8ED}html.dark #dsfig-u1-03 .t.inv{fill:#0F1115}html.dark #dsfig-u1-03 .kd{stroke:#E6E8ED}html.dark #dsfig-u1-03 .dot{fill:#E6E8ED}html.dark #dsfig-u1-03 .ann{fill:#8FA3FF}html.dark #dsfig-u1-03 .lbl{fill:#858D9C}html.dark #dsfig-u1-03 .ptr{fill:#8FA3FF}html.dark #dsfig-u1-03 .ah{fill:#B1B7C3}html.dark #dsfig-u1-03 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u1-03 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u1-03 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u1-03 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah3" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh3" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M59,40 L449,40" marker-end="url(#ah3)"/><path class="e" d="M115.5,153.2 L51.6,57.5" marker-end="url(#ah3)"/><path class="e" d="M196.8,157.6 L56.8,52.6" marker-end="url(#ah3)"/><path class="e" d="M279,169 L233,169" marker-end="url(#ah3)"/><path class="e" d="M394.5,153.2 L458.4,57.5" marker-end="url(#ah3)"/><circle class="n" cx="40" cy="40" r="18"/><text class="t" x="40" y="40" dy=".35em" text-anchor="middle">C</text><circle class="n" cx="126" cy="169" r="18"/><text class="t" x="126" y="169" dy=".35em" text-anchor="middle">F</text><circle class="n" cx="212" cy="169" r="18"/><text class="t" x="212" y="169" dy=".35em" text-anchor="middle">I</text><circle class="n" cx="298" cy="169" r="18"/><text class="t" x="298" y="169" dy=".35em" text-anchor="middle">G</text><circle class="n" cx="384" cy="169" r="18"/><text class="t" x="384" y="169" dy=".35em" text-anchor="middle">O</text><circle class="n" cx="470" cy="40" r="18"/><text class="t" x="470" y="40" dy=".35em" text-anchor="middle">H</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">LSTM cell: cell state C (top line), forget gate F, input gate I with tanh candidate G, output gate O giving hidden state H</figcaption></figure>

Asked: [7 marks] (Nov 2023) Explain LSTM (Long Short Term Memory).

Encoder Decoder Models

Definition. An encoder-decoder (seq2seq) model uses an encoder RNN to compress the input sequence into a context vector and a decoder RNN to generate the output sequence from it.

Key points.

Input and output lengths can differ, as in translation.
The encoder's final hidden state $c$ is the context, and the decoder predicts each word from $c$ and its previous outputs.
A single fixed vector is a bottleneck for long sequences, which attention solves.

Attention Mechanism

Definition. Attention lets the decoder look at all encoder states and weight them: $e_{ti}=\mathrm{score}(s_{t-1},h_i)$, $\alpha_{ti}=\mathrm{softmax}(e_{ti})$, $c_t=\sum_i\alpha_{ti}h_i$.

Key points.

Each output step builds a fresh context vector $c_t$ focused on the relevant inputs.
This removes the fixed-vector bottleneck and improves long sentences.
The weights $\alpha$ are interpretable as alignment.
Self-attention (transformers) uses $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$.

Attention over images

Definition. Visual attention lets a model focus on different image regions at each step, for example while captioning.

Key points.

A CNN gives a grid of region feature vectors, and an RNN decoder weights them with attention at each word.
The weights show which region produced each word.
Soft attention is differentiable and trained by backpropagation; hard attention samples one region.

Last-minute revision

Deep network means 2 or more hidden layers; a shallow network has 0 or 1.
McCulloch-Pitts neuron: binary inputs, $y=1$ if $\sum w_ix_i\ge\theta$; AND has $\theta=2$, OR has $\theta=1$.
Linear activations make any MLP equal to one layer: $W'=W_2W_1$.
GD update: $\theta\leftarrow\theta-\eta\nabla L$.
AdaGrad's rate shrinks because $G_t$ only accumulates; RMSProp uses a moving average.
Adam: $\beta_1=0.9$, $\beta_2=0.999$, with bias correction.
RNN: $h_t=\tanh(W_{hh}h_{t-1}+W_{xh}x_t)$; BPTT sums gradients over time.
Vanishing gradient is Jacobian norm below 1; exploding is above 1; clip for exploding.
LSTM has 3 gates (forget, input, output) and a cell state; GRU has 2 (update, reset).
Attention: $c_t=\sum\alpha_{ti}h_i$ with softmax weights.

Memory hooks

FIO for LSTM gates: Forget, Input, Output.
GRU is "Zero Reset": Z for update, R for reset, only two gates.
AdaGrad Accumulates, RMSProp Remembers only recently, Adam = Momentum + RMSProp.
Vanish = Very small, Explode = Enormous; clip explosions, gate vanishing.

Coverage checklist

History of Deep Learning: Q6, Q7, Q8
McCulloch Pitts Neuron: Q10
Thresholding Logic: no past question
Activation functions: Q3
Gradient Descent (GD): Q1
Momentum Based GD: covered under Q1
Nesterov Accelerated GD: covered under Q1
Stochastic GD: covered under Q1
AdaGrad: Q1
RMSProp: Q1
Adam: Q1
Eigenvalue Decomposition: Q5
Recurrent Neural Networks: Q11, Q5
Backpropagation through time (BPTT): Q4
Vanishing and Exploding Gradients: Q2
Truncated BPTT: no past question
GRU: Q2
LSTMs: Q9, Q2
Encoder Decoder Models: no past question
Attention Mechanism: no past question
Attention over images: no past question

Account

Deep & Reinforcement Learning (CS-702 (B)) - Unit 1 Short Notes

How unit 1 is examined

History of Deep Learning

McCulloch Pitts Neuron

Thresholding Logic

Activation functions

Gradient Descent (GD)

Momentum Based GD

Nesterov Accelerated GD

Stochastic GD

AdaGrad

RMSProp

Adam

Eigenvalue Decomposition

Recurrent Neural Networks

Backpropagation through time (BPTT)

Vanishing and Exploding Gradients

Truncated BPTT

GRU

LSTMs

Encoder Decoder Models

Attention Mechanism

Attention over images

Last-minute revision

Memory hooks

Coverage checklist