Deep & Reinforcement Learning (CY-802 (B)) - Important Questions

Describe the architecture of LeNet-5. Draw and explain each layer, including kernel sizes, feature map dimensions, activation functions, and parameter count. How does LeNet perform spatial downsampling and what are its limitations on modern image tasks?

Explain the AlexNet architecture and its key innovations (ReLU activations, local response normalization, overlapping pooling, data augmentation, dropout). Discuss why these choices enabled successful training on the ImageNet dataset.

Describe the VGG family of networks. Explain why stacking multiple $3\times 3$ convolutional layers is effective compared to single larger filters. Compare parameter count and receptive field growth for a stack of two $3\times 3$ convolutions versus a single $5\times 5$ convolution.

Explain the motivation and structure of ResNet (residual networks). Show how a residual block implements learning of $F(x)+x$ and explain, at a high level, why identity (skip) connections help with training very deep networks (in terms of gradient flow and optimization landscape).

Describe the Inception (GoogLeNet) architecture and the design of an Inception module. Explain the motivation for using parallel filters of different sizes and the role of $1\times 1$ convolutions and auxiliary classifiers in the architecture.

Derive and explain the formula for the spatial output size $O$ of a 2D convolution given input size $I$, kernel size $K$, padding $P$, and stride $S$:

$O = \left\lfloor \frac{I + 2P - K}{S} \right\rfloor + 1$.

Explain each term and compute the output size for an example with $I=224$, $K=7$, $P=3$, and $S=2$.

Explain greedy layer-wise pre-training for deep neural networks. Describe the algorithmic steps (how each layer is trained), the motivation behind this approach, and how pre-training helps representation learning and optimization. Relate the method to stacked autoencoders and stacked Restricted Boltzmann Machines (RBMs).

Discuss weight initialization strategies for deep networks. Present the formulas and intuition for Xavier (Glorot) initialization and He initialization, and explain when each should be used depending on activation functions (e.g., sigmoid/tanh vs. ReLU).

Explain distributional word vectors and the skip-gram model. Write the skip-gram objective and describe the negative sampling approximation. Explain why skip-gram with negative sampling learns semantic and syntactic relationships between words.