High Performance computing (AL-802 (B)) - Important Questions

Explain the concept of memory hierarchy in high performance computers. Given L1, L2 and main memory latencies and hit rates, show how to compute the average memory access time (AMAT). For example, compute AMAT when L1 hit time = 1 ns with hit rate = 95%, L2 hit time = 5 ns with local hit rate = 80% (conditional on L1 miss), and main memory access time = 100 ns.

Apply Amdahl's law to compute the theoretical speedup for a program where fraction $p$ is perfectly parallelizable and the rest is serial. Derive the formula for speedup $S$ as a function of $p$ and number of processors $n$, and discuss limits to scaling as $n\rightarrow\infty$. Use the formula to compute the speedup when $p=0.90$ for $n=4,\;8,$ and $\;16$ processors.

Given a kernel with total floating-point operations $F$ and total memory traffic $T$ (bytes), compute the operational intensity $I=\frac{F}{T}$. Using a machine with peak compute performance $P_{peak}$ (FLOP/s) and peak memory bandwidth $B$ (bytes/s), use the roofline model to determine whether the kernel is compute-bound or memory-bound and estimate the attainable performance. Apply this to a kernel with $F=2\times 10^{9}$ FLOPs and $T=1.6\times 10^{9}$ bytes on a machine with $P_{peak}=500\times 10^{9}$ FLOP/s and $B=100\times 10^{9}$ bytes/s.

Describe the principal design features of a vector processor. Explain vector pipelining, chaining, and the role of vector length and stride in performance. Discuss how memory bandwidth and alignment affect vector performance.

Define common performance metrics used in high performance computing such as FLOPS, throughput, latency, and efficiency. Briefly describe benchmark suites like LINPACK and SPEC and explain what aspect of system performance each is intended to measure.

Explain the OpenMP parallel for construct and the meaning of data scoping clauses: shared, private and reduction. Describe common pitfalls such as race conditions and false sharing and how to avoid them.

Compare MPI point-to-point communication with collective communication. When would you prefer non-blocking communication over blocking communication? Give examples where non-blocking communication improves parallel performance.