How unit 4 is examined
This unit covers ways to see data with many attributes; the marks sit in PCA (numerical), t-SNE and the confusion matrix, while parallel coordinates, scatter plot matrices, glyphs and dimension stacking are unasked but short.
Multivariate visualization techniques: parallel coordinates, scatter plot matrices
<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Not asked since 2022</span>
Definition. <mark>Multivariate visualization shows three or more variables together; a parallel coordinates plot draws each variable as a vertical axis and each record as a polyline, while a scatter plot matrix draws a grid of pairwise scatter plots.</mark>
Key points.
- In parallel coordinates every record is a line that crosses each parallel axis at its value for that attribute, so one line represents one row of the table.
- Clusters appear as bundles of similar lines, and crossing lines between two neighbouring axes show a negative correlation.
- A scatter plot matrix with $n$ variables has $n \times n$ panels, with the diagonal showing each variable's histogram or name and the off-diagonal cells showing one pair each.
- The scatter matrix reveals correlation, clusters and outliers for every pair, but it becomes unreadable when $n$ is large.
PCA (Principal Component Analysis)
<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Low weight</span>
Definition. <mark>PCA is a linear dimensionality-reduction technique that projects data onto new orthogonal axes, the principal components, which are the eigenvectors of the covariance matrix ordered by the variance they capture.</mark>
Key points.
- The first principal component is the direction of maximum variance, and each later component is orthogonal to the earlier ones.
- The eigenvalue of a component equals the variance along it, so the share of variance kept is $\lambda_i / \sum \lambda$.
- Data must be mean-centred first, and standardised if the attributes have different scales.
Steps.
Step 1: Compute the mean of each attribute.
Step 2: Subtract the mean from every value (centre the data).
Step 3: Compute the covariance matrix C.
Step 4: Solve det(C - lambda I) = 0 for the eigenvalues and find the eigenvectors.
Step 5: Choose the eigenvector of the largest eigenvalue as PC1.
Step 6: Project the centred data onto PC1: y = e1^T (x - mean).
Formula. $C = \frac{1}{n-1}\sum (x_i-\bar{x})(x_i-\bar{x})^T$ and $\det(C-\lambda I)=0$.
Example. Given $X = \{2,3,4,5,6,7\}$, $Y = \{1,5,3,6,7,8\}$, $n = 6$.
| Item | Working | Result |
|---|---|---|
| Means | $\bar{x}=27/6$, $\bar{y}=30/6$ | $4.5,\ 5$ |
| Centred X | $x-4.5$ | $-2.5,-1.5,-0.5,0.5,1.5,2.5$ |
| Centred Y | $y-5$ | $-4,0,-2,1,2,3$ |
| Sums | $\sum dx^2=17.5$, $\sum dy^2=34$, $\sum dx\,dy=22$ | divide by $n-1=5$ |
| Covariance | $\mathrm{cov}(x,x)=3.5$, $\mathrm{cov}(y,y)=6.8$, $\mathrm{cov}(x,y)=4.4$ | $C=\begin{bmatrix}3.5&4.4\\4.4&6.8\end{bmatrix}$ |
Characteristic equation: $\lambda^2 - 10.3\lambda + 4.44 = 0$, since trace $=10.3$ and determinant $=3.5\times6.8-4.4^2=4.44$. So $\lambda_1 = 9.849$ and $\lambda_2 = 0.451$. For $\lambda_1$: $(3.5-9.849)e_x + 4.4e_y = 0$, so $e_y = 1.443\,e_x$, and normalising gives $e_1 = (0.570,\ 0.822)$. Projection of the first point: $(-2.5)(0.570)+(-4)(0.822) = -4.71$; all six are $-4.71, -0.85, -1.93, 1.11, 2.50, 3.89$.
Principal component: $e_1 = (0.570, 0.822)^T$ with $\lambda_1 = 9.849$, which keeps $9.849/10.3 = 95.6\%$ of the variance.
Answer frame. Open with the definition and write the given data; list the six steps; do mean, covariance, eigenvalues and eigenvector in that order as a table; close with the boxed $e_1$ and the projected values.
Pitfall: Forgetting to subtract the mean, or dividing by $n$ in one place and $n-1$ in another, changes the covariance matrix.
Asked: [7 marks] (Dec 2024) For a given data = {2, 3, 4, 5, 6, 7 ; 1, 5, 3, 6, 7, 8}, compute the principal component using PCA algorithm.
t-SNE (t-Distributed Stochastic Neighbour Embedding)
<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Low weight</span>
Definition. <mark>t-SNE is a non-linear dimensionality-reduction technique that maps high-dimensional points to 2D or 3D so that neighbours stay close, by matching pairwise similarity distributions and minimising their KL divergence.</mark>
Key points.
- In the high-dimensional space the similarity of point $j$ to point $i$ is a Gaussian probability $p_{j|i}$, so near points get high probability and far points almost zero.
- The Gaussian width is set by the perplexity, a user parameter (typically 5 to 50) that acts as the effective number of neighbours.
- In the low-dimensional map similarity $q_{ij}$ uses a heavy-tailed Student t-distribution with one degree of freedom, which prevents crowding of points.
- Gradient descent moves the map points to minimise the Kullback-Leibler divergence between $P$ and $Q$.
- Example: on MNIST, 784-pixel digit images are reduced to a 2D map in which the ten digits form ten separate clusters.
- Advantages: it shows local structure and clusters very well. Limitations: it is slow on large data, cluster sizes and distances between clusters are not meaningful, and results change with perplexity and the random seed.
Formula. $q_{ij}=\dfrac{(1+\lVert y_i-y_j\rVert^2)^{-1}}{\sum_{k\ne l}(1+\lVert y_k-y_l\rVert^2)^{-1}}$ and cost $C=\sum_i\sum_j p_{ij}\log\dfrac{p_{ij}}{q_{ij}}$.
Answer frame. Open with the definition and its purpose for high-dimensional data; list the steps of similarity, perplexity, t-distribution map and KL optimisation; give the MNIST example; close with advantages and limitations.
Asked: [7 marks] (Dec 2024) Explain t-Distributed Stochastic Neighbor embedding technique with suitable example.
Clustering and classification visualization: dendrograms, decision trees, confusion matrices
<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Low weight</span>
Definition. <mark>A confusion matrix is a table that compares the actual classes with the predicted classes of a classifier, with the counts TP, TN, FP and FN.</mark>
Key points.
- TP (true positive) is a positive predicted as positive; TN (true negative) is a negative predicted as negative.
- FP (false positive) is a negative wrongly predicted as positive; FN (false negative) is a positive wrongly predicted as negative.
- A dendrogram is a tree that shows hierarchical clustering, where the height of a merge is the distance between the joined clusters and cutting it at a level gives the clusters.
- A decision tree shows classification as internal nodes that test an attribute, branches that give outcomes and leaves that give the class.
<figure class="ds-fig" style="margin:1.4rem 0;overflow-x:auto"><svg xmlns="http://www.w3.org/2000/svg" id="dsfig-u4-01" viewBox="0 0 252 209" width="252" height="209" role="img" aria-label="2x2 confusion matrix, actual class by row and predicted class by column. TP true positive, FN false negative, FP false positive, TN true negative"><style>#dsfig-u4-01 .e{stroke:#454C5A;stroke-width:1.4;fill:none}#dsfig-u4-01 .e.hi{stroke:#2340B8;stroke-width:2.6}#dsfig-u4-01 .n{fill:#FFFFFF;stroke:#16181D;stroke-width:1.4}#dsfig-u4-01 .n.hi{fill:#E3E9FC;stroke:#2340B8;stroke-width:2.2}#dsfig-u4-01 .n.rb-b{fill:#16181D;stroke:#16181D}#dsfig-u4-01 .n.rb-r{fill:#BD3227;stroke:#BD3227}#dsfig-u4-01 text{font-family:"JetBrains Mono",ui-monospace,Menlo,Consolas,monospace;font-size:13px}#dsfig-u4-01 .t{fill:#16181D;font-weight:500}#dsfig-u4-01 .t.inv{fill:#FFFFFF;font-weight:700}#dsfig-u4-01 .kd{stroke:#16181D;stroke-width:1.2}#dsfig-u4-01 .dot{fill:#16181D}#dsfig-u4-01 .ann{fill:#2340B8;font-size:11px;font-weight:700}#dsfig-u4-01 .lbl{fill:#6F7787;font-family:system-ui,-apple-system,sans-serif;font-size:12px;font-weight:700}#dsfig-u4-01 .ptr{fill:#2340B8;font-size:12px;font-weight:700}#dsfig-u4-01 .ah{fill:#454C5A}#dsfig-u4-01 .ah.hi{fill:#2340B8}#dsfig-u4-01 .wl rect{fill:#FFFFFF;stroke:#DCE0E7}#dsfig-u4-01 .wl .t{font-size:12px;font-weight:700}#dsfig-u4-01 .wl.hi rect{fill:#2340B8;stroke:#2340B8}#dsfig-u4-01 .wl.hi .t{fill:#FFFFFF}html.dark #dsfig-u4-01 .e{stroke:#B1B7C3}html.dark #dsfig-u4-01 .e.hi{stroke:#8FA3FF}html.dark #dsfig-u4-01 .n{fill:#161920;stroke:#E6E8ED}html.dark #dsfig-u4-01 .n.hi{fill:#1E2748;stroke:#8FA3FF}html.dark #dsfig-u4-01 .n.rb-b{fill:#E6E8ED;stroke:#E6E8ED}html.dark #dsfig-u4-01 .n.rb-r{fill:#FF7E71;stroke:#FF7E71}html.dark #dsfig-u4-01 .t{fill:#E6E8ED}html.dark #dsfig-u4-01 .t.inv{fill:#0F1115}html.dark #dsfig-u4-01 .kd{stroke:#E6E8ED}html.dark #dsfig-u4-01 .dot{fill:#E6E8ED}html.dark #dsfig-u4-01 .ann{fill:#8FA3FF}html.dark #dsfig-u4-01 .lbl{fill:#858D9C}html.dark #dsfig-u4-01 .ptr{fill:#8FA3FF}html.dark #dsfig-u4-01 .ah{fill:#B1B7C3}html.dark #dsfig-u4-01 .ah.hi{fill:#8FA3FF}html.dark #dsfig-u4-01 .wl rect{fill:#161920;stroke:#2A2E37}html.dark #dsfig-u4-01 .wl.hi rect{fill:#8FA3FF;stroke:#8FA3FF}html.dark #dsfig-u4-01 .wl.hi .t{fill:#0F1115}</style><defs><marker id="ah3" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah" d="M0,1 L9,5 L0,9 z"/></marker><marker id="ahh3" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="ah hi" d="M0,1 L9,5 L0,9 z"/></marker></defs><path class="e" d="M59,40 L193,40"/><path class="e" d="M40,59 L40,150"/><path class="e" d="M212,59 L212,150"/><path class="e" d="M59,169 L193,169"/><circle class="n" cx="40" cy="40" r="18"/><text class="t" x="40" y="40" dy=".35em" text-anchor="middle">TP</text><circle class="n" cx="212" cy="40" r="18"/><text class="t" x="212" y="40" dy=".35em" text-anchor="middle">FN</text><circle class="n" cx="40" cy="169" r="18"/><text class="t" x="40" y="169" dy=".35em" text-anchor="middle">FP</text><circle class="n" cx="212" cy="169" r="18"/><text class="t" x="212" y="169" dy=".35em" text-anchor="middle">TN</text></svg><figcaption style="font-size:.82em;opacity:.72;margin-top:.45rem">2x2 confusion matrix, actual class by row and predicted class by column. TP true positive, FN false negative, FP false positive, TN true negative</figcaption></figure>
Steps.
Step 1: List the actual and predicted class of every sample.
Step 2: Count TP, FN, FP and TN by comparing each pair.
Step 3: Fill the 2x2 table with actual on rows and predicted on columns.
Step 4: Compute accuracy, precision, recall and F1 from the counts.
Example. Ten samples, 5 actually positive and 5 negative; the classifier predicts 6 positive, of which 4 are correct.
| Predicted +ve | Predicted -ve | |
|---|---|---|
| Actual +ve | TP = 4 | FN = 1 |
| Actual -ve | FP = 2 | TN = 3 |
- Accuracy $=\dfrac{TP+TN}{TP+TN+FP+FN}=\dfrac{7}{10}=0.70$
- Precision $=\dfrac{TP}{TP+FP}=\dfrac{4}{6}=0.67$
- Recall $=\dfrac{TP}{TP+FN}=\dfrac{4}{5}=0.80$
- F1 $=\dfrac{2PR}{P+R}=0.73$
Answer frame. Open with the definition and the four terms; draw the 2x2 table; write the steps and work through your own ten-sample example; close with accuracy, precision and recall from the table.
Asked: [7 marks] (Dec 2024) What do you mean by Confusion Matrix? Explain step by step that how you will calculate confusion matrix with the given data set?
Visualizing high-dimensional data: glyph-based visualization, parallel coordinates, dimension stacking
<span style="display:inline-block;padding:.16em .6em;border:1.5px solid currentColor;border-radius:999px;font-size:.68em;font-weight:700;letter-spacing:.06em;text-transform:uppercase;opacity:.75">Not asked since 2022</span>
Definition. <mark>High-dimensional visualization encodes many attributes in one picture; glyph-based methods map attributes to the shape features of a small icon, and dimension stacking embeds one pair of dimensions inside another in a grid.</mark>
Key points.
- A glyph, such as a Chernoff face or star glyph, maps each attribute to a feature such as eye size, ray length or colour, and one glyph is drawn per record.
- Dimension stacking divides the space into a grid by two dimensions and, inside each cell, a grid by the next two, recursively, colouring cells by the value.
- Parallel coordinates place all attributes on parallel axes with one polyline per record, so many dimensions fit on one plane.
- Glyphs suit few records, while stacking and parallel coordinates handle more records but need axis ordering and interaction to stay readable.
Last-minute revision
- PCA components are eigenvectors of the covariance matrix; eigenvalue is variance; centre the data first.
- Covariance divides by $n-1$; characteristic equation is $\det(C-\lambda I)=0$.
- Dec 2024 PCA: means $4.5, 5$; $C=[3.5, 4.4; 4.4, 6.8]$; $\lambda=9.849, 0.451$; $e_1=(0.570, 0.822)$.
- t-SNE is non-linear: Gaussian similarities in high-D, Student t in low-D, KL divergence minimised.
- Perplexity is roughly the number of neighbours, typically 5 to 50.
- t-SNE distances between clusters and cluster sizes are not meaningful.
- Confusion matrix: TP, TN, FP, FN; actual on rows, predicted on columns.
- Accuracy $=(TP+TN)/N$, precision $=TP/(TP+FP)$, recall $=TP/(TP+FN)$.
- Dendrogram shows hierarchical clustering; decision tree shows class rules.
- Scatter matrix has $n\times n$ panels; parallel coordinates use one line per record.
Memory hooks
- PCA: "Centre, Covary, Eigen, Project".
- Precision looks at what you predicted positive (column); recall looks at what was actually positive (row).
- t-SNE: Similarity, Perplexity, KL, Map.
- FP is a false alarm; FN is a miss.
Coverage checklist
- Multivariate visualization techniques: parallel coordinates, scatter plot matrices: definition and points (no past questions).
- PCA (Principal Component Analysis): Dec 2024 numerical on {2..7; 1,5,3,6,7,8}.
- t-SNE (t-Distributed Stochastic Neighbour Embedding): Dec 2024 explain t-SNE with example.
- Clustering and classification visualization: dendrograms, decision trees, confusion matrices: Dec 2024 confusion matrix.
- Visualizing high-dimensional data: glyph-based visualization, parallel coordinates, dimension stacking: definition and points (no past questions).