AI for Computer Vision (AD-701) Unit 3 Notes

How unit 3 is examined

This unit covers aligning images and 3D data from matched features, recovering camera pose and 3D structure, and estimating motion; the only topic asked recently is triangulation (7-mark comparison, Dec 2024).

2D feature-based alignment

Not asked since 2022

Definition. 2D feature-based alignment finds the 2D transformation that best maps matched feature points of one image onto the other by minimising the sum of squared residuals.

Key points.

The transformation family is chosen by its degrees of freedom: translation 2, similarity 4, affine 6, homography 8.
Least squares minimises $\sum_i \lVert x'_i - f(x_i;p)\rVert^2$; linear models such as affine give a closed-form solution.
Mismatched features are outliers, so RANSAC fits minimal samples and keeps the model with most inliers.

3D feature-based alignment

Definition. 3D feature-based alignment (absolute orientation) finds the rotation $R$ and translation $t$ that map one set of matched 3D points onto another.

Key points.

It minimises $\sum_i \lVert q_i - (Rp_i + t)\rVert^2$ over matched points $p_i,q_i$.
Subtract both centroids, form $H=\sum p_i q_i^T$, take its SVD $H=U\Sigma V^T$, and set $R=VU^T$ and $t=\bar q - R\bar p$.
When correspondences are unknown, ICP alternates nearest-point matching with this solution.

Pose estimation

Definition. Pose estimation (Perspective-n-Point, PnP) computes the camera rotation $R$ and translation $t$ from known 3D points and their 2D image projections.

Key points.

Each correspondence gives two equations, and the camera has 6 unknowns, so at least 3 points are needed (P3P gives up to 4 solutions).
The linear DLT method needs at least 6 points to solve the 3x4 projection matrix.
Results are refined by minimising reprojection error, and RANSAC rejects wrong matches.

Geometric intrinsic calibration

Definition. Intrinsic calibration estimates the internal camera parameters (focal length, principal point, skew, lens distortion) collected in the matrix $K$.

Formula. $$K=\begin{bmatrix} f_x & s & c_x\\ 0 & f_y & c_y\\ 0&0&1\end{bmatrix}$$

Key points.

Calibration images a known pattern such as a checkerboard from several views and matches the corners.
Zhang's method uses the homography of each planar view, and at least 3 views give the intrinsics.
Radial distortion coefficients are estimated by non-linear least squares.

Triangulation

Definition. Triangulation computes a 3D point as the intersection of the viewing rays from two or more cameras with known projection matrices, given the point's matched image positions.

Formula. Each view gives $x\times(PX)=0$; stacking the views gives $AX=0$, solved by SVD. For rectified stereo, depth is $Z=\dfrac{fB}{d}$ ($f$ focal length, $B$ baseline, $d$ disparity).

Method	Principle	Accuracy	Noise sensitivity	Speed
Linear (DLT)	Solve $AX=0$ by SVD	Moderate, minimises algebraic error	High, rays do not meet	Fastest, closed form
Optimal	Minimise reprojection error, e.g. two-view correction	Best, statistically optimal	Low	Slower, iterative
Robust	Use RANSAC or robust loss over many views	High with outliers	Low to outliers	Slowest

Applications: SfM, stereo depth, 3D reconstruction; a longer baseline improves accuracy.

Answer frame. Open with the definition; draw two camera centres with rays meeting at a point; then give the table; close with the applications.

Asked: [7 marks] (Dec 2024) Compare triangulation methods and their applications in 3D reconstruction.

Two-frame structure from motion

Definition. Two-frame structure from motion recovers relative camera motion and 3D structure from point matches in two images.

Key points.

Matches satisfy $x'^T F x=0$, and $F$ is estimated by the 8-point algorithm.
With calibration, $E=K'^T F K$ and $E=[t]_\times R$ is decomposed to get $R$ and $t$.
Translation is known only up to scale, so structure is recovered up to scale.

Factorization

Definition. Factorization (Tomasi-Kanade) recovers structure and motion from many frames at once by taking the SVD of the centred measurement matrix.

Key points.

The $2F\times P$ matrix $W$ of tracked points, after subtracting each row mean, has rank 3 for an affine camera.
SVD gives $W=MS$, where $M$ is motion and $S$ is 3D structure.
A metric upgrade removes the affine ambiguity, and missing data are a limitation.

Bundle adjustment

Definition. Bundle adjustment jointly refines all camera parameters and 3D points by minimising total reprojection error.

Formula. $\min \sum_{i,j}\lVert x_{ij}-\pi(P_i X_j)\rVert^2$

Key points.

It is solved by non-linear least squares such as Levenberg-Marquardt.
The Jacobian is sparse, and the Schur complement makes large problems fast.
It is the final step of SfM pipelines and needs a good initial estimate.

Constrained structure and motion

Definition. Constrained structure and motion adds prior knowledge, such as planar scenes, pure rotation or known line and point relations, to make estimation stable.

Key points.

A planar scene links two views by a homography, avoiding degeneracies of $F$.
Pure rotation gives a homography without depth.
Constraints reduce unknowns and improve accuracy.

Translational alignment

Definition. Translational alignment finds the shift $(u,v)$ that best aligns two images.

Formula. $E(u)=\sum_i [I_1(x_i+u)-I_0(x_i)]^2$ (SSD)

Key points.

Full search tests every shift, and image pyramids make it coarse-to-fine.
Cross-correlation or the FFT speeds up the search.
Lucas-Kanade gives sub-pixel refinement.

Parametric motion

Definition. Parametric motion describes the motion of a whole region by a low-order global model such as affine or homography.

Key points.

Affine has 6 parameters and homography has 8.
Parameters are found by minimising intensity difference with Gauss-Newton.
It suits planar or distant scenes and camera rotation.

Spline-based motion

Definition. Spline-based motion represents a dense flow field by interpolating displacements at a coarse grid of control points with splines.

Key points.

Each pixel's flow is a weighted sum of control-point motions using B-spline basis functions.
It handles non-rigid motion with far fewer unknowns than per-pixel flow.
Finer grids give more flexibility at more cost.

Optical flow

Definition. Optical flow is the apparent motion of brightness patterns between two frames, giving a velocity $(u,v)$ per pixel.

Formula. Brightness constancy: $I_x u+I_y v+I_t=0$.

Key points.

One equation has two unknowns (aperture problem), so a constraint is needed.
Lucas-Kanade assumes constant flow in a window and solves least squares: $\begin{bmatrix}\sum I_x^2&\sum I_xI_y\\ \sum I_xI_y&\sum I_y^2\end{bmatrix}\begin{bmatrix}u\\v\end{bmatrix}=-\begin{bmatrix}\sum I_xI_t\\ \sum I_yI_t\end{bmatrix}$.
Horn-Schunck adds a global smoothness term and gives dense flow.
Pyramids handle large motion.

Layered motion

Definition. Layered motion models a scene as several layers, each with its own parametric motion, ordering and opacity.

Key points.

Pixels are assigned to layers by segmentation, usually with the EM algorithm.
It handles occlusion and motion boundaries that a single model cannot.
Each layer's motion is affine or a homography.

Last-minute revision

Degrees of freedom: translation 2, similarity 4, affine 6, homography 8.
Absolute orientation: SVD of $H=\sum pq^T$, $R=VU^T$.
PnP: 3 points minimum, 6 for DLT.
Triangulation: $AX=0$ by SVD; stereo $Z=fB/d$.
Triangulation methods: linear (fast), optimal (accurate), robust (outliers).
Essential matrix $E=K'^TFK$; translation known up to scale.
Factorization: centred $W$ has rank 3.
Bundle adjustment: minimise reprojection error with Levenberg-Marquardt.
Optical flow: $I_xu+I_yv+I_t=0$.
Lucas-Kanade is local; Horn-Schunck is global.

Memory hooks

LOR for triangulation: Linear, Optimal, Robust.
"Bundle = all cameras and points together."
LK is Local, HS is Holistic.
Layers = one motion per layer.

Coverage checklist

2D feature-based alignment: covered.
3D feature-based alignment: covered.
Pose estimation: covered.
Geometric intrinsic calibration: covered.
Triangulation: Q1 (Dec 2024).
Two-frame structure from motion: covered.
Factorization: covered.
Bundle adjustment: covered.
Constrained structure and motion: covered.
Translational alignment: covered.
Parametric motion: covered.
Spline-based motion: covered.
Optical flow: covered.
Layered motion: covered.

Account

AI for Computer Vision (AD-701) - Unit 3 Short Notes

How unit 3 is examined

2D feature-based alignment

3D feature-based alignment

Pose estimation

Geometric intrinsic calibration

Triangulation

Two-frame structure from motion

Factorization

Bundle adjustment

Constrained structure and motion

Translational alignment

Parametric motion

Spline-based motion

Optical flow

Layered motion

Last-minute revision

Memory hooks

Coverage checklist