UNIT 2: AUGMENTED AND VIRTUAL REALITY - EXAM-FOCUSED SHORT NOTES

I. FUNDAMENTALS OF IMMERSIVE TECHNOLOGIES

Definition and Core Features of Virtual Reality (VR)

• VR Definition: A computer-generated, immersive, interactive simulation of a 3D environment that users can explore and manipulate in real-time.

• The 3 I's of VR:

  1. Immersion: The sensation of being "present" inside the virtual world. Achieved through wide FOV displays, stereoscopic 3D, and spatial audio.

  2. Interaction: The ability to manipulate objects and navigate the environment naturally (e.g., using controllers, gestures).

  3. Imagination: The creative potential to design and experience environments beyond physical reality.

• Key Characteristics:

  • Synthetic Environment: Entirely computer-generated.

  • Real-Time Simulation: Graphics and physics update instantly with user action.

  • User Navigation: Ability to move through the virtual space (locomotion).

[!TIP] Exam Focus: Be ready to list and explain the 3 I's with examples. VR is about replacing reality.

Augmented Reality (AR) vs. Virtual Reality (VR)

Reality-Virtuality Continuum (Milgram Spectrum):

[!TIP] Common Pitfall: Do not confuse AR with VR. AR adds to reality; VR replaces it.

Mixed Reality (MR)

• Definition: A superset of AR where virtual objects are not just overlaid but interact with and are anchored to the real world (occlusion, physics).

• Positioning: Sits between AR and VR on the continuum. Often used synonymously with "advanced AR."

• Example: HoloLens 2 placing a virtual chair that is partially hidden behind a real table.

II. VR DEVELOPMENT & MODELING LANGUAGES

Virtual Reality Modeling Language (VRML)

• Purpose: File format standard for describing interactive 3D vector graphics, especially for the web.

• File Format: .wrl (world) files.

• Basic Structure: Hierarchical scene graph. Root node is Scene, containing Group nodes with Transform and Shape nodes.

Key Node Types with Examples:

[!TIP] Exam Focus: You must write a small VRML snippet for each node type. Remember Shape is the most basic visual node.

VR Toolkits and Frameworks

Selection Criteria:

• Target Platform: PC VR (Unity/Unreal), Mobile AR (Unity/ARKit/ARCore), Web (WebXR).

• Graphics Fidelity Needed: Unreal > Unity > WebXR.

• Team Skillset: C# (Unity) vs. C++/Blueprint (Unreal) vs. HTML/JS (WebXR).

• Development Speed: Unity often faster for prototyping.

III. COMPUTER GRAPHICS & RENDERING FOR VR/AR

Geometric Algorithms: Parametric Line Clipping (Cyrus-Beck)

• Theory: Clips a line segment P(t) = P₀ + t(P₁ - P₀), 0 ≤ t ≤ 1, against a convex polygon window.

• Uses parametric form and calculates t values for entry (t_E) and exit (t_L) points by checking against each edge's outward normal n_i.

• Algorithm Steps:

  1. For each edge i of the clip window, compute n_i · (P₁ - P₀).

  2. If n_i · (P₁ - P₀) = 0, line is parallel to edge. Check if inside.

  3. Else, compute t_i = n_i · (P_i - P₀) / n_i · (P₁ - P₀).

  4. If n_i · (P₁ - P₀) < 0, t_i is a potential entering point (t_E = max(t_E, t_i)).

  5. If n_i · (P₁ - P₀) > 0, t_i is a potential leaving point (t_L = min(t_L, t_i)).

  6. If t_E > t_L, line is completely outside.

  7. Accepted segment is P(t_E) to P(t_L).

Numerical Problem (Nov 2023):

Clip line A(10,10) to B(70,40) against window with corners (20,20) and (40,50).

• Window is a rectangle. Edges (clockwise from bottom-left (20,20)):

  1. Left: x=20, normal n₁ = (-1, 0)

  2. Bottom: y=20, normal n₂ = (0, -1)

  3. Right: x=40, normal n₃ = (1, 0)

  4. Top: y=50, normal n₄ = (0, 1)

• P₀ = (10,10), P₁ = (70,40), d = P₁ - P₀ = (60, 30)

• Edge 1 (Left, x=20): n₁·d = (-1)*60 + 0*30 = -60 < 0 (entering). t₁ = n₁·(P_i - P₀)/n₁·d. For point on edge (20, y), use P_i = (20,20) (corner). t₁ = [(-1)*(20-10) + 0*(20-10)] / -60 = 10/60 = 0.1667. t_E = max(0, 0.1667) = 0.1667.

• Edge 2 (Bottom, y=20): n₂·d = 0*60 + (-1)*30 = -30 < 0 (entering). t₂ = [0*(20-10) + (-1)*(20-10)] / -30 = 10/30 = 0.3333. t_E = max(0.1667, 0.3333) = 0.3333.

• Edge 3 (Right, x=40): n₃·d = 1*60 + 0*30 = 60 > 0 (leaving). t₃ = [1*(40-10) + 0*(20-10)] / 60 = 30/60 = 0.5. t_L = min(1, 0.5) = 0.5.

• Edge 4 (Top, y=50): n₄·d = 0*60 + 1*30 = 30 > 0 (leaving). t₄ = [0*(40-10) + 1*(50-10)] / 30 = 40/30 = 1.333. t_L = min(0.5, 1.333) = 0.5.

• Result: t_E = 0.3333, t_L = 0.5. Since t_E < t_L, line is partially inside.

  • P(t_E) = (10,10) + 0.3333*(60,30) = (10+20, 10+10) = (30, 20)

  • P(t_L) = (10,10) + 0.5*(60,30) = (10+30, 10+15) = (40, 25)

\boxed{\text{Clipped line segment: } (30, 20) \text{ to } (40, 25)}

[!TIP] Exam Tip: The Cyrus-Beck algorithm only works for convex clipping windows. For a rectangle (convex), it's perfect. Always check t_E < t_L for a visible segment.

Interpolation Techniques

[!TIP] Why Nonlinear? Linear motion can look robotic. Cubic splines provide continuous velocity and acceleration (C² continuity), essential for realistic animation.

Shading Models

• Purpose: Determine the color of each pixel on a surface based on light sources, material properties, and viewing angle to create realism.

• Algorithms:

  1. Flat Shading: Compute one normal per polygon (face normal). Lighting calculated once per polygon. Result: Faceted, low-poly look. Fast.

  2. Gouraud Shading: Compute normals at each vertex (average of adjacent face normals). Interpolate vertex colors across the polygon. Smooth appearance, but specular highlights may be missed or distorted.

  3. Phong Shading: Interpolate vertex normals across the polygon. Compute per-pixel lighting using the interpolated normal. Most realistic, captures sharp specular highlights. Computationally expensive.

[!TIP] Key Difference: Gouraud interpolates color; Phong interpolates normal and computes lighting per pixel.

Kinematics & Motion Simulation

Vector Function for Position (Worm-on-Wheel Problem - Nov 2023):

• Scenario: Wheel in y-z plane, center at origin. Spoke along +y at t=0. Worm crawls outward at 1 unit/sec along spoke. Wheel rotates at 1 radian/sec counterclockwise (view from +x).

• Position Vector r(t): Two components:

  1. Radial (along spoke): Worm's distance from center = 1 * t (since speed=1 unit/sec). Direction of spoke at time t is angle θ = ωt = 1*t = t radians from +y axis.

  2. Rotational: The entire spoke (and worm on it) rotates with the wheel.

• In y-z plane: y = radial * cos(θ) = t * cos(t), z = radial * sin(θ) = t * sin(t). x = 0.

\boxed{\vec{r}(t) = \langle 0,; t\cos t,; t\sin t \rangle}

[!TIP] Derivation Tip: Always decompose motion into radial (along spoke) and rotational (due to wheel spin). Remember the initial spoke orientation (+y axis) sets the phase.

Flight Dynamics Simulation

• Purpose: Model aircraft motion for realistic simulators (pilot training, games).

• Degrees of Freedom (6-DOF): 3 translational (surge, sway, heave) and 3 rotational.

• Primary Rotations (Euler Angles):

  1. Roll (φ): Rotation about x-axis (longitudinal). Wings up/down.

  2. Pitch (θ): Rotation about y-axis (lateral). Nose up/down.

  3. Yaw (ψ): Rotation about z-axis (vertical). Nose left/right.

• Technique: Use rotation matrices or quaternions (to avoid gimbal lock) to transform aircraft body axes to world axes. Forces (lift, drag, thrust, gravity) are applied in body frame, then transformed to world frame for position/velocity integration.

• Application: Input from control yoke/stick (pitch/roll) and pedals (yaw) changes these angles, altering the aircraft's orientation and subsequent flight path.

IV. INTERACTION & PHYSICS IN VIRTUAL ENVIRONMENTS

Models of Interaction in VE (Taxonomy)

[!TIP] Exam Focus: Be able to give a specific example for each model. Often, a single task (like "pick up a cup") involves all three: Select (look at cup), Manipulate (grab), Navigate (walk to table).

Collision Detection

• Purpose: Critical for presence, physics realism, and game rules. Prevents objects from passing through each other.

• Generic VR System Approach:

  1. Broad Phase: Quickly find potential colliding pairs from all objects.

     • Bounding Volume Hierarchies (BVH): Wrap objects in simple shapes (AABB, spheres, OBBs). Test BV-BV collisions first.

     • Sweep and Prune: Sort objects along an axis (e.g., x). Only test overlapping intervals. Efficient for dynamic scenes.

  2. Narrow Phase: For each candidate pair from broad phase, perform precise, expensive test (e.g., triangle-triangle intersection).

• Example Scenario (User picking up object):

  1. Broad Phase: User's hand (sphere BV) overlaps with table's AABB and cup's AABB.

  2. Narrow Phase: Precise test confirms hand mesh intersects cup mesh.

  3. Response: Trigger "grab" event, attach cup to hand, disable cup's gravity.

[!TIP] Key Concept: Broad phase is about speed (culling non-colliders). Narrow phase is about accuracy.

V. HARDWARE COMPONENTS & SENSORS

VR/AR Sensor Hardware

Input Devices: 3D controllers (6-DOF), data gloves (finger tracking), wands, eye-tracking modules.

Display Technologies

VR Displays:

• Head-Mounted Display (HMD): Wearable device with screens close to eyes. Types: PC-tethered (Valve Index), Standalone (Meta Quest), Smartphone-based (Cardboard).

• CAVE (Cave Automatic Virtual Environment): Room-sized with projected walls/floors. User wears shutter glasses. Immersion without headset weight.

AR Displays:

1. Optical See-Through:

   • Principle: Semi-transparent combiner (glass) reflects virtual image into user's eye while allowing real light through.

   • Diagram:

     DiagramCANVAS: Side view. User's eye sees through a slanted glass pane. Real world scene (tree) passes through glass. Virtual image (floating text) is reflected off glass into eye. Combiner merges them optically.

   • Example: Microsoft HoloLens, Google Glass.

   • Pros: No video delay, real world brightness preserved.

   • Cons: Limited brightness/contrast of virtual objects, registration errors.

2. Video See-Through:

   • Principle: Cameras capture real world, which is composited with virtual graphics and displayed on an opaque screen (like a video feed).

   • Diagram:

     DiagramCANVAS: Side view. Cameras on device capture real scene (tree). Processor combines camera feed with virtual graphics (text). Resulting video frame is displayed on an opaque screen in front of user's eye.

   • Example: Smartphone AR (ARKit/ARCore), Apple Vision Pro (in VR mode).

   • Pros: Full control over brightness/contrast, can modify real scene (e.g., background blur).

   • Cons: Latency, resolution loss, "video" feel.

[!TIP] Distinguish: Optical = see-through glass (real light enters eye directly). Video = see-through camera (real light captured by camera, then shown on screen).

Acoustic Hardware

• Purpose: Spatial Audio - Sound appears to come from a specific 3D location, crucial for presence and localization.

• Hardware:

  • Headphones/Headsets: Most common. Use Head-Related Transfer Function (HRTF) filters to simulate sound direction.

  • Speaker Arrays: Multiple speakers around a room (CAVE). Less personal, can cause crosstalk.

  • Bone Conduction: Transducers on skull, leaving ears free. Used in some AR glasses.

• Key Concept: 3D Sound uses interaural time difference (ITD) and interaural level difference (ILD) cues to simulate direction.

VI. AUGMENTED REALITY SPECIFICS

Tracking in AR

• Marker-based (Fiducial): Track predefined visual patterns (QR codes). Simple, robust, but requires placing markers. (Briefly mentioned).

• Marker-less Tracking:

  1. Sensor-based (GPS, Compass, Accelerometer):

     • Use: Outdoor, large-scale localization.

     • Example: Outdoor navigation app showing direction to a landmark. Accuracy limited (~5m for GPS).

  2. Vision-based (SLAM - Simultaneous Localization and Mapping):

     • Core Problem: "Where am I?" (localization) and "What does the world look like?" (mapping) at the same time, from a moving camera.

     • How: Detect visual features (corners) in camera frames. Track them across frames to estimate camera motion (pose) and build a sparse 3D point cloud map of the environment.

     • Example: Indoor navigation (museum guide), furniture placement (IKEA Place app). Device understands room geometry and can anchor virtual objects to real surfaces (tables, floors).

[!TIP] SLAM is the KEY to persistent AR. It allows virtual objects to stay "pinned" to a real spot as you move around.

Sensory Integration in AR

• Goal: Create a seamless, believable composite of real and virtual.

• Role of Eyes (Visual):

  • Overlay: Rendering virtual graphics aligned to real-world coordinates.

  • Registration: The most critical challenge. Ensuring virtual object's position, scale, and orientation match the real world perfectly (e.g., virtual chair's legs touch real floor). Requires accurate tracking (SLAM).

• Role of Ears (Audio):

  • Spatial Audio Cues: Sound from virtual object should emanate from its apparent location.

  • Example: In a surgical AR overlay, a beep from a virtual vital sign monitor should sound like it's coming from the monitor's position.

• Integration of Other Senses (Haptics):

  • Haptic Feedback: Vibrations or forces to simulate touch.

  • Example: In surgical AR, a controller could vibrate when the virtual surgical tool "touches" a virtual organ, providing tactile confirmation.

• Concrete Example: Medical AR Surgery Overlay

  1. Eyes: Pre-operative CT scan is registered to patient's body. Surgeon sees virtual tumor overlay on real anatomy.

  2. Ears: Spatial audio alert if surgical instrument gets too close to a critical blood vessel.

  3. Haptics: Controller resistance when cutting through virtualized tissue layers of different density.

[!TIP] Exam Answer Structure: For "role of eyes, ears, senses," use a single, detailed example (like surgery) to illustrate all three points coherently.