Augmented and Virtual Reality (AL-703 (B)) - Unit 1 Short Notes

UNIT 1: CORE CONCEPTS, HARDWARE, SOFTWARE & MATHEMATICAL FOUNDATIONS

1. Core Concepts and Definitions

Virtual Reality (VR)

Definition: A computer-generated, immersive simulation of a 3D environment that users can interact with in a seemingly real or physical way through specialized hardware (e.g., HMD).

Essential 3 I's Features:

• Immersion: Sensory isolation from the real world, creating a sense of "being there" (presence). Achieved via HMDs, spatial audio.

• Interaction: Real-time response to user actions (head/hand movement, controllers). Enables manipulation of virtual objects.

• Imagination: Ability to create and experience environments, objects, or scenarios that do not exist in reality (design, training, entertainment).

Augmented Reality (AR)

Definition: A technology that superimposes computer-generated digital content (images, text, 3D models) onto the user's real-world view in real-time, enhancing rather than replacing reality.

Key Characteristics:

• Overlaying Digital Content: Combines real and virtual worlds.

• Real-Time Interaction: Digital content responds to changes in the real environment (e.g., object placement on a table).

• 3D Registration: Virtual objects are accurately aligned and anchored in the real 3D space.

Mixed Reality (MR) Continuum

Definition: A spectrum (continuum) ranging from the entirely real environment to the entirely virtual environment. AR and VR are subsets of this continuum.

Spectrum:

Real Environment <-------------------> Virtual Environment

    |                |                |

(See-through AR) (Video-see-through AR) (Immersive VR)

• AR: Real world dominant, virtual objects overlaid.

• VR: Virtual world dominant, real world blocked.

• MR (sometimes used synonymously with AR): Often refers to advanced AR where virtual objects interact convincingly with the real world (occlusion, physics).

Comparison between AR and VR

Diagram Concept:

[!TIP] Exam Tip: Remember the 3 I's for VR and "overlaying" for AR. The key difference is replacement vs. enhancement.

Human Senses in AR

AR primarily leverages:

• Vision (Primary): Displays (HMD, phone) must provide high-resolution, low-latency, correctly registered graphics. Critical for convincing registration.

• Hearing (Secondary): 3D Spatial Audio cues users to the location of virtual sound sources relative to the real world (e.g., a virtual dog barking from behind).

• Other Senses (Emerging):

  • Haptics: Vibrations or force feedback to simulate touch of virtual objects (e.g., feeling a virtual button press).

  • Proprioception: User's sense of body position must align with virtual interactions.

Example: An AR maintenance app shows a floating 3D diagram over a machine (vision), plays a "click" sound when a virtual part is correctly placed (hearing), and the controller vibrates upon "contact" (haptics).

2. Hardware Infrastructure

Sensor Systems (for VR/AR)

• Motion Trackers: Measure head/body position and orientation.

  • Inertial Measurement Units (IMU): Accelerometers, gyroscopes, magnetometers. Low latency, prone to drift.

  • Optical Trackers: External cameras (e.g., Oculus Constellation) or inside-out cameras (on HMD) track infrared LEDs or visual features.

  • Magnetic Trackers: Use alternating magnetic fields (less common now due to interference).

• Haptic Feedback Devices: Provide tactile sensation.

  • Vibrotactile: Small motors (e.g., controller rumble).

  • Force Feedback: Exoskeletons or grounded devices (e.g., Novint Falcon) that resist user motion.

• Eye-Tracking Sensors: Infrared cameras to detect pupil position and gaze direction (for foveated rendering, interaction).

Display Technologies

Virtual Reality Displays

1. Head-Mounted Display (HMD):

   • Stereo Displays: Separate screen for each eye, creating depth.

   • Lenses: Magnify screen to fill FOV, correct distortion.

   • Types: PC-tethered (high fidelity), Standalone (e.g., Quest), Mobile (smartphone-based).

2. CAVE (Cave Automatic Virtual Environment):

   • Concept: A room-sized cube with rear-projected screens on walls, floor, ceiling.

   • User wears lightweight shutter glasses. Multiple projectors create a seamless 3D environment.

   • Advantage: High resolution, multi-user, natural walking.

   • Disadvantage: Expensive, fixed space.

Augmented Reality Displays

Acoustic Hardware

• 3D Audio / Spatial Sound: Simulates how sound propagates in 3D space.

  • Purpose: Enhance immersion, provide auditory cues for location (e.g., hearing an enemy approach from behind).

  • Implementation: Uses Head-Related Transfer Function (HRTF)—filters that account for how the human head, torso, and ears shape sound. Personalized HRTFs yield best results.

  • Hardware: Standard stereo headphones can be used with software HRTF processing. Specialized speaker arrays (ambisonics) for room-scale.

3. Software Frameworks and Modeling

Virtual Reality Modeling Language (VRML)

Overview: A standard file format for representing 3D interactive vector graphics, designed for the web (precursor to X3D). Describes 3D scenes as a hierarchical scene graph.

Purpose: To create and share 3D worlds that can be navigated and interacted with in real-time across different platforms.

Key Node Types (with examples):

VR Toolkits and Development Environments

[!TIP] Exam Tip: Know one key feature of each major toolkit (Unity's XR Interaction Toolkit, Unreal's Blueprint, OpenXR's hardware abstraction).

4. Mathematical and Graphical Foundations

Geometric Algorithms: Parametric Line Clipping

Purpose: To efficiently determine the visible portion of a line segment within a rectangular clipping window.

Algorithm (Parametric Form):

1. Represent line segment parametrically: $$\displaystyle P(t) = P_0 + t(P_1 - P_0) $$, where $t \in [0,1]$.

2. For each window edge (left: $$\displaystyle x = w_{xmin} $$, right: $$\displaystyle x = w_{xmax} $$, bottom: $$\displaystyle y = w_{ymin} $$, top: $$\displaystyle y = w_{ymax} $$), compute $t$ where line intersects.

• For vertical edge $$\displaystyle x = w_x $$: $$\displaystyle t = \frac{w_x - x_0}{x_1 - x_0} $$ (if $$\displaystyle x_1 \neq x_0 $$).

3. Collect all $t$ values. Let $$\displaystyle t_E $$ be the largest entering $t$ (where line enters window), $$\displaystyle t_L $$ be the smallest leaving $t$ (where line exits).

4. Decision: If $$\displaystyle t_E > t_L $$, line is entirely outside. Otherwise, clipped segment is $$\displaystyle P(t_E) $$ to $$\displaystyle P(t_L) $$.

Example Problem (from Nov 2023):

Clip line $A(10,10)$ to $B(70,40)$ against window with corners $(20,20)$ and $(40,50)$.

Solution:

1. Parametric: $$\displaystyle x = 10 + 60t $$, $$\displaystyle y = 10 + 30t $$, $t \in [0,1]$.

2. Window: $x \in [20,40]$, $y \in [20,50]$.

3. Compute $t$ for each boundary:

• $$\displaystyle x=20 $$: $$\displaystyle 20 = 10 + 60t \Rightarrow t = 10/60 = 0.1667 $$ (entering, as $x$ increases)

• $$\displaystyle x=40 $$: $$\displaystyle 40 = 10 + 60t \Rightarrow t = 30/60 = 0.5 $$ (leaving)

• $$\displaystyle y=20 $$: $$\displaystyle 20 = 10 + 30t \Rightarrow t = 10/30 = 0.3333 $$ (entering)

• $$\displaystyle y=50 $$: $$\displaystyle 50 = 10 + 30t \Rightarrow t = 40/30 = 1.333 $$ (outside $[0,1]$, ignore)

4. Entering $$\displaystyle t_E = \max(0.1667, 0.3333) = 0.3333 $$

Leaving $$\displaystyle t_L = \min(0.5, 1) = 0.5 $$

5. Since $$\displaystyle t_E < t_L $$, segment is partially inside.

6. Clipped endpoints:

• Start: $$\displaystyle P(0.3333) = (10+60*0.3333, 10+30*0.3333) = (30, 20) $$

• End: $$\displaystyle P(0.5) = (10+60*0.5, 10+30*0.5) = (40, 25) $$

\boxed{\text{Clipped segment: } (30,20) \text{ to } (40,25)}

Interpolation Techniques

$$y = y_0 + \frac{(x - x_0)}{(x_1 - x_0)} (y_1 - y_0)$$

| Example: Color gradient between red (0) and blue (1). At $$\displaystyle x=0.5 $$, $$\displaystyle y = 0.5*blue + 0.5*red $$ = purple.<br>Use: Simple animation, texture mapping, UI sliders. | | Nonlinear Interpolation | Uses curves (polynomials, splines) for smooth transitions, avoiding linear "stiffness". | Methods:<br>- Cubic Spline: Piecewise cubic polynomials ensuring smooth first/second derivatives.<br>- Bezier Curves: Defined by control points (e.g., $$\displaystyle B(t) = (1-t)^3 P_0 + ... $$).<br>Use: Smooth camera paths, character animation, surface modeling. |

Shading Algorithms

Purpose: To compute the color and brightness of surfaces in a 3D scene based on lighting, material properties, and viewing angle, creating the illusion of solidity and depth.

Kinematic Modeling: Worm on Rotating Wheel

Problem (Nov 2023): Wheel in y-z plane, center at origin. Spoke along +y at $$\displaystyle t=0 $$. Worm crawls outward at 1 unit/s along spoke. Wheel rotates at 1 rad/s. Find $\vec{r}(t)$.

Solution:

• At time $t$, worm's radial distance from center: $$\displaystyle r = \text{speed} \times t = 1 \cdot t = t $$.

• Spoke's angular position from +y-axis: $$\displaystyle \theta = \text{angular speed} \times t = 1 \cdot t = t $$ (radians).

• Wheel in y-z plane (x=0 always). Coordinates relative to y-axis:

$$y = r \cos\theta = t \cos t$$

$$z = r \sin\theta = t \sin t$$

\boxed{\vec{r}(t) = (0,; t \cos t,; t \sin t)}

Simulation Mathematics: Flight Dynamics

Purpose: Model aircraft motion (6-DOF: 6 Degrees of Freedom) for realistic simulation.

Core Techniques:

1. Newton-Euler Equations: Solve for linear and angular motion.

• Linear: $$\displaystyle F = m \cdot a $$ (Forces: lift, drag, thrust, weight).

• Angular: $$\displaystyle \tau = I \cdot \alpha $$ (Torques from control surfaces).

2. Aerodynamic Coefficients: Use pre-computed lookup tables or polynomial fits for $$\displaystyle C_L $$ (lift), $$\displaystyle C_D $$ (drag), $$\displaystyle C_m $$ (moment) vs. angle of attack, Mach number.

3. Coordinate System Transformations: Convert between body-fixed, wind, and inertial frames using rotation matrices/Euler angles.

4. Numerical Integration: Use methods like Runge-Kutta (RK4) to integrate equations of motion over time steps $\Delta t$.

5. Interaction and Tracking

Models of Interaction in Virtual Environments

1. Selection: Identifying and choosing an object.

   • Techniques: Ray casting (point & click), hand gestures, gaze-based selection.

2. Manipulation: Changing an object's properties (position, rotation, scale).

   • Techniques: Direct manipulation (grab & move), widget-based (transform gizmo), indirect (input panels).

3. Navigation: Moving through the virtual space.

   • Techniques: Physical walking (room-scale), teleportation, continuous locomotion (thumbstick), automated path following.

Collision Detection in Generic VR Systems

Purpose: Prevent objects from passing through each other, provide physical feedback, and enable interaction (e.g., picking up).

Generic Approach (Example with Bounding Volumes):

1. Broad Phase: Quickly find potential colliding pairs using simple bounding volumes.

• Example: Use Axis-Aligned Bounding Boxes (AABB). For two objects with AABB1 $$\displaystyle [x_{min}, x_{max}] $$ and AABB2 $$\displaystyle [x'_{min}, x'_{max}] $$, they overlap if:

$$x_{min} \leq x'_{max} \;\text{and}\; x'_{min} \leq x_{max}$$

 (Similarly for y and z). If all three axes overlap, objects are *potentially* colliding.

2. Narrow Phase: For each potential pair, perform precise test using actual geometry (e.g., triangle-triangle intersection, distance checks).

3. Response: Upon confirmed collision, respond (stop movement, play sound, trigger event).

Example: VR hand controller (AABB) and virtual cup (AABB). Broad phase checks AABB overlap. If overlapping, narrow phase checks if hand mesh intersects cup mesh. If yes, cup is "grabbed".

Tracking Methods: Marker-less Tracking in AR

Principle: Determine the device's (e.g., smartphone/glasses) 6-DOF pose (position + orientation) in the real world without artificial markers (like QR codes). Relies on natural features or sensor fusion.

Common Techniques & Example:

• Feature-Based Tracking (Visual-Inertial SLAM):

1. Camera captures frames, detects natural features (corners, edges) using algorithms like ORB, FAST.

2. Track these features across frames to estimate motion (egomotion).

3. Fuse with IMU data (accelerometer, gyroscope) for high-frequency, drift-resistant pose estimation.

4. Build/Update a sparse 3D map of feature points (point cloud).

• Example: ARKit (iOS) / ARCore (Android). When you point your phone at a table, the system detects distinctive points on the table surface, tracks them as you move, and simultaneously uses IMU to estimate pose. This allows placing a virtual chair that stays fixed on the table as you walk around.

[!TIP] Common Pitfall: Confusing marker-based (QR code) with marker-less tracking. Marker-less is harder but more seamless for end-users.