Augmented and Virtual Reality (AL-703 (B)) - Unit 4 Short Notes

UNIT 4: AUGMENTED AND VIRTUAL REALITY – CORE CONCEPTS, TECHNOLOGIES, AND APPLICATIONS

1.0 FUNDAMENTALS OF VR AND AR

1.1 Virtual Reality (VR): Definition and Core Features

Virtual Reality (VR) is a simulated, interactive, computer-generated environment that immerses a user, making them feel present in a synthetic world.

[!TIP] Exam Focus: All four features (IIPI) are frequently asked. Define each concisely and link to examples (e.g., HMD for immersion, hand tracking for interactivity).

1.2 Augmented Reality (AR) vs. Virtual Reality (VR)

Reality-Virtuality Continuum:

[Real Environment] <--- [Augmented Reality (AR)] --- [Mixed Reality (MR)] --- [Augmented Virtuality (AV)] --- [Virtual Environment (VR)] --- [Virtual Reality]

[!TIP] Common Pitfall: Do not confuse Mixed Reality (MR) with AR. MR involves real and virtual objects interacting in real-time (occlusion, physics).

1.3 Mixed Reality (MR)

Mixed Reality (MR) is the merging of real and virtual worlds where physical and digital objects co-exist and interact in real-time. It sits on the continuum between AR and AV.

• Key Trait: Virtual objects are anchored to and respond to the real environment (e.g., a virtual ball bouncing behind a real table).

• Example: Microsoft HoloLens applications where holograms are placed on physical surfaces.

2.0 MATHEMATICAL AND GRAPHICAL FOUNDATIONS

2.1 Parametric Line Clipping (Liang-Barsky Algorithm)

Purpose: Efficiently clip a line segment against a rectangular window using parametric equations.

Parametric Form of Line: For endpoints $$\displaystyle P_0(x_0, y_0) $$ and $$\displaystyle P_1(x_1, y_1) $$:

$$x = x_0 + t (x_1 - x_0) = x_0 + t \Delta x$$

$$y = y_0 + t (y_1 - y_0) = y_0 + t \Delta y$$

where $t \in [0, 1]$ for the segment.

Algorithm Steps:

1. Initialize $$\displaystyle t_{E} = 0 $$, $$\displaystyle t_{L} = 1 $$ (Enter and Leave parameters).

2. For each window edge (Left, Right, Bottom, Top), compute $t$ where the line intersects the edge boundary.

   • For Left ($$\displaystyle x = x_{wmin} $$): $$\displaystyle t = \frac{x_{wmin} - x_0}{\Delta x} $$. If $$\displaystyle \Delta x < 0 $$, update $$\displaystyle t_E = \max(t_E, t) $$; if $$\displaystyle \Delta x > 0 $$, update $$\displaystyle t_L = \min(t_L, t) $$.

   • Repeat for Right, Bottom, Top with appropriate conditions on $\Delta y$.

3. If $$\displaystyle t_E > t_L $$ at any point, line is entirely outside (reject).

4. If $$\displaystyle t_E \le t_L $$, clipped line endpoints are at $$\displaystyle t = t_E $$ and $$\displaystyle t = t_L $$.

Example Problem: Clip $A(10,10)$, $B(70,40)$ against window $(20,20)$ to $(40,50)$.

• $$\displaystyle x_0=10, y_0=10, x_1=70, y_1=40 \rightarrow \Delta x=60, \Delta y=30 $$.

• Left ($$\displaystyle x=20 $$): $$\displaystyle t = (20-10)/60 = 0.1667 $$. $$\displaystyle \Delta x>0 \Rightarrow t_L = \min(1, 0.1667) = 0.1667 $$.

• Right ($$\displaystyle x=40 $$): $$\displaystyle t = (40-10)/60 = 0.5 $$. $$\displaystyle \Delta x>0 \Rightarrow t_L = \min(0.1667, 0.5) = 0.1667 $$ (no change).

• Bottom ($$\displaystyle y=20 $$): $$\displaystyle t = (20-10)/30 = 0.3333 $$. $$\displaystyle \Delta y>0 \Rightarrow t_L = \min(0.1667, 0.3333) = 0.1667 $$ (no change).

• Top ($$\displaystyle y=50 $$): $$\displaystyle t = (50-10)/30 = 1.333 $$. $$\displaystyle \Delta y>0 \Rightarrow t_L = \min(0.1667, 1.333) = 0.1667 $$ (no change).

• Result: $$\displaystyle t_E=0, t_L=0.1667 $$. Since $$\displaystyle t_E \le t_L $$, line is partially inside.

  • Clipped Start: $$\displaystyle t_E=0 \rightarrow (10,10) $$ (but this is outside window left/bottom? Wait, our $$\displaystyle t_L $$ became 0.1667 from Left edge, meaning the line leaves the window immediately after entering? Let's re-evaluate carefully).

  • Correction: For Left edge ($$\displaystyle x=20 $$), since $$\displaystyle \Delta x>0 $$, the line goes from left to right. Intersection at $$\displaystyle t=0.1667 $$ is where it enters the window from the left. So this should update $$\displaystyle t_E $$, not $$\displaystyle t_L $$. Let's re-apply rules correctly.

Correct Step-by-Step Application:

1. $$\displaystyle t_E=0, t_L=1 $$.

2. Left ($$\displaystyle x=20 $$): $$\displaystyle t = (20-10)/60 = 0.1667 $$. $$\displaystyle \Delta x>0 $$ (moving right) → entering from left → $$\displaystyle t_E = \max(0, 0.1667) = 0.1667 $$.

3. Right ($$\displaystyle x=40 $$): $$\displaystyle t = (40-10)/60 = 0.5 $$. $$\displaystyle \Delta x>0 $$ (moving right) → exiting right → $$\displaystyle t_L = \min(1, 0.5) = 0.5 $$.

4. Bottom ($$\displaystyle y=20 $$): $$\displaystyle t = (20-10)/30 = 0.3333 $$. $$\displaystyle \Delta y>0 $$ (moving up) → entering from bottom → $$\displaystyle t_E = \max(0.1667, 0.3333) = 0.3333 $$.

5. Top ($$\displaystyle y=50 $$): $$\displaystyle t = (50-10)/30 = 1.333 $$. $$\displaystyle \Delta y>0 $$ (moving up) → exiting top → $$\displaystyle t_L = \min(0.5, 1.333) = 0.5 $$.

6. Final: $$\displaystyle t_E=0.3333 $$, $$\displaystyle t_L=0.5 $$. Since $$\displaystyle t_E \le t_L $$, segment is partially inside.

   • Clipped Start ($$\displaystyle t_E $$): $$\displaystyle x = 10 + 0.3333*60 = 30 $$, $$\displaystyle y = 10 + 0.3333*30 = 20 $$ → (30, 20).

   • Clipped End ($$\displaystyle t_L $$): $$\displaystyle x = 10 + 0.5*60 = 40 $$, $$\displaystyle y = 10 + 0.5*30 = 25 $$ → (40, 25).

   • Clipped Line Segment: from (30,20) to (40,25).

[!TIP] Exam Pattern: This exact problem (A(10,10), B(70,40) vs window (20,20)-(40,50)) has appeared. Box the final clipped coordinates.

$$\boxed{\text{Clipped Segment: } (30,20) \text{ to } (40,25)}$$

2.2 Vector Functions for 3D Motion

Position Vector: $$\displaystyle \vec{r}(t) = \langle x(t), y(t), z(t) \rangle $$ describes an object's location over time.

Example: Worm on Rotating Wheel

• Wheel lies in y-z plane, center at origin.

• Worm crawls outward along a spoke at radial velocity $$\displaystyle v_r = 1 $$ unit/sec.

• Wheel rotates at angular velocity $$\displaystyle \omega = 1 $$ radian/sec.

• At $$\displaystyle t=0 $$, spoke along +y-axis, worm at origin.

Solution:

1. Radial distance at time $t$: $$\displaystyle r(t) = v_r \cdot t = t $$.

2. Angular position: $$\displaystyle \theta(t) = \omega \cdot t = t $$.

3. Since wheel is in y-z plane:

   • $$\displaystyle y = r(t) \cos(\theta(t)) = t \cos(t) $$

   • $$\displaystyle z = r(t) \sin(\theta(t)) = t \sin(t) $$

   • $$\displaystyle x = 0 $$ (no motion in x-direction).

4. Vector Function:

$$\vec{r}(t) = \langle 0,\; t \cos t,\; t \sin t \rangle$$

[!TIP] Key Insight: Decompose motion into radial (along spoke) and angular (rotation) components. Remember coordinate assignment based on plane (y-z here).

2.3 Interpolation Techniques

Purpose: Estimate intermediate values between known data points (positions, colors, rotations).

[!TIP] Critical for VR: Use Slerp (Spherical Linear Interpolation) for rotations (quaternions) to avoid gimbal lock and ensure constant velocity. Linear interpolation of angles causes artifacts.

2.4 Shading Algorithms

Purpose: Determine the color of each pixel on a polygon surface based on lighting, material, and viewer position.

[!TIP] Comparison Rule: Quality: Flat < Gouraud < Phong. Cost: Flat < Gouraud < Phong. Phong is preferred for shiny objects; Gouraud is a good compromise for real-time VR.

3.0 VR MODELING WITH VRML (VIRTUAL REALITY MODELING LANGUAGE)

3.1 Overview of VRML

• Purpose: File format for describing interactive 3D vector graphics, especially for the web.

• File Extension: .wrl

• Nature: Scene-graph based, text/ASCII format. Objects are nodes in a hierarchical tree.

• Role: Enabled early web-based VR before WebGL/WebXR.

3.2 Key VRML Nodes and Their Functions

3.3 VRML Examples

Example 1: Simple Colored Box with Collision

# Red Box with collision proxy (same size)

Transform {

  translation 0 0 5

  children [

    Collision {

      collide TRUE

      proxy Shape {

        geometry Box { size 2 2 2 }

      }

      children [

        Shape {

          geometry Box { size 2 2 2 }

          appearance Appearance {

            material Material { diffuseColor 1 0 0 }

          }

        }

      ]

    }

  ]

}

Example 2: Anchor Linking to Another World

Anchor {

  url "next_scene.wrl"

  description "Go to next room"

  children [

    Shape {

      geometry Text { string "Click Here" }

      appearance Appearance { material Material { } }

    }

  ]

}

[!TIP] Exam Pattern: Be ready to write short code snippets for each of the four nodes (Anchor, Collision, Group, Shape). Understand the children field hierarchy.

4.0 INTERACTION AND PHYSICS IN VIRTUAL ENVIRONMENTS

4.1 Models of Interaction

1. Navigation: Moving through the environment.

   • Examples: Walking (room-scale), teleportation, flying (flight sim).

2. Selection/Manipulation: Picking up, moving, or modifying objects.

   • Examples: Grabbing a virtual tool, pressing a button, resizing an object.

3. System Control: Invoking system-level commands (menu, exit, settings).

   • Examples: Opening a settings panel via gesture, voice command "save".

[!TIP] Link to Hardware: Navigation often uses trackers/controllers; manipulation uses haptics; system control uses UI gestures or buttons.

4.2 Collision Detection in VR

Generic VR System Architecture for Collision Detection:

[User Input/Tracker Data] --> [Update Object Poses] --> [Broad Phase] --> [Narrow Phase] --> [Collision Response]

1. Broad Phase: Quickly find potential colliding pairs (e.g., using bounding volume hierarchies like AABB trees).

2. Narrow Phase: Precise test for each candidate pair (e.g., triangle-triangle intersection).

3. Collision Response: Apply physics (forces, bounce, sound) or logical consequence (object selected, game over).

Example: User's virtual hand (tracked by controller) intersects with a virtual cube.

• Broad Phase: Hand's AABB overlaps Cube's AABB.

• Narrow Phase: Detailed mesh check confirms intersection.

• Response: Cube changes color, plays "click" sound, becomes attached to hand.

4.3 Flight Dynamics Simulation

Goal: Realistically simulate aircraft motion under forces.

• Forces Modeled:

  • Lift: $$\displaystyle L = \frac{1}{2} \rho v^2 S C_L(\alpha) $$ (depends on air density $\rho$, velocity $v$, wing area $S$, angle of attack $\alpha$).

  • Drag: $$\displaystyle D = \frac{1}{2} \rho v^2 S C_D(\alpha) $$.

  • Thrust: From engine model.

  • Gravity: $$\displaystyle W = mg $$.

• Controls: Pilot inputs (stick, throttle) affect:

  • Aerodynamic surfaces: Elevator (pitch), Ailerons (roll), Rudder (yaw).

  • Thrust.

• Environmental Effects:

  • Wind/Turbulence: Adds stochastic forces, changes local air velocity vector.

  • Atmosphere: Density variation with altitude affects lift/drag.

• Integration: Solve 6-DOF (Degrees of Freedom) equations of motion (Newton-Euler) over time.

[!TIP] Simplification for VR: Often use simplified "arcade" physics (e.g., direct mapping of stick to angular rate) instead of full CFD for real-time performance.

5.0 HARDWARE AND SENSORY INTEGRATION

5.1 Sensor Hardware for VR

5.2 Acoustic Hardware for VR

• Spatial Audio Systems: Simulate 3D sound sources using Head-Related Transfer Functions (HRTFs).

• Hardware: High-fidelity headphones (most common), speaker arrays (CAVE-like systems).

• Contribution to Immersion: Provides critical cues for:

  • Direction & Distance: "Hear" an object behind you.

  • Environmental Context: Echoes, reverb for room size.

  • Presence: Auditory consistency with visual scene.

5.3 Display Technologies for Augmented Reality

Diagram Concept:

OPTICAL SEE-THROUGH:

[Real World] --> [Beam Splitter / Half-Silvered Mirror] <--> [Virtual Display]

User's eye sees combination directly.

VIDEO SEE-THROUGH:

[Real World] --> [Camera] --> [Compositor] + [Virtual Graphics] --> [Opaque Display] --> [User's Eye]

[!TIP] Key Difference: Optical = direct view of real world. Video = camera-mediated view.

5.4 Role of Human Senses in AR

• Visual (Eyes): Primary channel. Requires precise registration (overlay alignment), sufficient field of view (FOV) for context, high resolution for readability.

• Auditory (Ears): Spatial audio cues guide attention (e.g., "instruction coming from left"), confirm actions (e.g., "snap" sound when part is placed).

• Proprioception & Touch (Haptics): Sense of hand/body position. Haptic feedback (vibration, force) confirms virtual interactions (e.g., feeling a button click, tool resistance).

Example: AR Maintenance Application

1. Visual: Technician sees a virtual arrow highlighting a bolt, text instructions overlaid on the machine.

2. Auditory: A voice says "Remove bolt number 4" and a beep sounds from its location.

3. Haptic: When the virtual wrench is properly aligned, the controller vibrates.

6.0 TRACKING AND REGISTRATION IN AR

6.1 Marker-Based Tracking (Brief Context)

• Uses fiducial markers (e.g., QR codes, ARToolKit patterns) placed in the environment.

• Camera detects marker corners → computes pose (position & orientation) of marker relative to camera.

• Pros: Simple, robust, accurate.

• Cons: Requires markers, limited to marker FOV, intrusive.

6.2 Marker-Less Tracking for Augmented Reality

Goal: Estimate device pose without pre-placed markers.

Example Application: Indoor Navigation without Markers

1. User opens AR navigation app in a mall.

2. SLAM system starts: IMU provides initial motion; camera detects ceiling lights, floor patterns as features.

3. System builds a local map of the hallway and estimates user's pose within it.

4. Virtual arrows are rendered on the floor, anchored to the mapped space, guiding the user.

5. As user walks, continuous tracking updates arrow positions.

[!TIP] Exam Focus: SLAM is the most important marker-less technique for modern AR. Understand it as "map + localize simultaneously."

7.0 VR DEVELOPMENT TOOLKITS AND FRAMEWORKS

7.1 VR Toolkits and SDKs

Example Workflow: Importing a VRML Model

1. Convert: Use a 3D tool (Blender, MeshLab) to import .wrl and export as .fbx or .glb (glTF binary).

2. Import into Engine: Drag converted file into Unity/Unreal project.

3. Add Components: Attach XR Origin (Unity), collision components, interaction scripts.

4. Build & Run: Deploy to target VR headset.

[!TIP] Reality Check: VRML is historical. Modern pipelines use glTF 2.0 as the "JPEG of 3D". Know that conversion is necessary.

8.0 INTEGRATED APPLICATIONS AND SYSTEM DESIGN

8.1 Case Study: Flight Simulator VR System

• Graphics:

  • Shading: Phong shading for cockpit instruments, Gouraud for terrain to balance cost/quality.

  • Interpolation: Cubic spline for smooth camera path (chase plane), spherical linear (Slerp) for aircraft control surface rotations.

• Interaction:

  • Models: Navigation (yoke/joystick), Selection (flip switches), System Control (menu via button).

  • Collision Detection: Broad phase (grid for terrain chunks), narrow phase (precise mesh for runway contact).

  • Flight Dynamics: 6-DOF physics model with lift/drag/thrust/gravity, wind turbulence model, control surface effects.

• Hardware:

  • Sensors: High-precision 6-DOF tracker for head, force-feedback yoke.

  • Display: High-FOV HMD or multi-screen CAVE.

  • Acoustic: Spatial audio for engine roar, wind, ATC communications.

8.2 Design Considerations for AR/VR Systems

[!TIP] Golden Triangle: Latency, FOV, Resolution are the primary competing factors. Optimizing one often compromises others. Comfort is the ultimate constraint.