Augmented and Virtual Reality (AL-703 (B)) - Unit 3 Short Notes

UNIT 3: AUGMENTED AND VIRTUAL REALITY

I. FOUNDATIONS OF VR/AR/MR

Virtual Reality (VR)

• Definition: A computer-generated, immersive simulation of a 3D environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment (e.g., HMD, gloves).

• Core Features:

  • Immersion: Sensory isolation from the real world, creating a feeling of "being there" (presence).

  • Interaction: Real-time user input influences the virtual world (navigation, manipulation).

  • Imagination: Ability to create and experience non-realistic, hypothetical worlds.

  • 3D Computer Graphics: Real-time rendering of a stereoscopic, perspective-correct environment.

[!TIP] Exam often asks for features. List Immersion, Interaction, Imagination, and 3D Graphics as the four pillars.

Augmented Reality (AR) vs. Virtual Reality (VR)

Mixed Reality (MR)

• Concept: A continuum (Reality-Virtuality Continuum) where real and virtual objects are merged and interact in real-time. It is a superset of AR, where virtual objects are anchored and occluded by real objects convincingly.

• Positioning: Sits between AR (simple overlay) and VR (full replacement). In MR, virtual objects understand and respond to the physical environment (e.g., a virtual ball bouncing behind a real table).

II. VR MODELING LANGUAGES AND TOOLS

Virtual Reality Modeling Language (VRML)

• Purpose: A standard file format for representing 3D interactive vector graphics, designed for the World Wide Web. It describes 3D scenes (worlds) with objects, lights, cameras, and sensors.

• Structure: Text-based, hierarchical scene graph. A .wrl file contains a hierarchy of nodes.

• Key Node Types:

  • Anchor node: Creates a hyperlink to another URL or VRML file. Clicking the object within the Anchor's children field triggers the link.

    
    Anchor {
    
      url "https://www.rgpv.ac.in"
    
      children [ Shape { ... } ]  # Clickable object
    
    }
    
    

  • Collision node: Detects when the user's avatar collides with its children geometry. Can be set to trigger an event or be "proxy" (simplified geometry for efficiency).

  • Group node: A base node for grouping other nodes (children field). Used for hierarchical transformations (via Transform node) and collective properties.

  • Shape node: The fundamental node for visible geometry. Combines a geometry (e.g., Box, Sphere) and an appearance (Material, Texture).

    
    Shape {
    
      geometry Box { size 2 2 2 }
    
      appearance Material { diffuseColor 1 0 0 } # Red box
    
    }
    
    

VR Toolkits and Development Frameworks

• Overview: Software libraries/SDKs that provide pre-built functions for common VR tasks (rendering, input, tracking) to accelerate development.

• Common Examples:

  • Unity + XR Interaction Toolkit: Industry standard, C# based, vast asset store.

  • Unreal Engine: High-fidelity graphics, Blueprint visual scripting.

  • OpenVR / SteamVR: API for HTC Vive, Valve Index.

  • OpenXR: An open, royalty-free standard for VR/AR applications, aims for cross-platform compatibility.

  • A-Frame: Web-based VR framework using HTML-like syntax.

III. HARDWARE TECHNOLOGIES FOR VR/AR

Sensor Hardware for VR Environments

• Head Trackers: Measure head orientation (yaw, pitch, roll) and often position. Technologies: Inertial Measurement Units (IMUs), Optical (external cameras), Magnetic.

• Motion Capture Systems: Full-body tracking using markers (optical) or inertial sensors (e.g., Perception Neuron). Used for avatar animation.

• Data Gloves: Gloves with flex sensors on fingers and a tracker for hand position/orientation. Enable natural hand interaction and gesture recognition.

• Haptic Feedback Devices: Provide tactile sensation. Force-feedback (e.g., robotic arms, exoskeletons) resists user motion. Tactile feedback (e.g., vibration motors in controllers) simulates texture/impact.

Display Technologies for Augmented Reality

• Optical See-Through: User looks through transparent optical elements (e.g., beam splitters, holographic waveguides) that reflect virtual images into the eye. Example: Microsoft HoloLens, Magic Leap.

  • Architecture: Real world → Transparent combiner → Eye. Virtual image → Optical system → Combiner → Eye.

• Video See-Through: User views the real world through cameras mounted on the HMD. The video feed is combined with rendered virtual graphics and displayed on an opaque screen. Example: Smartphone AR (ARKit, ARCore), early AR systems.

  • Architecture: Real world → Cameras → Video stream → Compositing with virtual graphics → Opaque display → Eye.

DiagramCANVAS: Draw two side-by-side diagrams. Left: "Optical See-Through" showing eye, transparent combiner, real world path (straight through), virtual image path (reflected off combiner). Right: "Video See-Through" showing cameras capturing real world, computer mixing with graphics, display screen to eye.

Acoustic Hardware

• Spatial Audio Systems: Simulate 3D sound sources in a virtual environment. Uses Head-Related Transfer Functions (HRTFs) to filter audio based on the source's location relative to the listener's head.

• Hardware: Headphones/earphones are essential for individual HRTF application. Some systems use external speaker arrays (wave field synthesis) for room-scale audio without headphones.

IV. GRAPHICS, RENDERING & GEOMETRIC TECHNIQUES

Line Clipping: Liang-Barsky Algorithm

• Concept: A parametric line-clipping algorithm that uses the parametric equation of the line and inequalities defining the clipping window to find the visible segment. More efficient than Cohen-Sutherland for many cases.

• Parametric Equation: $$\displaystyle P(t) = P_0 + t(P_1 - P_0) $$, where $0 \leq t \leq 1$.

  • $$\displaystyle P_0 = (x_0, y_0) $$, $$\displaystyle P_1 = (x_1, y_1) $$ are line endpoints.

• Algorithm Steps:

  1. For each window edge (left: $$\displaystyle x = x_{min} $$, right: $$\displaystyle x = x_{max} $$, bottom: $$\displaystyle y = y_{min} $$, top: $$\displaystyle y = y_{max} $$), compute $t$ values where the line intersects the edge.

  2. For a vertical edge $$\displaystyle x = x_e $$, $$\displaystyle t = (x_e - x_0) / (x_1 - x_0) $$.

  3. For a horizontal edge $$\displaystyle y = y_e $$, $$\displaystyle t = (y_e - y_0) / (y_1 - y_0) $$.

  4. Initialize $$\displaystyle t_{enter} = 0 $$, $$\displaystyle t_{exit} = 1 $$.

  5. For each edge, update:

     • If $$\displaystyle dx < 0 $$ (line goes left), $$\displaystyle t_{enter} = \max(t_{enter}, t) $$.

     • If $$\displaystyle dx > 0 $$ (line goes right), $$\displaystyle t_{exit} = \min(t_{exit}, t) $$.

     • Similarly for $dy$ with top/bottom edges.

  6. If $$\displaystyle t_{enter} > t_{exit} $$, line is entirely outside. Reject.

  7. Otherwise, clip line to $$\displaystyle P(t_{enter}) $$ and $$\displaystyle P(t_{exit}) $$.

• Numerical Example (From Past Paper):

  • Line: $A(10,10)$, $B(70,40)$. Window: $(20,20)$ to $(40,50)$.

  • $$\displaystyle dx = 60 $$, $$\displaystyle dy = 30 $$.

  • Left edge ($$\displaystyle x=20 $$): $$\displaystyle t = (20-10)/60 = 0.1667 $$. $$\displaystyle dx>0 $$ → $$\displaystyle t_{exit} = \min(1, 0.1667) = 0.1667 $$.

  • Right edge ($$\displaystyle x=40 $$): $$\displaystyle t = (40-10)/60 = 0.5 $$. $$\displaystyle dx>0 $$ → $$\displaystyle t_{exit} = \min(0.1667, 0.5) = 0.1667 $$ (no change).

  • Bottom edge ($$\displaystyle y=20 $$): $$\displaystyle t = (20-10)/30 = 0.3333 $$. $$\displaystyle dy>0 $$ → $$\displaystyle t_{exit} = \min(0.1667, 0.3333) = 0.1667 $$.

  • Top edge ($$\displaystyle y=50 $$): $$\displaystyle t = (50-10)/30 = 1.333 $$. $$\displaystyle dy>0 $$ → $$\displaystyle t_{exit} $$ unchanged.

  • Conclusion: $$\displaystyle t_{enter}=0 $$, $$\displaystyle t_{exit}=0.1667 $$. Since $$\displaystyle t_{enter} < t_{exit} $$, line is clipped.

  • Clipped Points:

    • $$\displaystyle P_{new1} = P(0) = A(10,10) $$ (But $$\displaystyle t_{enter}=0 $$ means start point is inside? Wait, check logic. Actually, $$\displaystyle t_{exit}=0.1667 < t_{enter}=0 $$? No, $$\displaystyle t_{enter}=0 $$, $$\displaystyle t_{exit}=0.1667 $$. So segment from $$\displaystyle t=0 $$ to $$\displaystyle t=0.1667 $$ is inside? That can't be right because A(10,10) is outside the window (x=10<20).**

  • Correction: Must check all edges for t_enter and t_exit correctly.

    • Left ($$\displaystyle x=20 $$): $$\displaystyle t=0.1667 $$, $$\displaystyle dx>0 $$ → affects t_exit? Actually, standard Liang-Barsky:

      • For $$\displaystyle x_{min} $$: if $$\displaystyle dx < 0 $$, $$\displaystyle t_{enter} = \max(t_{enter}, t) $$; if $$\displaystyle dx > 0 $$, $$\displaystyle t_{exit} = \min(t_{exit}, t) $$.

      • Here $$\displaystyle dx=60>0 $$, so $$\displaystyle t_{exit} = \min(1, 0.1667) = 0.1667 $$.

    • Right ($$\displaystyle x=40 $$): $$\displaystyle t=0.5 $$, $$\displaystyle dx>0 $$ → $$\displaystyle t_{exit} = \min(0.1667, 0.5) = 0.1667 $$.

    • Bottom ($$\displaystyle y=20 $$): $$\displaystyle t=0.3333 $$, $$\displaystyle dy=30>0 $$ → $$\displaystyle t_{exit} = \min(0.1667, 0.3333) = 0.1667 $$.

    • Top ($$\displaystyle y=50 $$): $$\displaystyle t=1.333 $$, $$\displaystyle dy>0 $$ → $$\displaystyle t_{exit} $$ unchanged.

    • Now for t_enter: We need edges where parameter change would indicate entering.

      • Left ($$\displaystyle x=20 $$): $$\displaystyle dx>0 $$ does not affect t_enter.

      • Right ($$\displaystyle x=40 $$): $$\displaystyle dx>0 $$ does not affect t_enter.

      • Bottom ($$\displaystyle y=20 $$): $$\displaystyle dy>0 $$ does not affect t_enter.

      • Top ($$\displaystyle y=50 $$): $$\displaystyle dy>0 $$ does not affect t_enter.

    • So t_enter remains 0. But point at t=0 is (10,10) which is outside. This indicates we missed an edge that would set t_enter.

    • Proper check: For each edge, compute t. Then:

      • If $$\displaystyle dx < 0 $$ (moving left), the left edge is an entering boundary? Actually, the logic is:

        • For $$\displaystyle x_{min} $$: if $$\displaystyle dx < 0 $$, then as t increases, x decreases, so crossing $$\displaystyle x_{min} $$ from right to left is exiting? Let's recall standard Liang-Barsky pseudo-code.

    • Simpler Approach: Compute all four t values: $$\displaystyle t_1=(x_{min}-x_0)/dx $$, $$\displaystyle t_2=(x_{max}-x_0)/dx $$, $$\displaystyle t_3=(y_{min}-y_0)/dy $$, $$\displaystyle t_4=(y_{max}-y_0)/dy $$.

    • Then $$\displaystyle t_E = \max(0, \min(t_1,t_2,t_3,t_4)) $$? No.

    • Correct Logic:

      Let $$\displaystyle tL = (x_{min} - x_0)/dx $$, $$\displaystyle tR = (x_{max} - x_0)/dx $$, $$\displaystyle tB = (y_{min} - y_0)/dy $$, $$\displaystyle tT = (y_{max} - y_0)/dy $$.

      If $$\displaystyle dx < 0 $$, swap $tL$ and $tR$.

      If $$\displaystyle dy < 0 $$, swap $tB$ and $tT$.

      Then $$\displaystyle t_{enter} = \max(0, tL, tB) $$

      $$\displaystyle t_{exit} = \min(1, tR, tT) $$

      If $$\displaystyle t_{enter} > t_{exit} $$, reject.

    • Apply:

      $$\displaystyle dx=60>0 $$, so $$\displaystyle tL=0.1667 $$, $$\displaystyle tR=0.5 $$ (no swap).

      $$\displaystyle dy=30>0 $$, so $$\displaystyle tB=0.3333 $$, $$\displaystyle tT=1.333 $$ (no swap).

      $$\displaystyle t_{enter} = \max(0, 0.1667, 0.3333) = 0.3333 $$

      $$\displaystyle t_{exit} = \min(1, 0.5, 1.333) = 0.5 $$

      Since $$\displaystyle 0.3333 < 0.5 $$, visible segment exists.

    • Clipped Points:

      • $$\displaystyle P_{enter} = P(0.3333) = (10+0.3333*60, 10+0.3333*30) = (30, 20) $$.

      • $$\displaystyle P_{exit} = P(0.5) = (10+0.5*60, 10+0.5*30) = (40, 25) $$.

    • Final Answer: Clipped line segment from (30, 20) to (40, 25).

Interpolation Methods

• Linear Interpolation (Lerp):

  • Concept: Straight-line interpolation between two known values. Used for smooth transitions, animation, color blending.

  • Formula: Given points $$\displaystyle (x_0, y_0) $$ and $$\displaystyle (x_1, y_1) $$, value at $x$ is:

$$y = y_0 + \frac{(x - x_0)}{(x_1 - x_0)} (y_1 - y_0)$$

*   **Example:** Fading an object's opacity from 0% at frame 10 to 100% at frame 20. At frame 15 ($$\displaystyle x=15 $$), opacity $$\displaystyle = 0 + \frac{5}{10} \times 100 = 50\% $$.

• Nonlinear Interpolation:

  • Concept: Interpolation where the rate of change is not constant. Used for more natural motion (easing), smooth curves.

  • Common Types: Polynomial (Quadratic, Cubic), Spline (B-spline, Bezier), Trigonometric.

  • Example: Cubic Bezier curve defined by 4 points ($$\displaystyle P_0, P_1, P_2, P_3 $$). Point at parameter $t$:

$$B(t) = (1-t)^3 P_0 + 3(1-t)^2 t P_1 + 3(1-t) t^2 P_2 + t^3 P_3$$

    Used in animation paths and vector graphics (SVG).

Shading Algorithms

Collision Detection

• Purpose: Detect when virtual objects intersect or come into contact to trigger responses (physics, sound, game logic).

• In Generic VR Systems:

  1. Broad Phase: Quickly eliminate pairs of objects that are definitely not colliding. Uses bounding volumes (AABB - Axis-Aligned Bounding Box, Sphere). Spatial partitioning structures (Octree, BSP tree) can optimize.

  2. Narrow Phase: Precise test for remaining candidate pairs. Uses actual geometry (triangle mesh) or more complex bounding volumes (OBB - Oriented Bounding Box).

• Example-Based Explanation:

  • Scenario: VR user (represented by a capsule collider) picks up a virtual cube.

  • Broad Phase: Check if user's AABB intersects cube's AABB. If no, skip.

  • Narrow Phase: If AABBs intersect, perform triangle-triangle intersection test between user's hand mesh and cube mesh, or use simplified convex hulls.

  • Response: On detection, physics engine applies constraints (cube attaches to hand), plays a "pickup" sound, and updates game state.

V. INTERACTION, TRACKING & SIMULATION

Models of Interaction in Virtual Environments

• Classification:

  1. Selection & Manipulation: Choosing and moving objects (e.g., ray casting, direct hand grab).

  2. Navigation & Travel: Moving through the virtual world (e.g., walking, teleportation, flying).

  3. System Control: Executing commands (e.g., menus, voice commands, gestures).

  4. Communication: Social interaction with other users (avatars, chat, gestures).

• Interaction Paradigms:

  • Go-Ahead/Do-It: Separate actions for navigation (move) and manipulation (act).

  • Virtual Tools: Treat hands as tools (e.g., virtual hand holding a hammer).

  • World-in-Miniature (WIM): A small-scale model of the world in the user's hand for navigation and manipulation.

  • Multimodal: Combining input methods (voice + gesture).

Tracking Technologies: Marker-less Tracking for AR

• Concept: Determining the position and orientation of the AR device (phone/HMD) in the real world without using artificial visual markers (like QR codes). Relies on natural features.

• Methodologies & Examples:

  • Feature-Based (Visual-Inertial Odometry - VIO): Combines camera feed (detecting corners, edges) with IMU data. Example: Apple ARKit, Google ARCore. Phone detects planar surfaces (table, floor) and tracks its pose relative to them.

  • SLAM (Simultaneous Localization and Mapping): The device builds a sparse 3D map of the unknown environment while simultaneously determining its location within that map. Example: Microsoft HoloLens uses a custom SLAM engine.

  • Depth Sensing: Uses structured light, time-of-flight (ToF) cameras, or stereo cameras to get depth maps, improving tracking robustness and enabling occlusion. Example: iPhone LiDAR scanner.

Mathematical Modeling for VR Simulations: Worm on Rotating Wheel

• Problem: Worm crawls outward along a spoke of a wheel rotating in the y-z plane. Worm speed = 1 unit/sec, wheel angular speed = 1 rad/sec. At $$\displaystyle t=0 $$, spoke along +y axis, worm at origin. Find $\vec{r}(t)$.

• Solution:

  1. Wheel Rotation: The spoke rotates in the y-z plane. Its direction at time $t$ is given by a rotation matrix about the x-axis by angle $$\displaystyle \theta = \omega t = t $$ radians.

     Initial spoke vector: $$\displaystyle \hat{j} = (0,1,0) $$.

     Rotated spoke direction: $$\displaystyle \vec{d}(t) = R_x(t) \cdot (0,1,0) = (0, \cos t, \sin t) $$.

  2. Worm Position: Worm crawls along the spoke at speed $$\displaystyle v=1 $$. Distance from origin at time $t$ is $$\displaystyle s = v \cdot t = t $$.

  3. Vector Function: Position is distance along the direction vector.

$$\vec{r}(t) = s \cdot \vec{d}(t) = t \cdot (0, \cos t, \sin t)$$

$$\boxed{\vec{r}(t) = (0, \; t \cos t, \; t \sin t)}$$

Flight Dynamics Simulation in VR

• Techniques:

  1. Six-Degrees-of-Freedom (6-DOF) Model: Simulates motion along three translational axes (surge, sway, heave) and three rotational axes (roll, pitch, yaw). Based on rigid body dynamics.

  2. Aerodynamic Forces: Compute lift, drag, thrust, and weight using aircraft-specific coefficients and atmospheric models. Forces depend on airspeed, angle of attack, etc.

  3. Control Surfaces: Model deflection of ailerons, elevators, rudder to generate rotational moments.

  4. Simplified Models: For real-time VR, often use look-up tables (LUTs) or reduced-order models pre-calculated from high-fidelity simulations to balance realism and performance.

  5. Visual & Motion Cues: Use the computed motion to update the virtual cockpit view and, if available, a motion platform to enhance pilot sensation.

VI. SENSORY INTEGRATION IN AUGMENTED REALITY

Role of Human Senses in AR

• Visual Integration (Eyes):

  • Goal: Seamlessly overlay digital content onto the user's view of the real world.

  • Key Challenges: Registration (aligning virtual object with real-world location), Occlusion (virtual object correctly hidden by real objects), Lighting Consistency (virtual object matches real-world illumination).

  • Example: An AR maintenance app places a 3D schematic of a engine part over the real engine. The system uses SLAM to anchor the schematic, depth sensing to make it appear behind a real wrench, and environmental lighting estimation to shade the schematic realistically.

• Auditory Integration (Ears):

  • Goal: Use spatial audio to enhance presence and provide non-visual cues.

  • Implementation: 3D sound sources are positioned in the virtual/real space. Audio is filtered with HRTFs so a sound from your right is heard louder in your right ear.

  • Example: In the same maintenance app, when the user looks at a specific bolt, a voice instruction ("Turn this bolt counter-clockwise") seems to emanate from the bolt's location, guiding attention.

• Multisensory Example: A AR museum guide:

  • Visual: Historical figure avatar appears standing next to an artifact, explaining it.

  • Auditory: The avatar's voice comes from its position. Ambient sounds of the historical period play.

  • (Optional) Haptic: Controller vibrates when user "touches" a virtual artifact in the exhibit.

[!TIP] For sensory integration questions, always link the sense (visual/auditory) to a specific AR challenge (registration, spatial audio) and a concrete example.