Augmented and Virtual Reality (IT-702 (D)) - Unit 3 Short Notes

I. FUNDAMENTALS OF VR AND AR

Virtual Reality (VR) is a computer-generated, immersive simulation of a 3D environment that replaces the user's real-world perception. The primary goal is to create a sense of presence—the feeling of being physically located within the virtual world.

Augmented Reality (AR) overlays digital content (graphics, sound, haptic feedback) onto the real world in real-time, enhancing rather than replacing the user's perception of reality.

History & Evolution: Key milestones include Ivan Sutherland's Sword of Damocles (1968) as the first HMD, the development of VRML in the 1990s for web-based 3D, and the modern consumer boom driven by Oculus Rift (2012) and smartphone-based AR (ARKit/ARCore).

Core Characteristics of Immersive Systems:

• Immersion: Fidelity of sensory stimuli (visual, auditory, haptic).

• Presence: Psychological sensation of "being there."

• Interactivity: Real-time response to user actions.

• Consistency: Plausible and stable simulation rules.

[!TIP] Exam Focus: AR vs. VR comparison is a high-frequency question. Use the table structure for clear, marks-friendly answers. Always link characteristics to their technological implications (e.g., VR needs higher latency tolerance than AR).

II. DISPLAY AND VISUALIZATION TECHNOLOGIES

Stereo Technology (Stereoscopic Vision)

The principle relies on binocular disparity—the slight difference in the images seen by each eye—which the brain processes to perceive depth (stereopsis).

Hardware Technologies:

• Head-Mounted Displays (HMDs): Integrated stereo displays (e.g., Oculus Quest, HTC Vive). Each eye sees a separate image.

• Shutter Glasses: Active glasses synchronized with a high-refresh-rate display (120Hz+), alternately blocking each eye's view.

• Polarized Displays: Passive glasses with orthogonal polarizing filters; common in 3D cinemas and some CAVE systems.

Software Techniques:

• Stereo Rendering Pipeline: The graphics engine renders the scene twice per frame—once from the perspective of the left eye, once from the right eye—with a slight horizontal offset (interpupillary distance, IPD).

• Asynchronous Timewarp/Reprojection: A critical technique to reduce judder. If a new frame isn't ready, the system reuses the previous frame but updates the head orientation using the latest tracking data, reducing perceived latency.

Display Quality Metrics:

• Field of View (FOV): The angular extent of the visible scene. Human horizontal FOV is ~200°; VR HMDs typically offer 90-110°.

• Resolution: Pixels per degree (PPD) is more relevant than absolute resolution. Higher PPD reduces the "screen-door effect."

• Latency: Motion-to-photon latency (time from head movement to display update) must be < 20ms to prevent simulator sickness.

• Refresh Rate: Must be high (90Hz minimum, 120Hz+ preferred) to ensure smooth motion.

[!TIP] Common Pitfall: Students often confuse field of view with resolution. Emphasize that FOV is about coverage (how much you see), while resolution is about clarity (how sharp it is).

III. TRACKING AND INPUT SYSTEMS

Tracking Technologies

Inside-Out Tracking: Sensors (cameras, IMUs) are placed on the device (e.g., HMD, phone). The system tracks the environment relative to itself. Used in standalone VR headsets (Oculus Quest) and mobile AR. Outside-In Tracking: External sensors/cameras (e.g., Lighthouse, OptiTrack) track markers on the user/device. Offers higher precision but requires setup.

Marker-less Tracking in AR (SLAM): SLAM (Simultaneous Localization and Mapping) is the cornerstone of modern marker-less AR. It solves two problems concurrently:

1. Localization: Determining the device's 6-DOF (Degrees of Freedom) pose (position + orientation) in the environment.

2. Mapping: Building a sparse or dense 3D representation of the surroundings.

Algorithm Core: Typically uses feature detection (e.g., ORB, SIFT) on camera frames, matches features across frames, and optimizes pose via bundle adjustment or pose graph optimization. Outputs a point cloud map and continuous pose estimates.

Other Tracking Methods:

• Marker-Based (Fiducial): Uses pre-defined visual patterns (e.g., QR codes). Simple, robust, but requires placing markers.

• Inertial Tracking: Uses IMUs (accelerometers, gyroscopes) for 3DOF (orientation only) or 6DOF (with sensor fusion). Prone to drift.

• Optical Tracking: Passive/active markers tracked by external cameras. High accuracy, limited range.

• Magnetic Tracking: Uses a fluctuating electromagnetic field. Unaffected by occlusions but sensitive to metal.

Degrees of Freedom (DOF):

• 3DOF: Rotation around X, Y, Z axes (yaw, pitch, roll). Sufficient for looking around in a fixed position (360° video).

• 6DOF: 3DOF + translation along X, Y, Z axes (moving forward/back, strafing, crouching). Essential for room-scale VR and AR.

[!TIP] Exam Focus: SLAM is a must-know. Be prepared to explain the simultaneous nature of localization and mapping. Contrast inside-out (self-contained) vs. outside-in (external infrastructure).

IV. INTERACTION AND NAVIGATION IN VIRTUAL ENVIRONMENTS

Movement and Navigation Techniques

Interpolation and Translation refer to methods for moving the user's viewpoint (virtual camera) through the environment.

Locomotion Issues:

• Simulator Sickness: Caused by sensory conflict (e.g., eyes see motion, inner ear does not). Symptoms: nausea, dizziness, sweating.

• Presence: Continuous physical walking maximizes presence; artificial locomotion can break it.

Collision Detection

Importance: Prevents the user from passing through virtual objects (immersion break) and is fundamental for physics-based interaction (grabbing, pushing).

Basic Algorithms & Techniques:

1. Bounding Volumes: Simple shapes enclosing complex objects for fast initial checks.

   • Axis-Aligned Bounding Box (AABB): Fastest, but must be recalculated on rotation.

   • Oriented Bounding Box (OBB): Tighter fit, more expensive.

   • Bounding Sphere: Rotation-invariant, simple distance check.

2. Spatial Partitioning: Divides the world into cells to reduce pairwise checks.

   • Grids/Octrees: Hierarchical subdivision of 3D space.

   • BSP Trees (Binary Space Partitioning): Pre-computed tree for static environments.

   • k-d Trees: Similar to BSP, often used for ray tracing acceleration.

3. Sweep and Prune: Sort objects along an axis; overlapping intervals indicate potential collisions. Efficient for dynamic scenes.

4. Broad Phase & Narrow Phase: Two-stage approach. Broad phase (using AABBs/grids) finds potential colliding pairs. Narrow phase performs precise tests (e.g., triangle-triangle intersection) on those pairs.

Collision Response: Once a collision is detected, determine the physical reaction (e.g., bounce, stop, slide) based on material properties and velocities, often handled by a physics engine (e.g., PhysX, Havok).

[!TIP] Common Pitfall: Do not confuse collision detection (finding if/when objects intersect) with collision response (calculating the physical reaction). Be ready to explain both.

V. MODELING AND SIMULATION

Geometric Modeling

Techniques to create 3D virtual objects:

• Polygon Meshes: Most common. Surfaces defined by vertices, edges, and faces (triangles/quads). Efficient for rasterization.

• NURBS (Non-Uniform Rational B-Splines): Mathematical curves/surfaces defined by control points. Used for smooth, precise surfaces (automotive, industrial design).

• Procedural Modeling: Algorithms generate geometry (e.g., fractals for terrain, L-systems for plants). Efficient for large, repetitive environments.

Simulation Types

VRML (Virtual Reality Modeling Language)

• Purpose: An ISO-standard (ISO/IEC 14772) file format for describing interactive 3D vector graphics, designed for the web.

• Basic Structure: Text-based, scene graph hierarchy. Core node types:

  • Shape: Combines geometry (e.g., Box, Sphere, IndexedFaceSet) and appearance (Material, Texture).

  • Transform: Applies translation, rotation, scale to child nodes.

  • Group: Collects nodes.

  • Sensor: Detects user interaction (e.g., TouchSensor, TimeSensor).

  • Route: Connects events between nodes (e.g., TOUCH_TIME to set_translation).

• Current Status: Largely superseded by X3D (its XML-based successor) and modern runtime formats like glTF (GL Transmission Format), which is optimized for efficient transmission and loading in real-time applications.

[!TIP] Exam Focus: Be able to contrast Behavior vs. Physical Simulation with clear examples. For VRML, know its scene graph nature and key node types (Shape, Transform, Sensor, Route). Mention glTF as the modern alternative.

VI. REAL-TIME COMPUTER GRAPHICS

Fundamentals of Real-Time Rendering Pipeline:

1. Application Phase: CPU prepares geometry, transforms, lighting data.

2. Geometry Processing (Vertex Shader): Transforms vertices from model space to clip space. Per-vertex lighting.

3. Rasterization: Fixed-function stage converting primitives (triangles) into fragments (potential pixels).

4. Fragment Processing (Pixel Shader): Computes final color for each fragment (texturing, per-pixel lighting, shadows).

5. Output Merging: Blends fragments into the framebuffer (depth test, alpha blending).

Rendering Techniques

Radiosity:

A global illumination algorithm that simulates the diffuse interreflection of light between surfaces. It treats surfaces as patches that emit, reflect, and receive light energy.

Key Concepts:

• Form Factor (F_ij): The fraction of light leaving patch i that arrives directly at patch j. It is a function of geometry, size, distance, and orientation.

$$F_{ij} = \frac{1}{A_i} \int_{A_i} \int_{A_j} \frac{\cos\theta_i \cos\theta_j}{\pi r^2} V_{ij} dA_j dA_i$$

where $A$ is area, $\theta$ is angle to normal, $r$ is distance, $$\displaystyle V_{ij} $$ is visibility (1 if visible, 0 if occluded).

• Radiosity Equation (for patch i):

$$B_i = E_i + \rho_i \sum_{j=1}^{n} F_{ij} B_j$$

$$\displaystyle B_i $$ = radiosity (total light energy leaving patch *i*), $$\displaystyle E_i $$ = emitted light, $$\displaystyle \rho_i $$ = reflectivity (albedo).

• Solution Methods:

  • Gathering/Progressive Radiosity: Solves the linear system iteratively. In each step, unshot light (radiosity) from "bright" patches is distributed to others. Visually converges quickly for initial bounces.

• Applications: Excellent for scenes with diffuse interreflection (e.g., indoor scenes with colored walls). Not suitable for specular reflections or real-time (traditionally offline).

Ray Tracing vs. Rasterization in Real-Time:

• Rasterization: Industry standard for real-time. Fast, efficient, but struggles with accurate reflections, shadows, and global illumination (requires approximations like shadow maps, lightmaps).

• Ray Tracing: Traces rays from camera/light to simulate light transport physically. Produces photorealistic effects (reflections, shadows, GI) but computationally expensive.

• Hybrid Approaches (e.g., NVIDIA RTX): Use rasterization for primary visibility and dedicated hardware (RT cores) for selective ray tracing (shadows, reflections).

Performance Optimization:

• Level of Detail (LOD): Swap high-poly models for low-poly versions at a distance.

• Culling:

  • Frustum Culling: Remove objects outside the camera's view frustum.

  • Occlusion Culling: Remove objects blocked by other objects.

• Shading Models: Use simpler models (e.g., Lambertian for diffuse) where possible. Physically Based Rendering (PBR) is now standard for realism.

[!TIP] Exam Focus: Radiosity is a high-weight topic. Explain the form factor intuition (geometry-dependent visibility/angle term) and the progressive solving method. Contrast ray tracing (accuracy) vs. rasterization (speed) clearly.

VII. AUDIO SYSTEMS IN VR

Role of Audio: Critical for immersion and presence. Provides spatial cues, emotional context, and feedback that visuals alone cannot. Poor audio can break the illusion faster than poor graphics.

Acoustic Hardware in VR Systems:

• Headphones vs. Speakers:

  • Headphones (Preferred): Deliver binaural audio directly to each ear, essential for accurate 3D sound localization. Prevents sound leakage and external noise.

  • Speakers: Used in some CAVE systems or for shared experiences, but harder to achieve precise spatialization.

• HRTF (Head-Related Transfer Function): The core technology for 3D audio. An HRTF is a filter that describes how sound from a specific direction is transformed by the human anatomy (head, torso, ears) before reaching the eardrum. By convolving a mono sound source with the appropriate HRTF for its virtual direction, the brain perceives the sound as coming from that location.

  • Personalization: HRTFs are highly individual. Generic HRTFs work but can cause "front-back confusion" or poor elevation perception for some users.

3D Audio & Spatial Sound Rendering Techniques:

• Binaural Rendering: Real-time application of HRTFs and Interaural Time Differences (ITD)/Level Differences (ILD) to place sounds in 3D space around the listener.

• Ambisonics (First-Order): A full-sphere sound field representation (4 channels: W, X, Y, Z). Efficient for decoding to any speaker layout or binaural. Common in VR engines (Unity/Unreal).

• Sound Propagation Simulation: Models how sound waves reflect, diffract, and absorb in the virtual environment (e.g., echo in a canyon, muffled sound through a wall). Computationally intensive; often uses pre-computed reverberation zones or simplified ray-based models.

[!TIP] Key Point: Always connect HRTF to binaural audio and headphones. Explain that HRTF is what makes a sound seem "in front" vs. "behind."

VIII. DEVELOPMENT TOOLS AND PLATFORMS

Features of VR Toolkits and Engines (Unity, Unreal Engine, WebXR):

Development Workflow & Best Practices:

1. Prototype Interaction First: Use simple shapes (cubes, spheres) to test locomotion, grabbing, UI.

2. Optimize Early: Target 90 FPS for PC VR, 72/90 FPS for standalone. Use GPU Profiler to find bottlenecks (fill-rate, shader complexity).

3. Use Prefabs/Actors: Reusable objects with consistent behavior.

4. Test on Target Hardware: Editor performance is not indicative of HMD performance.

[!TIP] Exam Focus: For "features of VR toolkits," list 4-5 key features with a brief example (e.g., "Cross-platform: A Unity project can build to Oculus Quest and HoloLens using the same codebase"). Mention Single-Pass Stereo Rendering and XR Interaction Toolkit as modern essentials.

IX. AUGMENTED REALITY TECHNIQUES AND METHODS

Marker-Less AR Deep Dive (SLAM):

• Front-End: Feature extraction & matching from camera frames. Estimates initial pose.

• Back-End: Optimization (bundle adjustment, pose graph) to minimize drift and create a consistent map.

• Plane Detection: A specific SLAM output that identifies horizontal/vertical surfaces (tables, floors, walls) for placing objects stably. Implemented in ARCore (Augmented Images, Cloud Anchors) and ARKit (Plane Detection, Scene Reconstruction).

Registration & Tracking Challenges:

• Drift: Cumulative error in pose estimation over time. SLAM loop closure helps.

• Occlusions: Virtual objects should be hidden by real objects. Requires depth sensing (LiDAR, stereo cameras) or depth-from-motion.

• Lighting Consistency: Virtual objects must match real-world lighting (shadows, color temperature). Requires environmental lighting estimation.

Display Methods:

• Optical See-Through: Uses transparent displays (e.g., Microsoft HoloLens waveguides). Combines real light with projected digital light. Low latency, but limited brightness/field of view.

• Video See-Through: Camera(s) capture real world, which is composited with rendered virtual content and displayed on an opaque screen (e.g., mobile AR). Allows full control over the final image (HDR, filters) but introduces camera-processing latency.

[!TIP] Exam Focus: Be able to differentiate the three AR methods in a table. For marker-less, explicitly state SLAM as the enabling technology. Contrast optical vs. video see-through on latency and image control.

X. APPLICATIONS OF VR AND AR

VR in Digital Entertainment

• Gaming: The largest sector. From room-scale experiences (Beat Saber, Half-Life: Alyx) to seated experiences (Elite Dangerous). Drives hardware innovation.

• Cinematic Experiences: 360° video (YouTube VR), interactive narratives (Carne y Arena by Alejandro Iñárritu). Challenges: directing viewer attention in a sphere.

• Theme Parks & Location-Based VR: Out-of-home entertainment. Examples: The Void (Star Wars: Secrets of the Empire) combines VR with physical sets, props, and haptic feedback for high immersion.

Other Application Domains (Brief):

• Education & Training: Safe simulations for medical procedures, flight training, hazardous equipment operation. High retention through "learning by doing."

• Healthcare: Exposure therapy (PTSD, phobias), surgical planning (viewing patient scans in 3D), pain distraction.

• Architecture & Urban Planning: Virtual walkthroughs of unbuilt structures, client presentations, urban scale visualization.

• Industrial Design & Prototyping: Virtual prototyping, assembly simulations, factory layout planning (digital twins).

• Social VR & Collaboration: Platforms like VRChat, Meta Horizon Workrooms, and Spatial for remote meetings in shared virtual spaces with avatars.

[!TIP] Exam Focus: For "applications in digital entertainment," structure the answer into Gaming, Cinematic, Theme Parks. Use specific, named examples to demonstrate knowledge. For other domains, list 2-3 key applications per domain.

XI. VR SYSTEM ARCHITECTURE AND COMPONENTS

Generic VR System Architecture (Input-Process-Output Loop):

[User]

   |

   v

+-------------------+

| INPUT (Tracking)  | --> Head pose, controller position, button presses.

+-------------------+

   |

   v

+-------------------+

| PROCESSING        | --> 1. Update virtual camera based on tracked pose.

| (Simulation)      | --> 2. Run physics simulation (collisions, rigid bodies).

|                   | --> 3. Update AI/behaviors.

|                   | --> 4. Determine visible objects (culling).

+-------------------+

   |

   v

+-------------------+

| OUTPUT (Render)   | --> Render stereo scene (left eye, right eye).

+-------------------+

   |

   v

[Display (HMD)]

How VR Technology Works (End-to-End Pipeline):

1. Tracking: Sensors (IMUs, cameras) capture head/hand motion. Data is filtered (sensor fusion: Kalman filter, complementary filter) to produce a smooth, low-latency 6-DOF pose.

2. Simulation Update: The application uses the new pose to:

   • Position the virtual camera(s).

   • Advance the physics simulation (time step).

   • Update game logic/AI.

3. Rendering:

   • View Frustum Culling: Determine objects visible to each eye.

   • Render Scene: Draw all visible objects with appropriate shaders, lighting, and effects. Stereo rendering is performed (two view matrices, two render targets).

   • Post-Processing: Apply effects like chromatic aberration correction, distortion correction (for lens), and temporal anti-aliasing (TAA).

4. Display & Scanout: Rendered left/right images are sent to the HMD's displays. The display hardware scans out the images at the refresh rate (e.g., 90Hz). Asynchronous Timewarp may reproject the final image if a frame is missed.

5. User Perception: The user sees the updated stereo image, completing the loop.

System Integration Challenges:

• Latency: The total motion-to-photon latency must be <20ms. Each stage (tracking, processing, rendering, display) contributes. Asynchronous Timewarp is a critical last-line defense.

• Synchronization: Tracking data must be time-stamped and matched to the correct simulation frame. GPU-CPU synchronization must be managed to avoid stuttering.

• Performance: Must maintain a consistent, high framerate (90Hz+). Dropped frames cause stutter and break presence.

[!TIP] Key Diagram: Be able to sketch the Input-Process-Output loop. For "How VR technology works," describe the 5-stage pipeline in sequence, emphasizing the stereo rendering and latency reduction techniques at each stage.