Augmented and Virtual Reality (IT-702 (D)) - Unit 4 Short Notes

1.0 FUNDAMENTALS OF VIRTUAL REALITY (VR)

1.1 Definition & Core Concept of VR

• Virtual Reality (VR) is a computer-generated simulation of a three-dimensional environment that can be interacted with in a seemingly real or physical way by a person using specialized electronic equipment.

• Core Principle: The Three I's

  1. Immersion: The sensation of being inside the virtual environment (achieved via HMDs, spatial audio).

  2. Interaction: The ability to manipulate objects or navigate within the environment (via trackers, controllers).

  3. Imagination: The creative potential to experience the impossible or hypothetical.

• [!TIP] Exam Focus: Distinguish VR from simple 3D graphics by emphasizing presence (the feeling of "being there") enabled by immersion and interaction.

1.2 Stereoscopic Technology (Hardware & Software)

• Concept: Based on stereopsis—the human brain's ability to perceive depth by combining two slightly different 2D images (one for each eye).

• Hardware Technologies:

  • Head-Mounted Displays (HMDs): Primary VR hardware. Contains two small displays (one per eye) within a headset.

  • Shutter Glasses: Liquid crystal lenses that alternate opacity rapidly in sync with the display showing left/right eye images.

  • Polarized Glasses: Use filters with orthogonal polarization for left/right eye images projected onto a single screen.

• Software Technologies:

  • Rendering: The graphics engine must render two distinct views (left-eye, right-eye) from slightly offset camera positions.

  • Synchronization: Software must perfectly synchronize the display of stereo pairs with shutter glasses or polarization switches to avoid crosstalk and ghosting.

1.3 Real-Time Computer Graphics

• Definition: The generation and rendering of imagery at a rate sufficient to create the illusion of motion and immediate response to user input.

• Critical Importance in VR: Latency (delay between user action and system response) must be < 20ms to prevent motion sickness and maintain presence. Frame Rate must be consistently high (≥ 90 FPS for modern VR, ≥ 60 FPS minimum).

• Key Techniques:

  • Level of Detail (LOD): Dynamically switching model complexity based on distance to reduce rendering load.

  • Culling: Not rendering objects outside the view frustum (frustum culling) or occluded by other objects (occlusion culling).

  • Rendering Pipeline Optimization: Efficient use of GPU for vertex processing, rasterization, and fragment shading.

2.0 FUNDAMENTALS OF AUGMENTED REALITY (AR)

2.1 Definition & Core Concept of AR

• Augmented Reality (AR) is a technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view.

• Key Goal: To enhance, not replace, the user's perception of the real environment by adding contextual digital information.

2.2 AR Methods & Techniques

• Marker-Based AR: Uses predefined visual fiducial markers (e.g., QR codes). The camera detects the marker, and software calculates its position/orientation to anchor virtual content.

• Marker-Less Tracking (Feature-Based): Does not rely on special markers. Techniques include:

  • SLAM (Simultaneous Localization and Mapping): Algorithm that builds a map of an unknown environment while simultaneously keeping track of the device's location within it.

  • Feature Matching: Detects and tracks natural features in the environment (edges, corners) from frame to frame.

  • Sensor Fusion: Combines data from inertial sensors (IMU: accelerometer, gyroscope) with camera data for robust, low-latency tracking.

• Projection-Based AR: Projects digital light onto real surfaces, turning them into interactive displays (e.g., spatial AR).

• Superimposition-Based AR: Replaces part of the real-world view with an augmented version (e.g., virtual try-on for glasses, medical imaging overlay).

2.3 AR vs. VR: Comprehensive Differentiation

3.0 VR/AR SYSTEM COMPONENTS & HARDWARE

3.1 Display Systems

• Head-Mounted Displays (HMDs):

  • Video See-Through: Uses cameras to capture the real world, which is then combined with virtual graphics and displayed on internal screens (e.g., Meta Quest). Allows complete control over the final image.

  • Optical See-Through: Uses semi-transparent displays (e.g., holographic waveguides) that allow the user to see the real world directly through the lenses, with virtual graphics projected onto them (e.g., Microsoft HoloLens). Lower latency for real-world view.

• CAVE (Cave Automatic Virtual Environment): A room-sized cube with projections on all walls/floor. Users wear shutter glasses. Provides high immersion without head-worn displays, but requires large space.

• Handheld Displays: Smartphones/tablets for mobile AR. The display acts as a "window" into the augmented world.

3.2 Input & Tracking Devices

• Tracking Technologies:

  • Optical: Uses external cameras to track markers on the HMD/controllers (e.g., VIVE Lighthouse, OptiTrack). High accuracy, limited by line-of-sight.

  • Magnetic: Uses a fluctuating electromagnetic field from a base station to track sensors. Prone to metal interference.

  • Inertial: Uses IMUs (accelerometers, gyroscopes) inside the HMD. Good for orientation, drifts over time for position (dead reckoning).

  • Hybrid: Combines multiple methods (e.g., optical + inertial) to compensate for individual weaknesses (most modern systems).

• Devices: Data gloves (hand/finger tracking), wand controllers, motion capture suits (full-body tracking).

3.3 Acoustic Hardware in VR Systems

• Importance: 3D Spatial Audio is critical for immersion and presence. It provides auditory cues about the location, distance, and movement of objects/events in the virtual environment.

• Hardware:

  • Headphones/Headsets: Most common. Can be stereo or binaural (individual channels for each ear) for precise localization.

  • Speaker Arrays: Surround sound setups (e.g., 5.1, 7.1) for CAVE systems or room-scale VR.

  • Bone Conduction: Transducers that vibrate the skull, leaving ears open for real-world sound (useful for AR/ambient awareness).

• Role: Enhances realism, provides non-visual feedback for interactions (e.g., sound of a picked-up object), and can be used as an interaction channel (voice commands).

4.0 GEOMETRIC MODELING & CONTENT CREATION

4.1 Geometric Modeling

• Definition: The mathematical representation of 3D objects' geometry (shape, form) for use in computer graphics and simulation.

• Key Representations:

  • Polygon Meshes: Most common. Surfaces defined by connected polygons (usually triangles). Simple, efficient for rendering.

  • Parametric Surfaces: Defined by mathematical functions with parameters (e.g., u, v). Includes:

    • NURBS (Non-Uniform Rational B-Splines): Industry standard for smooth, precise curves and surfaces (e.g., automotive, industrial design). Offer local control and exact conic representation.

  • Procedural Modeling: Objects generated algorithmically via rules/parameters (e.g., cities, terrain). Efficient for large-scale, repetitive content.

4.2 VRML (Virtual Reality Modeling Language)

• Definition: A standard file format and scene description language for representing 3D interactive vector graphics, especially for the World Wide Web. Precursor to X3D.

• Key Features:

  • Hierarchical Scene Graph: Objects are nodes in a tree structure (transformations, shapes, lights).

  • Basic Shapes: Box, cone, cylinder, sphere, text.

  • Sensors: Detect user interaction (e.g., TouchSensor, TimeSensor) and generate events.

  • Routes: Connect event outputs from one node to event inputs of another, defining behavior (e.g., TouchSensor -> PositionInterpolator -> Transform).

  • Prototypes (PROTO): Allow users to define new, reusable node types with custom interfaces.

4.3 Radiosity

• Definition: A global illumination algorithm that simulates the diffuse interreflection of light between surfaces in a scene to produce realistic soft shadows and color bleeding.

• Algorithm (Form-Factor Approach):

  1. Mesh Discretization: Surfaces are divided into small patches.

  2. Form-Factor Calculation: Compute the fraction of light leaving one patch that directly reaches another (F_ij). This is view-dependent and computationally expensive.

  3. Solving Radiosity Equation: For each patch i, its final brightness B_i is:

$$B_i = E_i + \rho_i \sum_{j=1}^{n} F_{ij} B_j$$

    where `E_i` is emitted light, `ρ_i` is reflectivity (albedo).

4.  This forms a large system of linear equations, solved iteratively (e.g., Gauss-Seidel).

• Comparison with Ray Tracing:

  • Radiosity: View-independent (once solved). Excellent for diffuse interreflection. Computationally heavy for setup, but view changes are cheap. Poor for specular reflections/mirrors.

  • Ray Tracing: View-dependent. Excellent for specular reflections, refractions, shadows. Computationally heavy per view.

• Application in VR: Primarily used for pre-computation of high-quality, static lighting (lightmaps) for VR scenes to achieve realism without real-time performance cost.

5.0 VR SOFTWARE, TOOLKITS & INTERACTION

5.1 VR Toolkits & Engines (Features)

• Purpose: Middleware that abstracts low-level hardware/API details, providing a framework for building VR applications efficiently.

• Key Features:

  • Rendering Engine: Handles stereo rendering, distortion correction for HMD lenses.

  • Physics Engine: Simulates rigid/soft body dynamics, collisions, constraints (e.g., NVIDIA PhysX, Bullet).

  • Input Handling: Abstracts trackers, controllers into a unified API.

  • Asset Pipeline: Tools for importing/optimizing 3D models, textures, animations.

  • Cross-Platform Support: Deploy to multiple HMDs/OS (e.g., OpenXR standard).

• Examples: Unity3D (C#, large asset store), Unreal Engine (C++, high-fidelity graphics), OpenVR/SteamVR (API layer), OpenSceneGraph (OSG) (open-source graphics toolkit).

5.2 Interpolation & Translation in Virtual Environments

• Interpolation: Estimating intermediate values between known keyframes for smooth animation/motion.

  • Linear Interpolation (Lerp): Simple straight-line path between points. $$\displaystyle P(t) = (1-t)P_a + tP_b $$.

  • Spherical Linear Interpolation (Slerp): For rotations (quaternions), maintains constant velocity along great circle on hypersphere.

  • Spline Interpolation (e.g., Bézier, Catmull-Rom): Uses control points to create smooth, curved paths.

• Translation (Types of Movement):

  • World-Relative: Movement is along global X/Y/Z axes (e.g., "move forward" always means +Z in world space).

  • Object-Relative: Movement is relative to the local coordinate system of the user's current avatar/object (e.g., "move forward" means in the direction the user is facing).

  • Hand-Relative: Movement is driven directly by hand/controller position (e.g., "fly" by moving controller forward).

5.3 Collision Detection in Generic VR Systems

• Definition: Determining if two or more objects in the virtual environment intersect, necessary for realistic interaction (e.g., object cannot pass through wall, hand grabs object).

• Two-Phase Approach:

  1. Broad Phase: Quickly find pairs of potentially colliding objects from the entire set. Uses simple, fast tests.

     • Techniques: Bounding Volume Hierarchies (BVH), Spatial Partitioning (Octrees, BSP Trees, Uniform Grids).

  2. Narrow Phase: Perform precise collision test on the pairs identified by the broad phase.

     • Techniques: Bounding Volumes: Axis-Aligned Bounding Boxes (AABB), Oriented Bounding Boxes (OBB), Bounding Spheres. Then, for meshes, triangle-triangle intersection tests (e.g., Möller–Trumbore algorithm).

• Response: Upon detection, resolve collision (e.g., prevent movement, generate sound/particle effect, apply physics forces).

• [!TIP] Exam Focus: Be prepared to differentiate Broad Phase (efficiency, culling) vs. Narrow Phase (accuracy, precision) and list common spatial partitioning structures.

6.0 SIMULATION & PHYSICS IN VR

6.1 Behaviour-Based vs. Physics-Based Simulation

• Behaviour-Based Simulation:

  • Definition: Objects exhibit pre-scripted, rule-driven behaviors. Animations are deterministic and artist-controlled.

  • Example: A door that opens when a user's hand enters a trigger volume, playing a canned animation.

  • Pros: Predictable, cheap computationally, full artistic control.

  • Cons: Lacks realism; cannot handle unforeseen interactions.

• Physics-Based Simulation:

  • Definition: Uses a physics engine to simulate real-world physical properties (mass, friction, elasticity) and laws (Newtonian mechanics).

  • Components: Rigid Body Dynamics (unbreakable objects), Soft Body Dynamics (deformable objects), Constraints/Joints (hinges, sliders).

  • Pros: High realism, emergent behavior (unscripted interactions), consistent with user expectations.

  • Cons: Computationally expensive, can be unstable ("explosions" from tunneling), less predictable.

• Comparison:

  | Aspect | Behaviour-Based | Physics-Based | | :--- | :--- | :--- | | Realism | Low (scripted) | High (dynamic) | | Computational Cost | Low | High | | Predictability | High | Low (can be chaotic) | | Use Case | Cutscenes, UI animations, simple props. | Object manipulation, vehicle sims, cloth, destruction. |

6.2 General Concepts in Simulation

• Purpose: To create a model of a real (or conceptual) system for the purposes of training, analysis, prototyping, or entertainment.

• Integration with VR: The simulation loop (physics, AI, logic updates) must be synchronized with the render loop. Typically, simulation runs at a fixed timestep (e.g., 60 Hz) for stability, independent of variable rendering framerate.

7.0 APPLICATIONS OF VR & AR

7.1 VR in Digital Entertainment

• Video Games: Immersive, first-person experiences across genres (horror, adventure, simulators). Room-scale and seated experiences.

• Cinematic Experiences: 360° Video (viewer looks around a pre-recorded sphere), Interactive Films (user choices affect narrative).

• Theme Parks & Location-Based Entertainment: VR attractions, motion platforms, haptic feedback suits for group experiences.

7.2 Other Key Application Domains

• Training & Simulation: Aviation (flight sims), medical (surgery rehearsal), military (battlefield scenarios), hazardous environment training.

• Education & Virtual Field Trips: Exploring historical sites, molecular structures, astronomical phenomena.

• Architecture & Urban Planning: Walkthroughs of unbuilt structures, urban scale visualization, design review.

• Healthcare: Therapy (PTSD, phobias), pain distraction, rehabilitation exercises, surgical planning with patient-specific 3D models.

• Retail & E-commerce: AR Try-Ons (glasses, makeup, clothes), virtual showrooms, product visualization in home space (furniture).