Unit 5: Augmented and Virtual Reality - Short Notes

I. Fundamentals of Virtual Reality

Virtual Reality (VR) is a simulated, interactive, computer-generated environment that can be experienced and interacted with in a seemingly real or physical way by a person using specialized electronic equipment (head-mounted display, gloves, etc.).

How VR Works (System Overview):

1. Input: Sensors (head tracking, hand controllers) capture user's movements and position.

2. Processing: A powerful computer/GPU processes this input, updates the virtual world state, and renders the next frame from the user's current viewpoint.

3. Output: The rendered stereoscopic images are displayed on the HMD, and spatial audio is delivered via headphones.

4. Feedback Loop: This cycle (sense → process → act) runs at high speed (≥90 Hz) to maintain immersion and prevent motion sickness.

[!TIP] Exam Focus: Be prepared to draw a simple block diagram of the VR system loop: User → Sensors → Processing Unit → Display/Audio → User.

II. Visual and Graphical Technologies in VR

Stereo Technology

• Purpose: Creates the illusion of depth (stereopsis) by presenting a slightly different image to each eye.

• Hardware:

  • Head-Mounted Display (HMD): Contains two separate screens or one screen split into two viewports, with lenses to focus and magnify the image.

  • Stereo Renderer: GPU capability to render two views (left/right eye) per frame.

• Software Techniques:

  • Off-Axis Projection: Corrects for lens distortion and non-parallel eye axes in HMDs.

  • Asynchronous Timewarp/Reprojection: A last-minute correction technique that uses the most recent head orientation to warp the last rendered frame, reducing judder if a new frame isn't ready.

Geometric Modeling

• Definition: The mathematical representation of 3D objects (geometry, surfaces) and their spatial relationships in a virtual scene.

• Key Techniques:

  • Polygon Mesh (Polygonal Modeling): Most common. Objects are defined by vertices, edges, and faces (usually triangles).

  • NURBS (Non-Uniform Rational B-Splines): Smooth, precise curves and surfaces defined by control points.

  • Procedural Modeling: Algorithms generate geometry automatically (e.g., terrain, buildings).

• Role in VR: Provides the foundational 3D content. Models must be optimized (low polygon count) for real-time performance.

Real-Time Computer Graphics

• Core Requirement: Generate images at a high, consistent frame rate (typically 72-144 Hz) to avoid latency and motion sickness.

• Rendering Pipeline (Simplified):

  1. Application Stage: CPU sets up scene objects, camera, lights.

  2. Geometry Stage (Vertex Shader): GPU transforms 3D vertices to 2D screen coordinates, applies lighting per vertex.

  3. Rasterization Stage: GPU converts primitives (triangles) into fragments (potential pixels).

  4. Pixel Stage (Fragment Shader): GPU calculates final color for each pixel (texturing, per-pixel lighting, shadows).

• Performance Bottlenecks: Fill-rate (pixel shading), vertex processing, memory bandwidth. Level of Detail (LOD) techniques are critical.

Radiosity

• Purpose: A global illumination algorithm that simulates diffuse interreflection of light between surfaces, creating soft shadows and color bleeding.

• Basic Theory: Based on energy conservation. Surfaces are divided into small patches. Light energy (B_i) leaving a patch is the sum of its emitted light and light reflected from all other patches.

• Radiosity Equation (Form-Factor Based):

$$B_i = E_i + \rho_i \sum_{j=1}^{n} F_{ij} B_j$$

Where:

* $$\displaystyle B_i $$ = radiosity of patch *i* (total light energy leaving)

* $$\displaystyle E_i $$ = emitted light from patch *i*

* $$\displaystyle \rho_i $$ = reflectivity (albedo) of patch *i*

* $$\displaystyle F_{ij} $$ = **form factor** (fraction of light leaving *j* that arrives at *i*)

• Algorithm (Progressive/Shooting):

  1. Discretize scene into patches.

  2. Calculate form factors $$\displaystyle F_{ij} $$ (expensive, often using hemicube or ray-tracing).

  3. Initialize unshot light list with emitting patches.

  4. Iteratively "shoot" energy from brightest unshot patch to all others, updating their radiosities.

• VR Limitation: Radiosity is view-independent but computationally heavy (O(n²) for form factors). Used for pre-computation of static lighting in VR scenes, not for real-time dynamic lighting.

[!TIP] Common Pitfall: Radiosity solves for diffuse interreflection only. For specular reflections (mirrors) or caustics, ray-tracing is needed.

III. VR System Components and Tools

VR Toolkits (Engines)

• Purpose: Provide a framework with built-in systems for rendering, physics, audio, input, and asset management.

• Essential Features:

  • Stereo Rendering Pipeline: Built-in support for HMDs (OpenVR, Oculus SDK).

  • Cross-Platform Deployment: Build for PC VR, standalone (Quest), mobile.

  • Asset Pipeline: Import models (FBX, OBJ), textures, animations.

  • Physics Engine: Integrated (e.g., NVIDIA PhysX in Unreal, built-in in Unity).

  • Scripting/Logic: C# (Unity), C++/Blueprints (Unreal).

  • XR Interaction Toolkit: Pre-built components for grabbing, UI interaction.

• Examples: Unity (C#, larger asset store, popular for indie/mobile VR), Unreal Engine (C++/Blueprints, higher fidelity graphics out-of-box).

Collision Detection

• Goal: Detect when virtual objects intersect to prevent passing through each other and enable interaction.

• Generic Algorithms:

  • Bounding Volume Hierarchies (BVH): Wrap objects in simple shapes (AABB, spheres, OBBs). Check hierarchy for broad-phase culling before expensive mesh checks.

  • Sweep and Prune: Sort AABBs along an axis; only check overlapping intervals.

  • GJK Algorithm: Computes the minimum distance between two convex shapes efficiently.

  • Separating Axis Theorem (SAT): For convex polygons/polyhedra.

• Implementation in VR: Physics engines (PhysX, Bullet) handle continuous collision detection (CCD) to prevent fast-moving objects from tunneling.

Acoustic Hardware & Spatial Sound

• Components:

  • Headphones: Essential for isolating binaural audio cues.

  • 3D Audio Systems: Software that renders sound with position and distance cues.

• Spatial Sound Integration:

  • Binaural Rendering: Uses Head-Related Transfer Functions (HRTFs) to simulate how sound waves are filtered by the human head and ears before reaching the eardrums. Creates the illusion of sound coming from a specific 3D point.

  • Implementation: Engines provide audio sources with 3D attributes (position, velocity) and listener (player's head) orientation. The audio engine applies HRTF filters in real-time.

IV. Interaction and Simulation in VR

Interpolation and Translation

• Purpose: Smooth movement and animation of objects/avatars in the virtual environment.

• Interpolation (Between Keyframes):

  • Linear Interpolation (Lerp): $$\displaystyle P(t) = (1-t)P_0 + tP_1 $$. Simple, constant velocity.

  • Spherical Linear Interpolation (Slerp): For rotations on a sphere (quaternions), constant angular velocity.

  • Cubic Splines (Bezier, B-Spline): Smooth curves defined by control points. Used for complex, natural motion paths.

• Translation (Moving Objects in World):

  • Direct Position Setting: object.position = newPosition. Can cause jerky motion.

  • Velocity-Based: position += velocity * deltaTime. More physical.

  • Use Case: Interpolation for animation playback; translation for object manipulation or character movement.

Simulation Types

[!TIP] Exam Question: "Differentiate between behaviour and physical based simulation." Use this table structure in your answer.

V. Augmented Reality (AR)

AR Methods

1. Marker-Based: Uses predefined visual markers (QR codes, fiducials) detected by camera to anchor virtual content. Simple, accurate, but requires markers.

2. Marker-Less (Location/Feature-Based): Uses sensors (GPS, compass, IMU) or computer vision to track environment without markers.

3. Projection-Based: Projects digital light onto real surfaces, turning them into interactive displays (e.g., spatial AR).

4. Superimposition-Based: Replaces or augments part of the real-world view (e.g., medical AR overlaying scans on patient).

Marker-Less Tracking in AR

• Concept: Estimate the 6-DOF (six degrees of freedom: x, y, z, roll, pitch, yaw) pose of the camera relative to the environment.

• Sensors:

  • Camera: For visual tracking (feature matching).

  • IMU (Inertial Measurement Unit): Accelerometer, gyroscope, magnetometer for fast, short-term motion tracking (dead reckoning).

• Key Algorithms:

  • SLAM (Simultaneous Localization and Mapping): Core algorithm. Builds a map of an unknown environment while simultaneously keeping track of the device's location within it.

    • Visual SLAM (VSLAM): Uses camera input. Front-end: feature detection/tracking (ORB, SIFT). Back-end: pose graph optimization.

  • Feature Matching: Detect and match distinctive keypoints (corners, blobs) between current frame and a known map or previous frames. Used in VSLAM and plane detection.

Differentiation: AR vs. VR

VI. Applications of VR

VR in Digital Entertainment

• Gaming: Fully immersive interactive experiences. Genre-defining titles (e.g., Half-Life: Alyx). Requires intuitive locomotion (teleport, smooth) and interaction mechanics.

• Cinematic Experiences (VR Film/360° Video): Passive or slightly interactive storytelling where viewer looks around a spherical video. Challenges: narrative direction, user agency.

• Interactive Media & Social VR: Platforms (VRChat, Meta Horizon Worlds) where users socialize as avatars, attend live concerts or events in virtual spaces.

VII. Modeling and Description Languages

VRML (Virtual Reality Modeling Language)

• Purpose: A standard file format for representing 3D interactive vector graphics, designed for the web. Describes 3D scenes and objects.

• Syntax: Text-based, scene-graph structure.

  • Nodes: Basic building blocks (e.g., Shape, Transform, Group).

  • Fields: Properties of nodes (e.g., geometry, appearance, translation).

  • Prototyping: Allows defining new, reusable node types.

• Example Snippet:

  
  Transform {
  
    translation 1 2 3
  
    children [
  
      Shape {
  
        geometry Box { size 2 2 2 }
  
        appearance Appearance {
  
          material Material { diffuseColor 1 0 0 }
  
        }
  
      }
  
    ]
  
  }
  
  

• Role: Early standard for sharing 3D models on the web. Largely superseded by X3D (its successor) and modern game engine formats (glTF, FBX), but its scene-graph concepts remain foundational.

[!TIP] Exam Focus: Know the basic VRML node structure (Transform, Shape, geometry, appearance). You may be asked for a simple example.