Augmented and Virtual Reality (IT-702 (D)) - Unit 2 Short Notes

I. FOUNDATIONS OF VIRTUAL AND AUGMENTED REALITY

Definition and Core Concepts

• Virtual Reality (VR): A simulated, computer-generated environment that immerses the user in a 3D world, allowing interaction and exploration. Core principles:

  • Immersion: Sensory engagement that creates a sense of "being there" (presence).

  • Interaction: Real-time response to user actions (movement, manipulation).

  • Imagination: System's ability to render believable, dynamic worlds.

• How VR Works:

  • System Components: Head-Mounted Display (HMD), tracking sensors (head, hands), input devices (controllers), high-performance GPU/CPU, software engine.

  • Process: User movement tracked → scene re-rendered from new viewpoint → displayed to HMD with low latency (<20 ms) to prevent motion sickness.

• Differentiate AR vs VR:

Augmented Reality Specifics

• AR Methods:

  1. Marker-based: Uses visual markers (QR codes) to anchor virtual content.

  2. Marker-less: No markers; uses sensors (GPS, IMU) or computer vision (SLAM).

  3. Projection-based: Projects digital light onto real surfaces for interaction.

  4. Superimposition-based: Replaces real objects with augmented ones (e.g., medical imaging).

• Marker-less Tracking:

  • Sensor-based: Uses device sensors (accelerometer, gyroscope, GPS) for coarse tracking.

  • Computer vision-based: Uses cameras and algorithms like SLAM (Simultaneous Localization and Mapping) to map environment and track position.

  • Hybrid: Combines sensor data and computer vision for accuracy and robustness.

  [!TIP] Marker-less tracking is essential for mobile AR (ARKit, ARCore) and requires real-time performance.

II. VISUAL DISPLAY TECHNOLOGIES & GEOMETRY

Stereoscopic Vision

• Stereo Technology: Based on stereopsis—depth perception from two slightly different views (like human eyes). Displays separate images to each eye to create 3D illusion.

  • Hardware: HMDs with separate lenses, shutter glasses (active), polarized glasses (passive), autostereoscopic displays (no glasses).

  • Software: Render scene from two viewpoints (left/right eye), apply distortion correction for lenses, synchronize with display refresh.

  [!TIP] Key challenge: vergence-accommodation conflict—eyes focus at display distance but converge at virtual depth, causing discomfort.

3D Modeling and Representation

• Geometric Modeling: Creating mathematical representations of 3D objects.

  • Basic Elements: Vertices (points), Edges (lines), Faces (polygons, usually triangles), Meshes (collection of vertices/edges/faces).

  • Techniques:

    • Polygonal modeling: Uses polygons (triangles/quads); most common in real-time graphics.

    • NURBS (Non-Uniform Rational B-Splines): Smooth curves/surfaces; used in CAD.

    • Constructive Solid Geometry (CSG): Combines primitive shapes (cubes, spheres) via Boolean operations.

• VRML (Virtual Reality Modeling Language):

  • Purpose: Text-based file format for describing interactive 3D vector graphics, originally for web.

  • Structure: Scene graph with nodes (e.g., Shape, Transform, Group) and fields (attributes like geometry, appearance).

  • Example:

    
    Transform {
    
      translation 1 0 0
    
      children [
    
        Shape {
    
          geometry Sphere { radius 0.5 }
    
        }
    
      ]
    
    }
    
    

  [!TIP] VRML is largely superseded by X3D and glTF, but its scene graph concepts remain foundational.

Rendering Techniques

• Real-time Computer Graphics:

  • Pipeline: Application → Geometry processing (vertex shader) → Rasterization → Pixel processing (fragment shader) → Output.

  • Shading: Calculating color/brightness per pixel; Phong shading interpolates normals and computes per-pixel lighting.

  • Lighting: Directional, point, spot lights; shadows via shadow maps.

  • Performance: Must run at high FPS (90+ for VR); use level-of-detail (LOD), occlusion culling, efficient shaders.

• Radiosity:

  • Theory: Global illumination method simulating diffuse light interreflection between surfaces.

  • Form Factor: Fraction of light leaving surface A that reaches surface B; depends on geometry and orientation.

  • Progressive Refinement: Iteratively solves radiosity equations; starts with direct lighting, then adds bounced light.

  • Application: Used for realistic lighting in architectural visualization, but computationally expensive; not real-time.

  [!TIP] Radiosity handles diffuse interreflection well; ray tracing handles specular effects and shadows better.

III. SIMULATION, INTERACTION & BEHAVIOR

Navigation and Manipulation

• Interpolation: Estimating values between known points.

  • Types:

    • Linear interpolation (lerp): $$\displaystyle p = (1-t)p_0 + t p_1 $$; simple but not smooth.

    • Spline interpolation (cubic spline, B-spline): Smooth curves through control points; used for camera paths, object motion.

• Translation: Moving objects or user in virtual space.

  • User movement: Teleportation, continuous locomotion (walking, flying).

  • Object manipulation: Grabbing, moving, scaling via controllers.

  • Navigation metaphors: "Walk-through", "fly-over", "examination" (rotate/zoom object).

  [!TIP] In VR, locomotion must avoid motion sickness; teleportation is often preferred over artificial walking.

Physics and Dynamics

• Behaviour-based simulation: Uses rules or AI to simulate appearance of behavior; not physically accurate. Example: flocking (boids), finite state machines for NPCs.

• Physical-based simulation: Models real-world physics using Newtonian mechanics; includes rigid body dynamics (collision response, forces), soft body dynamics, fluid simulation.

• Differentiate: Behaviour-based is faster, used for non-critical elements; physical-based is computationally heavy but realistic, essential for object interaction.

System Integrity

• Collision Detection: Determining if objects intersect in virtual environment.

  • Algorithms:

    • Bounding Volume Hierarchies (BVH): Enclose objects in simple shapes (spheres, AABBs) and organize in tree; test volumes first.

    • Spatial partitioning: Divide space (grid, octree, BSP tree) to reduce checks.

  • Purpose: Provide feedback (haptic/audio), trigger physics responses, prevent interpenetration.

  [!TIP] Collision detection is often two-phase: broad phase (quick culling) and narrow phase (precise mesh intersection).

IV. DEVELOPMENT TOOLS & ENVIRONMENTS

VR Software Frameworks

• Features of VR Toolkits (e.g., Unity XR, Unreal Engine, OpenVR, WebXR):

  • Cross-platform support: Develop once, deploy to multiple HMDs (Oculus, HTC Vive, etc.).

  • Input device integration: APIs for controllers, trackers, gestures.

  • Rendering APIs: Abstraction over OpenGL, DirectX, Vulkan; handle stereoscopic rendering.

  • Physics engines: Built-in or integrated (PhysX, Bullet) for collision and dynamics.

  • Asset pipelines: Import 3D models, animations, textures; optimize for VR.

  [!TIP] Choose toolkit based on target platform, performance needs, and team expertise.

V. HARDWARE & PERIPHERALS

Output Devices (Auditory)

• Acoustic Hardware in VR Systems:

  • 3D Audio: Sound that appears to come from specific locations in 3D space.

  • HRTF (Head-Related Transfer Function): Filters modeling how sound is transformed by human anatomy; used for binaural rendering over headphones.

  • Speaker arrays: Multiple speakers around user for spatial sound without headphones.

  • Binaural rendering: Computes separate audio signals for left/right ears to simulate direction.

  • Spatial sound: Essential for immersion; helps locate objects, enhances presence.

  [!TIP] HRTF is personal; generic HRTF may not work perfectly for all users.

VI. APPLICATIONS & DOMAINS

Industry Use Cases

• Application of VR in Digital Entertainment:

  • Gaming: Immersive gameplay, new genres (rhythm, horror).

  • Cinematic experiences: VR films, 360° videos, interactive storytelling.

  • Theme parks: VR rides and attractions (e.g., The Void).

  • Interactive narratives: Users influence story through actions; branching narratives.

  [!TIP] Key challenges: motion sickness, high production costs, user adoption.