Augmented and Virtual Reality (AL-703 (B)) Unit 5 Notes

UNIT 5: AUGMENTED AND VIRTUAL REALITY

I. FOUNDATIONS OF VR AND AR

A. Virtual Reality (VR)

Definition and Core Characteristics
- VR is a computer-generated simulation of a three-dimensional environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment (e.g., HMD, gloves).
- Core Characteristics:
  - Immersion: The degree to which the user feels present in the virtual environment. Achieved through wide FOV, stereoscopy, and tracking.
  - Interaction: The ability to manipulate objects or navigate the virtual world in real-time.
  - Imagination: The system's capability to model and render realistic or fantastical worlds beyond physical constraints.
Key Features: Immersion, Interaction, Imagination
- Immersion: Sensory isolation from the real world (e.g., HMD blocks real sight/audio). Measured by presence.
- Interaction: Natural User Interfaces (NUI) like hand tracking, motion controllers, or haptic feedback.
- Imagination: Enables simulations for training, design, entertainment, or therapy.

B. Augmented Reality (AR) vs. Virtual Reality (VR)

Fundamental Differences
- VR: Replaces the real world with a completely synthetic one. User is immersed in virtuality.
- AR: Overlays digital information (graphics, sound, haptic) onto the real world. User's view of reality is augmented.
- Key Distinction: VR creates a new reality; AR enhances the existing one.
Comparative Examples and Diagrams
- VR Example: Flight simulator in a sealed cockpit.
- AR Example: Mobile app showing navigation arrows over a live camera view of a street.
- Spectrum: Reality-Virtuality Continuum.
  
  [!TIP] Exam often asks for a diagram showing the Mixed Reality Spectrum from real environment to virtual environment, with AR and Augmented Virtuality (AV) in between.

C. Mixed Reality (MR) Spectrum and Concepts

Mixed Reality (MR) is the merging of real and virtual worlds to produce new environments where physical and digital objects co-exist and interact in real-time.
Spectrum:
1. Real Environment: No virtual content.
2. Augmented Reality (AR): Real world dominant, virtual overlay.
3. Augmented Virtuality (AV): Virtual world dominant, real-world elements embedded.
4. Virtual Environment (VR): Completely synthetic.
Core Concept: Spatial Mapping – understanding the geometry of the real world to anchor virtual objects convincingly.

II. COMPUTER GRAPHICS AND RENDERING FOR VR/AR

A. Clipping Algorithms

Parametric Line Clipping (Liang-Barsky Algorithm)
- Uses parametric equation of the line: $$\displaystyle x = x_1 + t(x_2 - x_1) $$, $$\displaystyle y = y_1 + t(y_2 - y_1) $$, where $0 \le t \le 1$.
- Clips against each window edge ($$\displaystyle x_{min}, x_{max}, y_{min}, y_{max} $$) by solving inequalities for $t$.
- Defines parameters: $$\displaystyle p_1 = -\Delta x $$, $$\displaystyle p_2 = \Delta x $$, $$\displaystyle p_3 = -\Delta y $$, $$\displaystyle p_4 = \Delta y $$ and $$\displaystyle q_1 = x_1 - x_{min} $$, etc.
- Algorithm Steps:
  1. Initialize $$\displaystyle t_E = 0 $$, $$\displaystyle t_L = 1 $$.
  2. For each edge $i$ (1 to 4):
    - If $$\displaystyle p_i < 0 $$: $$\displaystyle t = q_i/p_i $$; if $$\displaystyle t > t_E $$, set $$\displaystyle t_E = t $$ (potential entering point).
    - If $$\displaystyle p_i > 0 $$: $$\displaystyle t = q_i/p_i $$; if $$\displaystyle t < t_L $$, set $$\displaystyle t_L = t $$ (potential leaving point).
    - If $$\displaystyle p_i = 0 $$ and $$\displaystyle q_i < 0 $$, line is parallel and outside → reject.
  3. If $$\displaystyle t_E > t_L $$, line is rejected.
  4. Accepted line segment endpoints: $$\displaystyle (x(t_E), y(t_E)) $$ and $$\displaystyle (x(t_L), y(t_L)) $$.
Step-by-Step Application to Example
- Problem: Clip line $A(10,10)$ to $B(70,40)$ against window $(20,20)$ to $(40,50)$.
- Solution:
  - $$\displaystyle \Delta x = 60 $$, $$\displaystyle \Delta y = 30 $$.
  - Left edge ($$\displaystyle x=x_{min}=20 $$): $$\displaystyle p_1 = -60 $$, $$\displaystyle q_1 = 10-20 = -10 $$. $$\displaystyle t = (-10)/(-60) = 1/6 \approx 0.167 $$. Since $$\displaystyle p_1<0 $$, $$\displaystyle t_E = \max(0, 0.167) = 0.167 $$.
  - Right edge ($$\displaystyle x=x_{max}=40 $$): $$\displaystyle p_2 = 60 $$, $$\displaystyle q_2 = 10-40 = -30 $$. $$\displaystyle t = (-30)/60 = -0.5 $$. Since $$\displaystyle p_2>0 $$, $$\displaystyle t_L = \min(1, -0.5) = -0.5 $$? Wait, $$\displaystyle q_2 $$ negative means line is left of window? Let's recalc properly.
  - Correct Approach:
    - Left: $$\displaystyle p_1 = -\Delta x = -60 $$, $$\displaystyle q_1 = x_1 - x_{min} = 10-20 = -10 $$. $$\displaystyle t = q_1/p_1 = (-10)/(-60) = 1/6 $$. $$\displaystyle t_E = \max(0, 1/6) = 1/6 $$.
    - Right: $$\displaystyle p_2 = \Delta x = 60 $$, $$\displaystyle q_2 = x_1 - x_{max} = 10-40 = -30 $$. $$\displaystyle t = q_2/p_2 = -30/60 = -0.5 $$. Since $$\displaystyle p_2>0 $$, $$\displaystyle t_L = \min(1, -0.5) = -0.5 $$. But $$\displaystyle t_L < t_E $$ already? Actually, if $$\displaystyle t_L $$ becomes negative, it means the line starts inside? Let's do systematically.
  - Full Calculation:
    - $$\displaystyle x_1=10, y_1=10; x_2=70, y_2=40; x_{min}=20, x_{max}=40; y_{min}=20, y_{max}=50 $$.
    - $$\displaystyle \Delta x=60, \Delta y=30 $$.
    - Left ($x \ge 20$): $$\displaystyle p_1 = -60 $$, $$\displaystyle q_1 = 10-20 = -10 $$. $$\displaystyle t_1 = q_1/p_1 = 1/6 \approx 0.167 $$. $$\displaystyle t_E = \max(0, 0.167) = 0.167 $$.
    - Right ($x \le 40$): $$\displaystyle p_2 = 60 $$, $$\displaystyle q_2 = 10-40 = -30 $$. $$\displaystyle t_2 = q_2/p_2 = -0.5 $$. Since $$\displaystyle p_2>0 $$, $$\displaystyle t_L = \min(1, -0.5) = -0.5 $$.
    - Bottom ($y \ge 20$): $$\displaystyle p_3 = -30 $$, $$\displaystyle q_3 = 10-20 = -10 $$. $$\displaystyle t_3 = q_3/p_3 = 1/3 \approx 0.333 $$. $$\displaystyle t_E = \max(0.167, 0.333) = 0.333 $$.
    - Top ($y \le 50$): $$\displaystyle p_4 = 30 $$, $$\displaystyle q_4 = 10-50 = -40 $$. $$\displaystyle t_4 = q_4/p_4 = -4/3 \approx -1.333 $$. $$\displaystyle t_L = \min(-0.5, -1.333) = -1.333 $$.
    - Now $$\displaystyle t_E = 0.333 $$, $$\displaystyle t_L = -1.333 $$. Since $$\displaystyle t_E > t_L $$, line is rejected? But visually, line from (10,10) to (70,40) should cross window. Let's check window corners: (20,20) to (40,50). Line equation: $$\displaystyle y = 10 + 0.5(x-10) $$. At x=20, y=15 (<20). At x=40, y=25 (within y 20-50). So it enters from bottom? Actually, window y_min=20. At x where y=20: 20=10+0.5(x-10) => 10=0.5(x-10) => x=30. So entry at (30,20). Exit at x=40, y=25. So it should be clipped to segment from (30,20) to (40,25).
    - Mistake in q calculations: For right edge, condition is $$\displaystyle x \le x_{max} $$, so $$\displaystyle x_1 + t\Delta x \le x_{max} $$ => $$\displaystyle t \le (x_{max} - x_1)/\Delta x $$. So $$\displaystyle p_2 = \Delta x $$, $$\displaystyle q_2 = x_{max} - x_1 $$. Similarly for others.
    - Standard Liang-Barsky:
      - $$\displaystyle p_1 = -\Delta x $$, $$\displaystyle q_1 = x_1 - x_{min} $$ (for $$\displaystyle x \ge x_{min} $$)
      - $$\displaystyle p_2 = \Delta x $$, $$\displaystyle q_2 = x_{max} - x_1 $$ (for $$\displaystyle x \le x_{max} $$)
      - $$\displaystyle p_3 = -\Delta y $$, $$\displaystyle q_3 = y_1 - y_{min} $$ (for $$\displaystyle y \ge y_{min} $$)
      - $$\displaystyle p_4 = \Delta y $$, $$\displaystyle q_4 = y_{max} - y_1 $$ (for $$\displaystyle y \le y_{max} $$)
    - Recalculate:
      - $$\displaystyle p_1 = -60 $$, $$\displaystyle q_1 = 10-20 = -10 $$, $$\displaystyle t_1 = (-10)/(-60)=1/6=0.167 $$, $$\displaystyle t_E=0.167 $$.
      - $$\displaystyle p_2 = 60 $$, $$\displaystyle q_2 = 40-10=30 $$, $$\displaystyle t_2=30/60=0.5 $$, $$\displaystyle t_L=\min(1,0.5)=0.5 $$.
      - $$\displaystyle p_3 = -30 $$, $$\displaystyle q_3 = 10-20=-10 $$, $$\displaystyle t_3=(-10)/(-30)=1/3=0.333 $$, $$\displaystyle t_E=\max(0.167,0.333)=0.333 $$.
      - $$\displaystyle p_4 = 30 $$, $$\displaystyle q_4 = 50-10=40 $$, $$\displaystyle t_4=40/30=4/3=1.333 $$, $$\displaystyle t_L=\min(0.5,1.333)=0.5 $$.
    - Now $$\displaystyle t_E=0.333 $$, $$\displaystyle t_L=0.5 $$. Since $$\displaystyle t_E \le t_L $$, line is accepted.
    - Clipped endpoints:
      - Start: $$\displaystyle t_E=1/3 $$: $$\displaystyle x=10+(1/3)*60=30 $$, $$\displaystyle y=10+(1/3)*30=20 $$ → $(30,20)$.
      - End: $$\displaystyle t_L=0.5 $$: $$\displaystyle x=10+0.5*60=40 $$, $$\displaystyle y=10+0.5*30=25 $$ → $(40,25)$.
    \boxed{\text{Clipped segment: } (30,20) \text{ to } (40,25)}

B. Interpolation Techniques

Linear Interpolation
- Definition: Estimating values between two known data points using a straight line. Formula: $$\displaystyle y = y_1 + (x - x_1) \frac{y_2 - y_1}{x_2 - x_1} $$ or parametric: $$\displaystyle P(t) = P_1 + t(P_2 - P_1), \; t \in [0,1] $$.
- Use Cases: Positioning objects along a path, simple color blending, basic animation keyframing.
Nonlinear Interpolation
- Types & Applications:
  - Polynomial Interpolation (e.g., Cubic, Bézier): Smooth curves for animation paths, font design.
  - Spherical Linear Interpolation (Slerp): Interpolating between two rotations (quaternions) on a 4D sphere. Crucial for smooth camera/object orientation in VR/AR.
  - Hermite Interpolation: Uses derivatives (tangents) for smooth transitions, common in keyframe animation.

C. Shading Algorithms

Flat Shading
- Computes lighting once per polygon (using face normal). All pixels in polygon have same color.
- Pros: Fast, computationally cheap.
- Cons: Faceted appearance, poor for curved surfaces.
Gouraud Shading
- Computes lighting at each vertex of polygon. Colors are then linearly interpolated across the polygon surface.
- Pros: Smooth shading, good performance.
- Cons: Highlights may be distorted or missing (specular highlights can be washed out).
Phong Shading
- Computes lighting at each pixel. Normal vectors are interpolated from vertex normals, then lighting equation applied per pixel.
- Pros: Most realistic, accurate specular highlights, smooth surfaces.
- Cons: Computationally expensive.
Comparative Analysis

| Algorithm | Computation | Visual Quality | Use Case | | :--- | :--- | :--- | :--- | | Flat | Per face | Faceted, low | Low-poly art, debugging | | Gouraud | Per vertex | Smooth, but highlights may distort | Real-time apps (games) where performance is critical | | Phong | Per pixel | Very smooth, accurate highlights | High-end rendering, VR where realism is key |

III. PHYSICS AND MOTION SIMULATION

A. Kinematic Modeling

Vector Functions for Position and Velocity
- Position vector: $$\displaystyle \vec{r}(t) = \langle x(t), y(t), z(t) \rangle $$.
- Velocity vector: $$\displaystyle \vec{v}(t) = \frac{d\vec{r}(t)}{dt} $$.
- Acceleration: $$\displaystyle \vec{a}(t) = \frac{d\vec{v}(t)}{dt} $$.
Example Problem: Worm on Rotating Wheel
- Problem: Wheel in y-z plane, center at origin, rotating at $$\displaystyle \omega = 1 $$ rad/s. Spoke along positive y-axis at $$\displaystyle t=0 $$. Worm crawls outward along spoke at speed $$\displaystyle v = 1 $$ unit/s. Find $\vec{r}(t)$.
- Solution:
  - At time $t$, worm's distance from center along spoke: $$\displaystyle s = v \cdot t = t $$.
  - The spoke has rotated by angle $$\displaystyle \theta = \omega t = t $$ radians.
  - In y-z plane, position relative to rotated spoke: along y-axis direction (now at angle $\theta$ from original y-axis).
  - Original y-axis unit vector is $\hat{j}$. After rotation by $\theta$ about x-axis: $$\displaystyle \hat{j}' = (0, \cos\theta, \sin\theta) $$.
  - So, $$\displaystyle \vec{r}(t) = s \cdot \hat{j}' = t \cdot (0, \cos t, \sin t) $$.
  \boxed{\vec{r}(t) = \langle 0,; t\cos t,; t\sin t \rangle}

B. Flight Dynamics Simulation

Aircraft Motion Parameters
- 6 Degrees of Freedom (6-DOF): 3 translational (x, y, z - position) and 3 rotational (roll, pitch, yaw - orientation).
- Key forces: Lift, Drag, Thrust, Gravity.
- Key moments: Rolling, Pitching, Yawing moments.
Techniques for Realistic Flight in VR
- Aerodynamic Models: Use lookup tables or computational fluid dynamics (CFD) approximations for force/moment coefficients based on angle of attack, airspeed, etc.
- Flight Control Systems: Simulate autopilot or control surface deflections (ailerons, elevators, rudder).
- Environmental Effects: Wind, turbulence, stalls, spins.
- Simplified Arcade Model: For non-simulator VR, use simplified physics (e.g., constant turn rate, limited stall).

C. Collision Detection

Generic VR System Approach
- Broad Phase: Quickly eliminate non-colliding object pairs using bounding volumes (AABB - Axis-Aligned Bounding Box, spheres). Use spatial partitioning (octrees, BSP trees) or sweep-and-prune.
- Narrow Phase: For potential pairs, perform precise intersection test using actual geometry (triangle-triangle intersection, GJK algorithm for convex shapes).
- Response: Upon detection, compute penetration depth and contact points to apply physical response (bounce, stop, sound).
Collision Detection in VRML (Collision Node)
- Purpose: Defines a collidable object and its properties for the VRML browser's collision system.
- Fields:
  - collide (SFBool): TRUE/FALSE to enable/disable collision.
  - proxy (SFNode): A simplified geometry (e.g., sphere, box) used for collision instead of the complex shape node. Crucial for performance.
  - surface (SFNode): Specifies a Material or Texture for the proxy (optional, for debugging).
- Example:
```
Collision {

  collide TRUE

  proxy Sphere { radius 2.0 }

  children [

    Transform { translation 0 0 5 children [ Shape { ... complex geometry ... } ] }

  ]

}
```
  Here, the complex geometry is for rendering, but a sphere of radius 2 is used for collision checks.

IV. INTERACTION IN VIRTUAL ENVIRONMENTS

A. Models of Interaction

Types of Interaction Models
- Go-Go Technique: Virtual hand extends from user's real hand proportionally, then "snaps" to a far object when a threshold is reached. Allows reaching distant objects.
- Ray-Casting: User points a virtual ray (from hand or controller). Object intersected by ray is selected. Often combined with a menu at ray origin.
- World-in-Miniature (WIM): A small-scale 3D map of the entire virtual world is held in the user's hand. Manipulating objects in the miniature manipulates them in the full-scale world.
- Hand-Direct Manipulation: Virtual hand position directly maps to real hand tracker position (1:1). Intuitive but limited by physical tracking space.
Examples and Applications
- Go-Go: Useful in large-scale VR where physical tracking space is small.
- Ray-Casting: Common in menu selection, object picking from a distance (e.g., VR desktop interfaces).
- WIM: Navigation and object manipulation in large virtual environments (e.g., architectural walkthroughs).

B. VRML Nodes for Interaction and Scene Structure

Anchor Node
- Purpose: Creates a hyperlink to another VRML file or a web URL. When user clicks on its children geometry, the browser loads the specified url.
- Example:
```
Anchor {

  url "room.wrl"

  children [

    Transform { translation 0 0 0 children [ Shape { ... } ] }

  ]

}
```
Collision Node
- Purpose: As detailed in III.C.2. Defines collision properties for its children.
- Example: See above.
Group Node
- Purpose: A generic container node that groups multiple children under a single transform. Its children are rendered/transformed as a unit. Base type for Transform, Switch, LOD, etc..
- Example:
```
Group {

  children [

    Shape { ... },

    Transform { translation 5 0 0 children [ Shape { ... } ] }

  ]

}
```
Shape Node
- Purpose: The fundamental renderable node. Combines geometry (what to draw) and appearance (how to draw it).
- Fields: geometry (e.g., Box, Sphere, IndexedFaceSet) and appearance (e.g., Material, Texture).
- Example:
```
Shape {

  appearance Material { diffuseColor 1 0 0 }

  geometry Box { size 2 2 2 }

}
```

V. HARDWARE TECHNOLOGIES

A. VR Sensor Hardware

Motion Tracking Sensors
- Optical: Uses external cameras and infrared LEDs on HMD/controllers. High accuracy, limited by line-of-sight. (e.g., Oculus Constellation, SteamVR Lighthouse).
- Magnetic: Uses a stationary emitter and sensors on tracked objects. Prone to metal distortion, lower accuracy.
- Inertial: Uses accelerometers, gyroscopes, magnetometers (IMU). No external hardware, but suffers from drift over time. Often fused with optical for 6-DOF.
Input Devices
- Gloves: Track finger flexion and hand position. Provide gesture input and potential haptic feedback (e.g., data gloves).
- Controllers: Hand-held devices with buttons, triggers, analog sticks, and tracking. Provide point-and-click, grasp simulation.

B. Acoustic Hardware

3D Audio Systems
- Simulates sound sources in 3D space. Sound changes based on listener's head position/orientation relative to source.
- Key Technique: Head-Related Transfer Function (HRTF). Filters that model how sound is modified by human head, torso, and ears before reaching eardrums. Personalized HRTFs yield better localization.
Spatial Sound Hardware and HRTF
- Hardware: Standard stereo headphones are sufficient. Binaural rendering is done in software.
- Implementation: For each sound source, apply an HRTF filter (based on source azimuth/elevation) to the audio signal before presenting to left/right ear.
- Challenges: Individual HRTF variation, distance attenuation, occlusion/obstruction effects.

C. Augmented Reality Display Technologies

Optical See-Through Displays
- Principle: User looks through a semi-transparent optical element (e.g., beam splitter, holographic optical element) onto which virtual images are projected.
- Pros: See real world directly, no video latency, high resolution for real world.
- Cons: Difficult to achieve high brightness/luminance for virtual objects (washed out in bright light), limited FOV, registration errors (virtual objects may not align perfectly with real).
- Examples: Microsoft HoloLens, Magic Leap.
Video See-Through Displays
- Principle: Cameras mounted on HMD capture real world. Video feed is combined with rendered virtual graphics and displayed on an opaque display (like VR HMD).
- Pros: Full control over final image (can occlude real objects, apply effects), easier registration (both real and virtual go through same display pipeline), high brightness for virtual objects.
- Cons: Video latency can cause motion sickness, limited camera resolution/FOV, heavier.
- Examples: Most smartphone-based AR (ARKit, ARCore), early AR systems.
Head-Mounted Displays (HMDs) for AR
- Combine see-through optics (optical or video) with tracking sensors in a wearable unit.
- Key Metrics: Field of View (FOV) (critical for immersion and utility), Resolution, Refresh Rate, Latency, Weight/Ergonomics.
Diagrams and Comparative Analysis

[!TIP] Draw a simple diagram for each type.
- Optical See-Through: Show user eye -> beam splitter -> real world path and projector -> beam splitter -> eye path.
- Video See-Through: Show cameras -> video pipeline -> display -> eye, with real world viewed by cameras.
Comparison Table:

| Aspect | Optical See-Through | Video See-Through | | :--- | :--- | :--- | | Real World View | Direct (no processing) | Via cameras (processed) | | Latency (Virtual) | Low (projector) | Higher (camera->render->display) | | Brightness (Virtual) | Low (compete with real light) | High (controlled display) | | Occlusion | Difficult (virtual always on top) | Easy (can mask real objects) | | Registration | Challenging (optical distortion) | Easier (same pipeline) | | Example | HoloLens | iPhone AR, Meta Quest Pro (pass-through) |

VI. DEVELOPMENT TOOLS AND LANGUAGES

A. Virtual Reality Modeling Language (VRML)

Overview and Purpose
- VRML (Virtual Reality Modeling Language, now ISO/IEC 14772) is a file format for representing 3D interactive vector graphics, designed for the World Wide Web.
- Purpose: To create and share 3D worlds and objects that can be navigated and interacted with in real-time over the internet.
Scene Graph Structure
- VRML uses a hierarchical scene graph. The root is a Scene node containing Group nodes.
- Transform Node: Most important for hierarchy. It applies translation, rotation, and scaling to its children. Children inherit the parent's transformation.
- Example Hierarchy: Transform (room position) -> Group (furniture group) -> Transform (table position) -> Shape (table top).
Key Nodes and Their Functions (See IV.B for Anchor, Collision, Group, Shape examples)
- Transform: Position/rotate/scale children.
- Group: Simple container.
- Switch: Shows only one child based on whichChoice field.
- LOD (Level of Detail): Switches between different complexity geometries based on viewer distance.
- Inline: Includes external VRML files (modularity).
- Script: Contains JavaScript or VRMLScript for custom behavior/logic.
- TimeSensor: Generates time events to drive animations.
- PositionInterpolator/ OrientationInterpolator: Animates Transform fields.

B. VR Toolkits and Frameworks

Popular Toolkits
- Unity with XR Interaction Toolkit: Most popular, C# scripting, vast asset store, supports all major HMDs.
- Unreal Engine: High-fidelity graphics, Blueprint visual scripting, C++.
- OpenVR/SteamVR SDK: Low-level API for HTC Vive/Valve Index. More control, steeper learning curve.
- WebXR: Standard for AR/VR in web browsers (JavaScript).
- A-Frame: Web framework for VR/AR using HTML-like syntax.
Selection Criteria and Use Cases
- Target Platform: Mobile AR (ARKit/ARCore, Unity), PC VR (SteamVR, Unreal), Web (WebXR, A-Frame).
- Graphics Fidelity: Unreal for high-end, Unity for balanced.
- Team Skills: C# (Unity) vs C++ (Unreal) vs JavaScript (WebXR).
- Prototyping Speed: Unity/A-Frame very fast.
- Specific Features: Need advanced physics? (Unity/Unreal). Need web deployment? (WebXR).

VII. AUGMENTED REALITY SPECIFIC TECHNOLOGIES

A. Tracking Methods for AR

Marker-Based Tracking
- Uses fiducial markers (e.g., QR codes, ARToolKit patterns) placed in the environment.
- Process: Camera detects marker, computes its 2D pose (position, orientation) in image. Using known marker size, solves for 6-DOF pose of the camera relative to the marker.
- Pros: Very accurate, robust, simple, low computational cost.
- Cons: Requires placing markers, unnatural appearance, limited to marker locations.
Marker-Less Tracking: Techniques and Examples
- SLAM (Simultaneous Localization and Mapping): The core technology for marker-less AR.
  - Process: Camera observes environment, extracts visual features (corners, edges). Matches features between frames to estimate camera motion (localization) and builds a sparse 3D map of feature points (mapping) simultaneously.
  - Examples: Apple ARKit (ARKit uses a form of SLAM with feature points and plane detection), Google ARCore.
- Feature Matching: Extracting distinctive keypoints (SIFT, SURF, ORB) from current frame and matching to a pre-built database of known scenes/objects (e.g., for image-based AR).
- Inertial Tracking (IMU): Uses accelerometer/gyro to estimate device motion. Often fused with visual SLAM (Visual-Inertial Odometry - VIO) for smooth, accurate, drift-reduced tracking. (e.g., ARKit's VIO).

B. Human Perception and Senses in AR

Visual Integration and Overlay
- Goal: Virtual objects must appear anchored to the real world.
- Key Factors:
  - Registration: Accurate alignment of virtual with real (depends on tracking accuracy).
  - Occlusion Handling: Virtual objects should be hidden by real objects in front. Requires depth sensing (stereo cameras, LiDAR) or plane estimation.
  - Lighting Estimation: Virtual object lighting should match real-world lighting (direction, intensity, color) for realism.
  - Scale & Perspective: Correct size and perspective based on camera pose.
Auditory Cues and Spatial Audio
- Use HRTF-based 3D audio to place sound sources in the environment.
- Applications: Guide user's attention ("look here"), indicate proximity/direction of virtual objects/events, enhance immersion (e.g., virtual character speaking from its location).
- Challenges: Same as VR (HRTF personalization), but also must mix with real-world audio (pass-through audio for video see-through AR).
Multisensory Design Considerations
- Haptic Feedback: Use controller vibrations or wearable haptics to simulate touch when interacting with virtual objects (e.g., button press, object grasp).
- Balance & Vestibular: Avoid rapid, conflicting visual-vestibular cues to prevent simulator sickness. Keep virtual camera motion smooth and matched to user's head movement.
- Cognitive Load: Don't overload user with too many simultaneous sensory inputs. Design for attention.
- Example: An AR maintenance app might:
  - Visual: Overlay arrows and labels on machine parts.
  - Auditory: Play a "click" sound when a virtual bolt is "tightened".
  - Haptic: Controller vibrates when user's hand is near a virtual hot surface.

Account

Augmented and Virtual Reality (AL-703 (B)) - Unit 5 Short Notes

UNIT 5: AUGMENTED AND VIRTUAL REALITY

I. FOUNDATIONS OF VR AND AR

A. Virtual Reality (VR)

B. Augmented Reality (AR) vs. Virtual Reality (VR)

C. Mixed Reality (MR) Spectrum and Concepts

II. COMPUTER GRAPHICS AND RENDERING FOR VR/AR

A. Clipping Algorithms

B. Interpolation Techniques

C. Shading Algorithms

III. PHYSICS AND MOTION SIMULATION

A. Kinematic Modeling

B. Flight Dynamics Simulation

C. Collision Detection

IV. INTERACTION IN VIRTUAL ENVIRONMENTS

A. Models of Interaction

B. VRML Nodes for Interaction and Scene Structure

V. HARDWARE TECHNOLOGIES

A. VR Sensor Hardware

B. Acoustic Hardware

C. Augmented Reality Display Technologies

VI. DEVELOPMENT TOOLS AND LANGUAGES

A. Virtual Reality Modeling Language (VRML)

B. VR Toolkits and Frameworks

VII. AUGMENTED REALITY SPECIFIC TECHNOLOGIES

A. Tracking Methods for AR

B. Human Perception and Senses in AR