XR Introduction

Posted on Edited on In

VR, AR introduction.

Virtual Reality

Virtual Reality (VR) is the technology to bring virtual digital content to users. It has the following 3 features.

  • Immersion: several feedback to users
  • Interaction: support several natural real-time interaction between users and virtual content
  • Imagination: immersive, real

VR includes 3 parts: users, computing machine containing the virtual content, human-machine interfaces.

Virtual Content Creation

2 Ways to create the virtual content

Similar to the computer games asset development.

  • Steps
    • Modeling
      • indicate the objects with math model
    • Rendering
      • determine the color of each pixel according to the model, light, shader, texture and etc.
      • requires real-time (at least 30 frames/s)
    • Animation
      • calculate each point’s motion in the math model
  • Commercial tools
    • Content Creation
      • 3DS Max, Maya, PhotoShop
    • Basic Graphics Library
      • OpenGL, D3D
    • Graphics System Development Library (3D Graphics Engines)
      • OSG (Open Scene Graph), OGRE (Object-oriented Graphic Rendering Engine)
        • Based on programming totally
        • Open-source
    • VR Engines
      • Vega, World Tool Kit, VRPlatform, Virtools
        • Professional VR Engine
        • Can meet the accuracy requirements of real environment data
        • Can build large scale environment map
      • Unity3D, Unreal
        • Game engines
    • Web Standard Language
      • VRML (Virtual Reality Modeling Language)/X3D, WebGL/Html5

Use real images to create the virtual environment. Image->Object->Image. The first step is 2D -> 3D reconstruction work, and the second step is 3D -> 2D rendering work.

  • IBMR Technology (Imaged Based Modeling and Rendering)
    • Jump the middle stage of Image->Object->Image
      • Discrete viewpoint sampling
      • Modeling the scene based on sampling images
      • Resampling at any viewpoint
  • IBMR - Panorama
    • Steps
      • Take Images and pre-process
      • Mapping to the cylinder coordinate
      • Image registration based on image features
      • Image blending
      • Project the image to a cylinder object
      • Generate the image according to the viewpoint
    • Commercial Tools
      • OpenCV, Panorama Tools, Hugin
      • Krpano, UtoVR
  • Other IBMR (Support humans wandering in the scene)
    • Concentric Mosaics
      • Use several camera to capture images at one position
    • Light Field
      • Use camera array to capture images
    • Volumetric Video

Human-machine Interfaces

The human-machine interfaces include

  • sensing interfaces (how computing machine show the virtual content to humans)
    • Visual form-factor
      • Helmet
      • Glasses
      • 180 big Screen
      • Cave
    • Audio
    • Touch
    • …
  • Tracking interfaces (how humans provides the input to computing machine)
    • SLAM, Eye Tracking, Hand Tracking, Body Tracking, Speech Recognition

Interaction interfaces is related to lots of subjects. Shown as below.

Sensing Interfaces

Basic categories of visual sensing devices

  • Head Mounted
  • Hand Held
  • Spatial Projection

Some visual sensing devices with immersion

  • HMD
    • Left eye, right eye screens
    • Computing system
    • Rotation sensors (gyro)
    • Commercial Tools
      • Use with phone: Gear VR, Google Cardboard
      • Use with PC: Oculus Rift, HTC Vive, PS VR
      • Standalone: Oculus Go
  • Stereo glasses (3D glasses)
    • Left eye, right eye filters
    • Categories
      • Time-sharing display
      • Color difference stereo display
      • Polarized stereo display
  • Big Stereo Screen (needs to work with stereo glasses together)
    • Fuse the overlapped area of several projectors
    • Big LED screen
    • Sphere screen
    • Cave
  • Auto stereo display, holographic
    • No need to wear devices and no tracking devices required
    • Autostereoscopic display
    • Volumetric 3D display
    • Integral Imaging 3D display
    • Hologram

There are also other types of Sensing Interfaces

  • Hearing sensing interface
    • Virtual surround sound technology
  • Tactile sensing interface

Tracking interfaces

  • Pose tracking interfaces
    • Skin/muscle: MYO Wristband
    • Optical
      • Based on stereo vision principle. Put several cameras to capture the human with trackers
    • IMU
  • Voice recognition and interaction
  • Brain-computer interface

AR / MR

Unlike VR which only renders virtual content, AR/MR renders virtual content on top of physical world. Depending on how to show the physical world, it could be categories into optical see-through (AR) and video see-through (MR).

Elements of immersion

  • Placing: place a virtual object in the real world position and it always stick to that position
  • Scale: the scale of the virtual object changes when you move nearer or faster to it
  • Occlusion: a virtual object should be blocked if another virtual object or a real object is between it and human’s view point
  • Lighting: a virtual object should shows different effects when the environmental light changes
  • Solid: the virtual object shouldn’t overlap with other objects or float in the air
  • Context awareness: knows every single real object in the environment and have proper relationship with those objects.

Tracking in AR

Motion tracking of devices

Motion tracking in AR requires lots of hardware module.

  • Motion Tracking
    • Accelerometer
    • Gyroscope
    • Camera
  • Location-based AR
    • Magnetometer
    • GPS

There are two main categories methods for AR motion tracking.

~ Outside-in tracking Inside-out tracking
Definition AR device uses external cameras or sensors fixed in the environment to detect motion and tracking positioning AR device uses internal cameras or sensors to detect motion and tracking positioning
Pros More accurate More portable
Cons Less portable Heavier, Heater, More energy consume
Product HTC Vive (VR) Microsoft HoloLens, Google ARCore enabled phones

The inside-out tracking method is related to the SLAM technology. Concurrent odometry and mapping (COM) is the motion tracking process for ARCore, which tracks the smartphone’s location in relation to its surrounding world.

Environmental Understanding

Unlike VR, AR/MR needs to fuse the virtual content with the physical world to improve the immersion. Therefore, it requires environmental understanding to recognize objects in real environment, and placing and orienting digital objects accordingly.

The most common environment understanding is to reconstruct the environment surface and fine planes. This requires to reconstruct the 3D points of the environment, cluster them and find a plane.

The virtual content also needs to be anchored into the exact position in space over time. Anchors allow the underlying system to correct that drifting error.

Commercial Products

AR/MR include hand-held AR experience and standalone AR experience. Below is some commercial products of AR.

Hardware OS Software Framework Company
Mobile Phone Android ARCore Google
Mobile Phone IOS ARKit Apple
HoloLens Windows Windows Holographic Microsoft
Quest Android - Meta
Vision Pro Apple ARKit Apple

Below is the structure of Microsoft Mixed Reality

Constraints

  • Lack standard and proper user interface design principle
  • Hardware & Devices
    • Size
    • Power
    • Heat
  • Current 3D design pipeline need to be integrated
  • Computer vision limitations
    • Occlusion

Optical Imaging

  • Lens based on raster
  • Lens based on waveguide

Reference

  1. 虚拟现实技术及应用(2020年春)，中国传媒大学，学堂在线
  2. Introduction to Augmented Reality and ARCore, Google Daydream Impact, Coursera