Towards Simulating Augmented Reality across Different Form Factors Using a Virtual Reality Device with Hand Tracking
Timeline
1 year, Feb 2021 - May 2022
Role & Responsibilities
Deliverables
Abstract
Making simulated AR accessible so developers can test their application across different devices and environmental conditions
Augmented reality (AR) experiences can vary significantly with the user’s environment, which poses a unique challenge for application development and testing. In this work, we explore virtual reality (VR) as a way to test AR experiences in a more programmable simulated environment. Our work proposes a simulation method that relies solely on a commercially available head-mounted VR system with hand tracking to simulate AR devices across different form factors. We explore two scenarios: one simulating a handheld AR device and another simulating an optical see-through head-mounted display (HMD), and ran a study to evaluate both simulations. Results show that participants perceived both simulations’ interaction techniques and tasks to be similar to their real-world counterparts. In terms of task completion time, the simulated handheld version is slower than its counterpart, but there is no significant difference between the simulated and real HMD versions. We find that typical touch-based interaction techniques for handheld devices are the most difficult to simulate, particularly high-precision tasks that rely on tactile feedback. However, this performance gap narrows for participants with more prior VR experience. Based on our findings, we discuss design implications for future work involving simulating AR using VR.
The problem
Simulating AR has key benefits
📐 Isolate parameters for testing
Different AR artifacts such as jitter, latency, FOV, and resolution can be modified to stress-test the application.💻 Build on 1 device only
Developer can evaluate their application on many types of AR devices such as the iPhone vs iPad vs Magic Leap vs Hololens to see which works best for their application without having to acquire these devices nor create separate builds to evaluate.🧯 Simulate different environmental conditions
All environmental factors can be controlled to ensure robust testing. Difficult environments can also be simulated.Method
Simulating AR using only a commercial VR headset with hand tracking
Handheld AR Simulation
The left side is the simulator and the right side is the real iOS application.
AR HMD Simulation
The left side is the simulator and the right side is the real Hololens application.
The team
Cheng-Wei Hu and me