Designing a Mixed Reality Framework for Enriching Interactions in Robot Simulations

Abstract

Experimentation of expensive robot systems typically requires complex simulation models and expensive hardware setups for constructing close-to-real world environments in order to obtain reliable results and draw insights to the actual operation. However, the test-development cycle is often time-consuming and resource demanding. A cost-effective solution is to conduct experiments by replacing expensive or dangerous components with simulated counterparts. Based on the concept of Mixed Reality (MR), robot simulation systems can be created to involve real and virtual entities in the simulation loop. However, seamless interaction between objects from the real and the virtual world remains a challenge. This paper presents a generic framework for constructing MR environments that facilitate interactions between objects from different dimensions of reality. In comparison to previous frameworks, we propose a new interaction scheme that describes the necessary stages for creating interactions between real and virtual objects. We demonstrate the strength of our MR framework and the proposed MR interaction scheme in the context of robot simulation. 1