Pen+Touch+Midair: Cross-Space Hybrid Bimanual Interaction on Horizontal Surfaces in Virtual Reality

We present and explore a design space for hybrid bimanual pen and touch input extended to midair interaction in desktop-based virtual reality (VR). The investigation focuses on asymmetric interaction patterns combining the pen with the other hand when interacting in the same “space” (either surface or midair), across both spaces, and with cross-space transitions (from surface to midair and vice versa). To show how these interactions and associated gestures can work in context, we create three testbed applications for modelling, volumetric rendering, and terrain editing. A qualitative evaluation with 16 participants provides insights into hand and space preferences for key tasks including object manipulation, navigation, and menu invocation. From the results, design implications are identified for VR systems that combine pen, touch, and midair input, and extensions to other forms of extended reality are discussed.