Above-Screen Fingertip Tracking with a Phone in Virtual Reality

Using a phone as a VR controller is challenging because users cannot see their fingers when aiming for targets on the touchscreen. We propose using two mirrors mounted above the screen that reflect the front camera and a purpose-built deep neural network to robustly infer the 3D position of fingertips manipulating the phone. Network training is self-supervised after only a few initial labelled images and does not require any external sensor. We present a few example scenarios showing potential applications that use our phone-based fingertip tracker for precise touch input and above-screen interaction.