Deep Learning-Based Hand Posture Recognition for Pen Interaction Enhancement

This chapter examines how digital pen interaction can be expanded by detecting different hand postures formed primarily by the hand while it grips the pen. Three systems using different types of sensors are considered: an EMG armband, the raw capacitive image of the touchscreen, and a pen-top fisheye camera. In each case, deep neural networks are used to perform classification or regression to detect hand postures and gestures. Additional analyses are provided to demonstrate the benefit of deep learning over conventional machine-learning methods, as well as explore the impact on model accuracy resulting from the number of postures to be recognised, user-dependent versus user-independent models, and the amount of training data. Examples of posture-based pen interaction in applications are discussed and a number of usability aspects resulting from user evaluations are identified. The chapter concludes with perspectives on the recognition and design of posture-based pen interaction for future systems.