Abstract
The input method based on free-hand gestures has gradually become a hot research direction in the field of human-computer interaction. Hand gestures like sign languages, however, demand quite a lot of knowledge and practice for interaction, and air writing methods require their users to hold the arm and hand in mid-air for a period of time. These methods limit the user experience and get severer when a large number of gestures are required. To address the problem, this paper presents a novel human-3DTV interaction system based on a set of simple free-hand gestures for direct-touch interaction with a virtual interface to facilitate human-3DTV interaction. Specifically, our system projects a virtual interface in front of the user who wears the 3D shutter glass, and the user just stretches the arm and touches the virtual interface like performing on a smart phone with a touch screen, using gestures such as Click, Slide, Hold, Drag and Zoom In/Out. Our system is able to recognize the user's gesture fast and accurately, as the system only needs to search for a small region neighboring the virtual interface for a small set of gesture types. Because we adopt the key gestures using on smart phones, our free-hand gestures can be easily used by anyone with only a brief training. The users feel more comfortable than traditional gesture input methods and can effectively interact with 3DTV using our system. We report a comprehensive user study on accuracy and speed to validate the advantages of the proposed human-3DTV interaction system.
Introduction
As Smart TVs evolve, traditional remote controls are increasingly inadequate for navigating complex applications or entering text. While free-hand gesture systems (like waving or "air writing") offer a potential solution, they often suffer from two major flaws: a steep learning curve for complex "sign languages" and physical fatigue, commonly known as "Gorilla Arm" syndrome. Users find it exhausting to hold their arms in mid-air without a physical or visual reference point, leading to high variance in movement and decreased sensor accuracy.
To address these issues, researchers have developed a human-3DTV interaction system that projects a "touchable" virtual interface in mid-air. By combining stereoscopic 3D projection with Kinect-based tracking, the system allows users to control their television using familiar smartphone-style gestures, providing a more intuitive and less fatiguing experience.
System Methodology
The system hardware stack consists of a 3DTV, active shutter glasses, a Microsoft Kinect sensor, and a processing unit.
The Virtual Touch Interface
The core innovation is the projection of a stereoscopic virtual interface exactly 0.5 meters in front of the user’s eyes. Because the user wears 3D glasses, the UI appears to float in space. To help the user "find" the interface, the authors implemented a "fly-in" effect where the UI moves from the TV screen toward the user upon activation. This provides a specific spatial coordinate for the user to "touch," drastically reducing movement variance.
Tracking and Gesture Recognition
The system uses the Kinect’s RGB and depth data to locate the user's eyes, ensuring the virtual UI is rendered with the correct perspective. The gesture engine recognizes three primitive gestures (Click, Hold, Slide) and three combo gestures (Double-Click, Drag, Multi-touch Zoom). These are processed using Hidden Markov Models (HMM), which convert finger trajectories into spatio-temporal patterns quantized into spherical coordinates ($r, \theta, \phi$).
Experimental Results
The system was benchmarked against the gesture control system of a commercial Samsung Smart TV in a user study with 20 participants.
- Accuracy: The proposed system achieved an average error rate of approximately 5%, compared to the commercial system’s 10%. The researchers noted that "Clicking" (Z-axis movement) had a slightly higher error rate than "Sliding" (XY-plane movement) due to the Kinect's inherent depth noise.
- Speed and Usability: In tasks requiring 30 trials, the proposed system was significantly faster and less cumbersome than the commercial baseline.
- User Preference: 90% of participants preferred the virtual touch interface over conventional remotes or traditional "air-waving" gestures, citing the familiarity of smartphone-style interactions.
Conclusion
By leveraging existing muscle memory from smartphone usage and providing a visual reference point in 3D space, this system offers a robust and user-friendly alternative to traditional TV interaction methods. Future work will focus on improving robustness to head movements and expanding the complexity of the virtual interface for more immersive applications.
Reference: Zhang, S., & Zhang, S. (2019). A Novel Human-3DTV Interaction System Based on Free Hand Gestures and a Touch-Based Virtual Interface. IEEE Access, 7, 165961-165973.
Summary updated