Abstract
Vision-based human-computer interaction is becoming important nowadays. It offers natural interaction with computers and frees users from mechanical interaction devices, which is favourable especially for wearable computers. This paper presents a human-computer interaction system based on a conventional webcam and hand gesture recognition. This interaction system works in real time and enables users to control a computer cursor with hand motions and gestures instead of a mouse. Five hand gestures are designed on behalf of five mouse operations: moving, left click, left-double click, right click and no-action. An algorithm based on Particle Filter is used for tracking the hand position. PCA-based feature selection is used for recognizing the hand gestures. A transition network is also employed for improving the accuracy and reliability of the interaction system. This interaction system shows good performance in the recognition and interaction test.
Introduction
Traditional input devices like mice and keyboards are often cumbersome for emerging technologies such as wearable computers. Vision-based Human-Computer Interaction (HCI) offers a more "natural" alternative but faces hurdles including environmental noise, high computational loads for real-time processing, and human jitter causing shaky cursor movement. Furthermore, systems must solve the "Midas Touch" problem—distinguishing between intentional control gestures and incidental hand movements. This paper proposes a real-time system using a standard webcam to replace the mouse with hand gestures.
System Overview and Methodology
The proposed system follows a multi-stage pipeline to transform raw video into precise mouse commands.
Hand Segmentation
The system utilizes the HSV (Hue, Saturation, and Value) color space to isolate skin tones. To refine the resulting binary image, a "longest contour" rule is applied, assuming the largest skin-colored blob is the hand and discarding smaller noise pixels.
Tracking and Recognition
- Tracking (Particle Filter): To follow hand movement, a Particle Filter distributes sample points across the image. Particles in the hand region gain "confidence," while others are resampled near high-confidence areas, ensuring robust tracking even during rapid motion.
- Recognition (PCA): For gesture identification (Move, Left Click, etc.), the system uses Principal Component Analysis (PCA). This reduces complex hand images into low-dimensional vectors in an "eigenspace," allowing for rapid matching against a small training set.
- Transition Network & Anti-Shake: A logic layer analyzes sequences of three frames to confirm actions, preventing errors from momentary twitches. A geometric check ignores movements with sharp angles (less than 100°) to eliminate unintentional jitter.
Results and Discussion
The system was evaluated using a custom dataset of 5,000 images from 10 participants, achieving a global mean recognition accuracy of 98.3%. The "Move" gesture was most reliable (99.5%), while "Left-double click" was the most challenging (97.6%).
The application maintains a real-time feel, running at 21 Frames Per Second (FPS) on standard hardware. In usability tests, such as operating a calculator or creating files, the system performed acceptably, though slightly slower than touch-based interfaces. A key practical feature is the "no-action" gesture (a closed fist), which acts like lifting a physical mouse to reposition the hand without moving the cursor.
Conclusion
While highly accurate, the system currently requires users to wear long sleeves to avoid arm-skin interference and is sensitive to extreme lighting. Future work will focus on optimizing algorithms for complex environments and closing the performance gap with mature touch-based interfaces.
Summary updated