TL;DR
This paper introduces a novel 3-degree-of-freedom (3DOF) control system for mobile devices that uses the built-in camera to enable intuitive scrolling and zooming of large documents and images. By tracking natural features and employing a Hidden Markov Model for motion state inference, the system offers a fluid and responsive interaction experience without requiring external hardware or pre-calibration.
What Problem Does This Paper Address?
The proliferation of mobile devices with wireless connectivity has made on-demand information retrieval ubiquitous. However, viewing large documents, maps, or images on small mobile screens presents a significant usability challenge. Traditional input methods like joysticks or styluses are often cumbersome, requiring users to switch between controls for scrolling and zooming, and rarely supporting simultaneous manipulation. This paper addresses the need for a more natural, integrated, and efficient way to navigate large visual content on mobile devices.
Methods
The core of the system relies on a camera-driven approach to infer device motion. Here's a breakdown of the key techniques:
- Feature Tracking: The system identifies and tracks natural features (keypoints) within the camera's field of view. This is achieved using a combination of Shi-Tomasi corner detection and autocorrelation matrices to find robust keypoints.
- Optical Flow: Optical flow algorithms (specifically a variation of the Lucas-Kanade method with image pyramids) are used to estimate the motion of these tracked keypoints between consecutive frames.
- Motion Amplification: To overcome the inherent sensitivity limitations of small hand movements, the detected motion vectors (for scrolling and zooming) are amplified by user-adjustable factors.
- Hidden Markov Model (HMM): To distinguish intentional user actions from accidental movements and image noise, a 3-state HMM is employed. This model infers the user's intended motion state (scrolling, zooming, or stationary) based on the observed motion measurements. This allows for smoother control by only amplifying intended movements.
- Affine Transformation: The tracked keypoint movements are used to compute a 3x3 affine transformation matrix that represents the overall device motion. This matrix is then decomposed to extract scroll (x and y) and zoom components.
- Bidirectional Optical Flow Filtering: To improve the reliability of optical flow estimates, a bidirectional filtering step is used, comparing forward and backward flow predictions to discard unreliable tracks.
Datasets
Not discussed in the paper. The authors mention informal tests with several users and demonstrate the system on a Sony Vaio UX Micro PC.
Key Insights and Ideas
- Leveraging Existing Hardware: The system ingeniously utilizes the ubiquitous camera on mobile devices as a primary input sensor, eliminating the need for additional hardware like inertial sensors or specialized markers.
- Natural Feature Tracking for Robustness: Relying on natural features rather than pre-defined markers makes the system adaptable to diverse environments without requiring calibration or setup.
- The Power of Amplification and Smoothing: The combination of motion amplification for usability and an HMM for smoothing is crucial. Amplification makes navigation feasible, while the HMM prevents jitter and unintended movements, leading to a fluid user experience.
- 3DOF for Integrated Interaction: Enabling simultaneous scrolling and zooming (3DOF) significantly enhances navigation efficiency compared to 2DOF systems, allowing users to quickly transition between overview and detail.
- User Experience Matters: The authors highlight that while the core idea of optical flow for navigation is straightforward, the implementation details (keypoint management, filtering, HMM) have a strong impact on reliability and usability, directly addressing previous criticisms of camera-based controls being frustrating.
Key Results
- The prototype system runs at approximately 17 fps on a Windows XP-based device, suggesting a useful frame rate is achievable on modern mobile hardware.
- Informal user tests indicated that the control was intuitive and easy to use, with users consistently able to perform zoom-out, detail selection, and zoom-in actions.
- The system demonstrated the ability to navigate documents many times larger than the display area.
- The authors found that scrolling is generally more reliable than zooming, especially in environments with distant tracked objects.
- The paper provides a table of empirically determined variable settings (standard deviations, amplification factors, consistency factor) that were found to be smooth and responsive.
Limitations and Future Work
- User Preference for "Backward" Control: Some users perceived the control as "backward," feeling their hand movements should "push" the document rather than the document moving with their hand. The authors suggest this could be an adjustable user preference.
- Potential for 6DOF Tracking: The authors envision future work to improve optical flow accuracy to enable 6DOF tracking, which could allow for more advanced interactions like fine-tuned viewing and editing of 3D models.
- Further User Testing: More formal user studies are recommended to understand user preferences and identify specific applications that would benefit most from this control.
- Application Integration: The paper suggests exploring the integration of this control into various applications, including map viewing with clickable elements, document navigation, and GUI components like lists and dropdown menus.
Summary updated