TL;DR
This paper introduces Vermeer, a novel 360° viewable 3D display that allows users to directly interact with virtual objects inside the display volume, overcoming a major limitation of previous systems. By cleverly using a pair of parabolic mirrors and an optical illusion, Vermeer enables intuitive, glasses-free 3D interaction and explores integrated sensing methods for a more compact and seamless user experience.
What Problem Does This Paper Address?
Existing 360° viewable 3D displays, while impressive in their ability to provide stereoscopic, viewpoint-corrected imagery to multiple users simultaneously without special eyewear, typically restrict user interaction to outside the display volume. This is often due to physical barriers like glass shields or the presence of moving parts that would interfere with direct touch. This limitation hinders the naturalness and intuitiveness of interacting with 3D content, forcing users to rely on indirect input methods. The authors aim to bridge this gap by enabling direct, in-volume interaction with 3D objects displayed in a 360° viewable format.
Methods
The core of Vermeer's innovation lies in its use of a pair of parabolic mirrors arranged in a "clamshell" configuration. This setup exploits a known optical illusion: an object placed in the focal point of the lower mirror is reimaged as a virtual object floating just above the upper mirror. This creates the perception of a 3D object within the display volume.
The system builds upon existing 360° viewable 3D display technology, specifically referencing a design that uses a spinning diffuser to create multiple viewpoints. However, Vermeer adapts this by using a rear-projection optical configuration, which is necessary to avoid occlusion by user input.
For interaction, two primary methods are explored:
- Kinect-based Input: Initially, a Kinect depth camera is mounted above the display to detect user fingertips entering the virtual volume. This allows for basic interactions like animating 3D characters by touching them.
- Integrated Sensing: Leveraging the optical properties of the parabolic mirrors, the system explores novel ways to sense input without external cameras.
- Infrared (IR) Reimaging: By illuminating an IR-diffusing object (like a dome) within the mirror pair, an IR camera can image the reimaged object. When a user's finger intersects this virtual IR object, it scatters IR light, which is then detected by the camera, enabling touch detection.
- Shared Optical Path: A further refinement uses a hot mirror to allow both the projector and an IR camera to share the same optical path. A ring of IR LEDs floods the area above the mirror pair. User fingertips scatter this IR light downwards, where it's reimaged and detected by the IR camera, allowing for integrated touch sensing.
Datasets
Not discussed in the paper.
Key Insights and Ideas
- Optical Illusions as Interaction Enablers: The paper demonstrates that well-understood optical illusions, like the one created by parabolic mirrors, can be repurposed not just for display but also for enabling novel interaction paradigms.
- Direct In-Volume Interaction is Transformative: The ability to "reach into" and touch 3D objects, rather than just pointing at them from the outside, significantly enhances the intuitiveness and immersion of 3D interaction.
- Integrated Sensing for Compactness: Moving away from external sensors like Kinect towards sensing directly within the display's optical path leads to more compact and aesthetically pleasing tabletop systems.
- Reversibility of Optics: The paper highlights how the optical properties of the mirror system can be exploited in reverse, allowing light from the real world (user's hand) to be reimaged and detected, creating a dual-purpose display and sensor.
Key Results
The paper presents proof-of-concept demonstrations of Vermeer's capabilities:
- 360° Viewable 3D Display: The system successfully renders 3D graphics viewable from all angles without eyewear.
- Direct In-Volume Touch Interaction: Users can place their fingers into the display volume and interact with virtual objects, as shown by animating a 3D character or manipulating physics-based objects.
- Integrated IR Sensing: Prototypes demonstrate the ability to detect fingertip intersections with virtual IR objects, enabling touch input without external cameras.
- Physics-Based Interaction: Demonstrations show users knocking over virtual objects using their fingers, highlighting the potential for realistic physical interactions.
Limitations and Future Work
Limitations:
- Display Size: The current display volume is small, limiting the number of simultaneous interactions and the scale of the virtual scenes.
- Viewpoint Constraints: While walk-up-and-use, there are still viewpoint limitations (truncation) that affect optimal viewing and interaction.
- Distortion: Minor distortions are present due to the imperfect manufacturing of the mirrors and the nature of reimaging a non-planar surface.
- Rendering Fidelity: The rendering speed is limited by the current hardware, impacting the achievable frame rate and image quality.
Future Work:
- Scaling Up: Exploring larger mirror pairs and potentially truncated optics to create larger display volumes.
- Higher Fidelity: Improving rendering speed and resolution through faster interfaces and projectors.
- Advanced Integrated Sensing: Utilizing high-speed cameras synchronized with the display to capture more accurate 3D touch information and potentially track user posture.
- Environmental Sensing: Using the optical setup to image the environment around the display, potentially for relighting virtual scenes or tracking user heads for perspective correction.
- Multi-User Scenarios: Investigating applications for personalized content display in multi-user settings.
- Mitigating Depth Distortion: Developing interesting feedback mechanisms to address depth perception issues arising from interaction.
Summary updated