TL;DR
This paper introduces "Display in the Wild" (DIW), a novel system that projects dynamic, large-format screens onto any surface, regardless of its shape or movement. By combining a projector with a depth camera and employing a new calibration and distortion correction method, DIW can adapt in real-time to non-planar and moving surfaces, overcoming the limitations of traditional displays.
What Problem Does This Paper Address?
Traditional displays like TVs and monitors are constrained by their fixed size, rigid flat surfaces, and static placement. This limits their application in dynamic environments or when a large, flexible display is needed. Imagine needing a big screen for an outdoor event on a tent, or a display that can adapt to the folds of a banner – current technology struggles with these scenarios. The paper addresses this by proposing a system that can project a stable, undistorted screen onto virtually any surface, anywhere, and at any time, effectively turning the "wild" environment into a display canvas.
Methods
The core of the DIW system lies in its hardware and software.
Hardware: The system uses a projector to generate the display and a depth camera (specifically, a Kinect v2) to capture the 3D geometry of the target surface in real-time. This combination is crucial because projectors can't sense the surface they're projecting onto, while depth cameras excel at capturing 3D information quickly, even in challenging lighting conditions.
Software: * DIW-Specific Calibration: A novel calibration process is introduced to accurately determine the spatial relationship between the projector and the depth camera. This method is designed to be fast, robust to changes in projector resolution, and doesn't require markers. It focuses on extrinsic parameters (position and orientation). * Fast Geometric Distortion Estimation and Correction: The system predicts distortions on dynamic, non-planar surfaces by analyzing the depth data. Crucially, it focuses on the boundary pixels of the projected surface to efficiently determine the correction area, rather than processing the entire surface. This significantly speeds up the process. * Real-time Undistortion: A backward mapping technique is used to warp the original image to compensate for the predicted distortions, resulting in a perceptually correct, rectangular screen. This process is optimized for speed by using Gaussian weight interpolation to create a dense depth map from sparse data.
Datasets
The paper does not explicitly mention training datasets. For evaluation, they created their own "datasets" by:
- Projecting a chessboard pattern onto various dynamic and non-planar surfaces.
- Using Unmanned Aerial Vehicles (UAVs) to create dynamic, lightweight canvas surfaces.
- Manually creating dynamic non-planar surfaces to simulate extreme environments.
These custom scenarios allowed them to test the system's performance in real-world, dynamic conditions, which differ from static, controlled environments often used in other calibration research.
Key Insights and Ideas
- "Display in the Wild" is more than just projection: It's about creating a display experience that is truly adaptive and integrated into any environment, overcoming the physical limitations of conventional screens.
- Depth cameras are key for dynamic surfaces: Their ability to capture 3D geometry in real-time is essential for correcting distortions on moving or irregularly shaped surfaces, something RGB cameras struggle with.
- Boundary pixels are enough for efficiency: Focusing only on the edges of the projected area for distortion correction dramatically speeds up the process, enabling real-time performance. This is a clever optimization that avoids computationally expensive full-surface analysis.
- User viewpoint adaptation is simplified: The paper notes that the human visual system is forgiving of slight viewpoint variations, allowing for a simplified, manually set user position without significantly impacting perceived distortion. This pragmatic approach keeps the system running in real-time.
Key Results
- Quantitative:
- The proposed calibration method achieved a low standard deviation of re-projection errors (0.022), comparable to or better than existing methods.
- Line Straightness Error Ratio (LSER) results were consistently high (average 0.99078 for vertical, 0.99467 for horizontal lines) with a tolerance of 10°, indicating accurate correction with less than 0.1° difference in straight lines.
- The DIW system demonstrated significantly faster correction times (0.103 seconds total) compared to existing methods ([7] at 9.745s, [9] at 8.148s) for a 1920x720 resolution image.
- Qualitative:
- The system successfully generated undistorted, TV-like rectangular screens on dynamic, non-planar surfaces created by UAVs and lightweight canvases.
- It effectively corrected significant geometric distortions on extremely varying screen surfaces, producing stable rectangular images.
- The results show the system's usefulness in both practical (e.g., outdoor events) and extreme environments.
Limitations and Future Work
- Bright Environments: Projectors inherently struggle in bright ambient light, limiting DIW's effectiveness in daylight or outdoor conditions.
- Off-Axis Projectors: The current calibration and correction methods are not directly applicable to off-axis projectors, which require different formulations.
- User Position Tracking: The user's position is currently manually set. Continuously tracking the user's position would require more advanced, computationally intensive algorithms, which are outside the scope of this paper.
Future Work: The authors suggest extending the DIW system to ultra-high definition (UHD) displays by implementing parallel processing with GPUs, which would further enhance its capabilities for high-resolution applications.
Summary updated