TL;DR
This paper tackles the challenge of temporal view synthesis, specifically filling in "disoccluded" regions that appear when a camera moves. Instead of directly predicting pixel colors, they propose an "infilling vector" approach that points to existing, known regions in the scene from which to copy information. This method, combined with temporal and depth cues, significantly improves the quality and temporal consistency of synthesized frames, crucial for applications like VR frame rate upsampling.
What Problem Does This Paper Address?
When a camera moves, especially in virtual reality (VR) or gaming applications, new parts of the scene can become visible while previously visible parts are hidden. This phenomenon is called "disocclusion." In temporal view synthesis, the goal is to predict the next frame in a video sequence given previous frames and camera motion. A key challenge is filling these disoccluded regions to create a seamless and realistic next frame. Traditional methods like warping the previous frame leave holes, and existing inpainting techniques often struggle with the temporal consistency required for video or don't effectively leverage the available depth and motion information. This paper aims to address this by proposing a novel method to intelligently fill these disoccluded regions.
Methods
The core of the proposed method is the Infilling Vector Prediction (IVP). Instead of directly predicting the color of a disoccluded pixel, the model predicts an "infilling vector" for that pixel. This vector points to a corresponding pixel in a known region of the warped frame. The intensity from that known pixel is then copied to fill the disoccluded region. This approach is inspired by the idea that disoccluded regions often reveal background elements, and by pointing to these background regions, the method can reconstruct the missing information.
The authors leverage two key pieces of information to guide this vector prediction:
- Temporal Guidance: Since camera motion is often correlated across frames, the disoccluded regions and the direction of infilling tend to be similar over time. The model estimates "fictitious" infilling vectors from a previous frame (fn-2) to the current frame (fn) and uses these as a temporal prior to predict the vectors for the next frame (fn+1). This prior is warped and smoothed before being fed into a U-Net architecture.
- Depth Guidance: The depth map of the scene provides crucial information about relative background regions. The model warps the depth map of the previous frame and uses this, along with the temporal prior, as input to the U-Net. This helps the infilling vectors point towards appropriate background locations.
Additionally, the paper introduces an Iterative Infilling process. At test time, the model can sometimes predict infilling vectors that incorrectly point to foreground objects. By using the warped depth, the system can identify and discard these incorrect vectors, iteratively refining the infilling process until the disoccluded regions are filled correctly.
Datasets
The authors introduce a new dataset called IISc-VEED (Indian Institute of Science Virtual Environment Exploration Dataset). This dataset consists of high-resolution (1920x1080) videos rendered at 30 frames per second, specifically designed to highlight disocclusion challenges. It comprises 800 video clips, each 12 frames long.
They also evaluate their method on the SceneNet RGB-D dataset, which features lower-resolution videos (320x240) with lower frame rates, leading to larger disocclusions.
Key Insights and Ideas
- Infilling Vectors as a Better Representation: Predicting an "infilling vector" is more robust than directly predicting pixel intensities. This avoids blurring and color gradation artifacts often seen in direct intensity prediction methods.
- Exploiting Temporal Correlation: The consistent nature of camera motion across frames provides a strong prior for infilling directions, significantly improving prediction accuracy.
- Depth as a Spatial Constraint: Depth information acts as a crucial spatial constraint, guiding the infilling vectors to point to the correct background regions and preventing them from incorrectly sampling foreground elements.
- Iterative Refinement for Robustness: The iterative infilling process is essential for correcting errors, especially when temporal or depth cues are less reliable, leading to sharper and more accurate results.
Key Results
The proposed IVP method, when combined with temporal and depth guidance, consistently outperforms existing state-of-the-art methods on both the IISc-VEED and SceneNet RGB-D datasets across various metrics, including MSE, SSIM, and a novel temporal consistency (TC) measure. For instance, on IISc-VEED, their model achieves a D-MSE of 47 and a D-SSIM of 0.7729, outperforming other methods. Qualitatively, their model produces sharp and artifact-free infilled regions, preserving textures and patterns effectively, as demonstrated in Figure 4 and Figure 6. The paper also highlights that their method is significantly faster than graphical rendering, making it suitable for real-time applications.
Limitations and Future Work
The authors acknowledge that while their method is faster than rendering, further optimization is needed for true real-time performance in low-compute VR devices. They also note that the SceneNet dataset, with its lower frame rates, presents challenges where depth information can sometimes be incorrect, leading to warping issues. Future work could explore more advanced techniques for handling noisy depth data and further improving the temporal consistency, especially in scenarios with rapid camera motion or scene changes. The paper also suggests that their approach could be extended to handle more complex scenarios beyond static scenes.
Summary updated