Abstract: Limited by hardware cost and system size, a camera's Field-of-View (FoV) is not always satisfactory. However, from a spatio-temporal perspective, information beyond the camera's physical FoV is off-the-shelf and can actually be obtained "for free" from the past. In this paper, we propose a novel task termed Beyond-FoV Estimation, aiming to exploit past visual cues and bidirectional break through the physical FoV of a camera. We put forward a FlowLens architecture to expand the FoV by achieving feature propagation explicitly by optical flow and implicitly by a novel clip-recurrent transformer, which has two appealing features: 1) FlowLens comprises a newly proposed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated in the temporal dimension. 2) A multi-branch Mix Fusion Feed Forward Network (MixF3N) is integrated to enhance the spatially-precise flow of local features. To foster training and evaluation, we establish KITTI360-EX, a dataset for outer- and inner FoV expansion. Extensive experiments on both video inpainting and beyond-FoV estimation tasks show that FlowLens achieves state-of-the-art performance.
Introduction
In the world of autonomous driving and robotics, Field-of-View (FoV) is a critical bottleneck. Hardware constraints—like the cost of high-end sensors or the physical size of camera mounts—often leave vehicles with blind spots. Even advanced spherical cameras frequently suffer from "inner blind spots" caused by their own mounting hardware. Missing a traffic light or a merging vehicle because it is just outside the frame is a major safety risk.
However, the authors point out a clever workaround: temporal redundancy. If a car was visible two seconds ago but has now moved into a blind spot, the information about that car still exists in the "past" of the video stream. The challenge is how to retrieve, align, and "paint" that past information into the current frame accurately and fast enough for a moving vehicle.
Methodology
The authors introduce FlowLens, an architecture designed for online (real-time) video inpainting. Unlike traditional video editing tools that look at an entire video file at once, FlowLens works frame-by-frame using the following core components:
- Explicit Flow-guided Propagation: The system estimates optical flow to determine where objects from previous frames should be in the current missing regions, warping features from the past into the present.
- Clip-Recurrent Transformer: This features a Clip-Recurrent Hub that caches features from the previous video clip and passes them to the next, maintaining a "memory" of the scene.
- 3D-Decoupled Cross Attention (DDCA): To manage computational costs, DDCA splits the attention mechanism into three dimensions (time, width, and height), allowing efficient querying of relevant past features.
- Mix Fusion Feed Forward Network (MixF3N): This module uses multiple branches with different convolution sizes (3x3 and 5x5) to capture fine local spatial details.
Experimental Results
The researchers evaluated FlowLens on three main datasets: YouTube-VOS, DAVIS, and a new benchmark called KITTI360-EX. The latter was derived from the KITTI360 dataset with specific masks to simulate both Outer FoV Expansion (for pinhole cameras) and Inner FoV Expansion (for spherical camera blind spots).
Key Performance Metrics
- Superior Reconstruction: On the KITTI360-EX dataset, FlowLens achieved a 20.13 dB PSNR for outer expansion, outperforming existing state-of-the-art (SOTA) models like E2FGVI and FuseFormer.
- Perception Gains: In previously "unseen" blind spots, the Mean Intersection over Union (mIoU) for semantic segmentation improved by 18.99%, and object detection precision (AP) increased by 6.5%.
- Edge-Ready Speed: A lightweight version, FlowLens-s, runs at 27.31 FPS on an NVIDIA Jetson Orin NX, meeting the requirements for real-time safety systems.
Conclusion
FlowLens demonstrates that online video inpainting can serve as a vital perception tool for robotics. By intelligently recycling past data, the model allows vehicles to "see" and track objects that are physically hidden from the camera. While outward expansion remains challenging due to complex distortions at frame edges, the system's ability to fill inner blind spots significantly enhances safety. Future work may integrate multi-modal fusion, combining camera memory with real-time data from LiDAR or event cameras for even greater robustness.
Summary updated