Abstract
Vision sensors are widely applied in vehicles, robots, and roadside infrastructure. However, due to limitations in hardware cost and system size, camera Field-of-View (FoV) is often restricted and may not provide sufficient coverage. Nevertheless, from a spatiotemporal perspective, it is possible to obtain information beyond the camera's physical FoV from past video streams. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety. To achieve this, we introduce the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. FlowLens offers two key features: 1) a newly designed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated over time; and 2) a multi-branch Mix Fusion Feed Forward Network (MixF3N) to enhance the precise spatial flow of local features. Extensive experiments on the modified KITTI360 dataset demonstrate that FlowLens achieves state-of-the-art performance in beyond-FoV semantics and object detection.
Introduction: The Challenge of Limited Field-of-View
Autonomous vehicles and robotic systems rely heavily on visual sensors to navigate complex environments. However, physical constraints and hardware costs often limit a camera's Field-of-View (FoV), creating critical blind spots. Missing a traffic signal or a pedestrian just outside the current frame can lead to catastrophic failures.
While humans maintain situational awareness by remembering past observations, most computer vision systems process frames in isolation or within very short temporal windows. Existing video inpainting techniques can fill missing regions but are typically "offline," requiring future frames to reconstruct the present. For real-time applications like self-driving, waiting for future data is not feasible. This paper introduces FlowLens, a framework for online video inpainting that treats the temporal dimension as a "virtual sensor," expanding the FoV using only current and past data.
The FlowLens Architecture
FlowLens is a transformer-based architecture designed to "remember" and propagate information from past observations into the current frame's blind spots.
1. Explicit Flow-guided Propagation
The model utilizes a flow completion network to estimate pixel movement within masked (blind) areas. By warping features from previous frames into the current coordinate system, the model provides a geometric foundation for reconstruction. To account for estimation errors, Deformable Convolutional Networks (DCN) are used to refine the alignment of these temporal features.
2. Clip-Recurrent Hub and 3D-Decoupled Cross Attention (DDCA)
The "memory" of FlowLens resides in its Clip-Recurrent Hub. Instead of re-processing the entire video history, the hub caches Keys and Values from previous clips. To query this memory efficiently, the authors developed 3D-Decoupled Cross Attention (DDCA). Standard 3D attention is computationally prohibitive for real-time use; DDCA solves this by performing attention along the temporal, horizontal, and vertical axes independently. This allows the model to capture global context across time and space with significantly reduced complexity.
3. Mix Fusion Feed Forward Network (MixF3N)
To preserve fine-grained local details—essential for tasks like object detection—the standard feed-forward layers are replaced with MixF3N. This multi-branch design uses varying convolution kernel sizes (3x3 and 5x5) to fuse local spatial features more effectively than standard linear layers.
Experimental Evaluation
To evaluate the system, the researchers derived a specialized benchmark from the KITTI360 dataset. They simulated two primary scenarios:
- Outer FoV Expansion: Simulating standard pinhole cameras where the periphery is missing.
- Inner FoV Expansion: Simulating spherical cameras with central blind spots (e.g., due to mounting hardware).
The model was also tested on standard video inpainting benchmarks like YouTube-VOS and DAVIS to ensure generalizability.
Key Results and Perception Enhancements
FlowLens demonstrates that reconstructing unseen scenes significantly boosts downstream perception tasks.
Quantitative Performance
- Inpainting Quality: On KITTI360, FlowLens achieved a PSNR of 20.13 dB for outer expansion, outperforming previous models significantly.
- Semantic Segmentation: The model improved the Mean Intersection over Union (mIoU) in previously "unseen" regions by nearly 19%.
- Object Detection: By "hallucinating" vehicles and signs that had recently left the physical frame, the model increased Average Precision (AP) by 6.5%.
Efficiency for Edge Deployment
The "small" variant, FlowLens-s, achieves 27.31 FPS on an NVIDIA Jetson Orin NX. This real-time performance confirms that the architecture is suitable for deployment on intelligent vehicles and mobile robots.
Conclusion
FlowLens proves that the "past" is a powerful resource for expanding a sensor's physical limits. By combining explicit geometric flow with an efficient recurrent transformer, the system provides reliable semantic context that improves safety and perception. Future work will explore multi-modal integration (e.g., LiDAR) and the safety implications of using generative "hallucinations" in critical decision-making pipelines.
Summary updated