Beyond the Field-of-View: Enhancing Scene Visibility and Perception With Clip-Recurrent Transformer

Hao Shi, Qi Jiang, Kailun Yang, Xiaoting Yin, Ze Wang, Kaiwei Wang

Abstract

Vision sensors are widely applied in vehicles, robots, and roadside infrastructure. However, due to limitations in hardware cost and system size, camera Field-of-View (FoV) is often restricted and may not provide sufficient coverage. Nevertheless, from a spatiotemporal perspective, it is possible to obtain information beyond the camera's physical FoV from past video streams. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety. To achieve this, we introduce the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. FlowLens offers two key features: 1) a newly designed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated over time; and 2) a multi-branch Mix Fusion Feed Forward Network (MixF3N) to enhance the precise spatial flow of local features. Extensive experiments on the modified KITTI360 dataset demonstrate that FlowLens achieves state-of-the-art performance in beyond-FoV semantics and object detection.

Introduction: The Challenge of Limited Field-of-View

Autonomous vehicles and robotic systems rely heavily on visual sensors to navigate complex environments. However, physical constraints and hardware costs often limit a camera's Field-of-View (FoV), creating critical blind spots. Missing a traffic signal or a pedestrian just outside the current frame can lead to catastrophic failures.

While humans maintain situational awareness by remembering past observations, most computer vision systems process frames in isolation or within very short temporal windows. Existing video inpainting techniques can fill missing regions but are typically "offline," requiring future frames to reconstruct the present. For real-time applications like self-driving, waiting for future data is not feasible. This paper introduces FlowLens, a framework for online video inpainting that treats the temporal dimension as a "virtual sensor," expanding the FoV using only current and past data.

The FlowLens Architecture

FlowLens is a transformer-based architecture designed to "remember" and propagate information from past observations into the current frame's blind spots.

Figure 1: Overview of the FlowLens architecture (a) showing the Clip-Recurrent Hub and its application to beyond-FoV perception tasks (b).

1. Explicit Flow-guided Propagation

The model utilizes a flow completion network to estimate pixel movement within masked (blind) areas. By warping features from previous frames into the current coordinate system, the model provides a geometric foundation for reconstruction. To account for estimation errors, Deformable Convolutional Networks (DCN) are used to refine the alignment of these temporal features.

2. Clip-Recurrent Hub and 3D-Decoupled Cross Attention (DDCA)

The "memory" of FlowLens resides in its Clip-Recurrent Hub. Instead of re-processing the entire video history, the hub caches Keys and Values from previous clips. To query this memory efficiently, the authors developed 3D-Decoupled Cross Attention (DDCA). Standard 3D attention is computationally prohibitive for real-time use; DDCA solves this by performing attention along the temporal, horizontal, and vertical axes independently. This allows the model to capture global context across time and space with significantly reduced complexity.

3. Mix Fusion Feed Forward Network (MixF3N)

To preserve fine-grained local details—essential for tasks like object detection—the standard feed-forward layers are replaced with MixF3N. This multi-branch design uses varying convolution kernel sizes (3x3 and 5x5) to fuse local spatial features more effectively than standard linear layers.

Experimental Evaluation

To evaluate the system, the researchers derived a specialized benchmark from the KITTI360 dataset. They simulated two primary scenarios:

  • Outer FoV Expansion: Simulating standard pinhole cameras where the periphery is missing.
  • Inner FoV Expansion: Simulating spherical cameras with central blind spots (e.g., due to mounting hardware).

The model was also tested on standard video inpainting benchmarks like YouTube-VOS and DAVIS to ensure generalizability.

Key Results and Perception Enhancements

FlowLens demonstrates that reconstructing unseen scenes significantly boosts downstream perception tasks.

Figure 4: Qualitative results showing Beyond-FoV vision, segmentation, and detection for both pinhole and spherical camera models.

Quantitative Performance

  • Inpainting Quality: On KITTI360, FlowLens achieved a PSNR of 20.13 dB for outer expansion, outperforming previous models significantly.
  • Semantic Segmentation: The model improved the Mean Intersection over Union (mIoU) in previously "unseen" regions by nearly 19%.
  • Object Detection: By "hallucinating" vehicles and signs that had recently left the physical frame, the model increased Average Precision (AP) by 6.5%.

Efficiency for Edge Deployment

The "small" variant, FlowLens-s, achieves 27.31 FPS on an NVIDIA Jetson Orin NX. This real-time performance confirms that the architecture is suitable for deployment on intelligent vehicles and mobile robots.

Conclusion

FlowLens proves that the "past" is a powerful resource for expanding a sensor's physical limits. By combining explicit geometric flow with an efficient recurrent transformer, the system provides reliable semantic context that improves safety and perception. Future work will explore multi-modal integration (e.g., LiDAR) and the safety implications of using generative "hallucinations" in critical decision-making pipelines.

Summary updated

Abstract

Vision sensors are widely applied in vehicles, robots, and roadside infrastructure. However, due to limitations in hardware cost and system size, camera Field-of-View (FoV) is often restricted and may not provide sufficient coverage. Nevertheless, from a spatiotemporal perspective, it is possible to obtain information beyond the camera's physical FoV from past video streams. In this paper, we propose the concept of online video inpainting for autonomous vehicles to expand the field of view, thereby enhancing scene visibility, perception, and system safety. To achieve this, we introduce the FlowLens architecture, which explicitly employs optical flow and implicitly incorporates a novel clip-recurrent transformer for feature propagation. FlowLens offers two key features: 1) FlowLens includes a newly designed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated over time. 2) It integrates a multi-branch Mix Fusion Feed Forward Network (MixF3N) to enhance the precise spatial flow of local features. To facilitate training and evaluation, we derive the KITTI360 dataset with various FoV mask, which covers both outer- and inner FoV expansion scenarios. We also conduct both quantitative assessments and qualitative comparisons of beyond-FoV semantics and beyond-FoV object detection across different models. We illustrate that employing FlowLens to reconstruct unseen scenes even enhances perception within the field of view by providing reliable semantic context. Extensive experiments and user studies involving offline and online video inpainting, as well as beyond-FoV perception tasks, demonstrate that FlowLens achieves state-of-the-art performance. The source code and dataset are made publicly available at https://github.com/MasterHow/FlowLens.
BibTeX
@article{shi2022beyond,
  author = {Hao Shi and Qi Jiang and Kailun Yang and Xiaoting Yin and Ze Wang and Kaiwei Wang},
  title = {{Beyond the Field-of-View: Enhancing Scene Visibility and Perception With Clip-Recurrent Transformer}},
  journal = {IEEE Trans. Intell. Veh.},
  year = {2022},
}
Figure