Peer Review #2 of "Self-supervised recurrent depth estimation with attention mechanisms (v0.1)"

Ilya Makarov, Maria Bakhanova, Sergey Nikolenko, Olga Gerasimova, K Cho, B Van Merrienboer, C Glc ehre, D Bahdanau, F Bougares, H Schwenk, Y Bengio, F Gers, J Schmidhuber, F Cummins, C Godard, O Mac Aodha, M Firman, G Brostow, V Guizilini, R Ambrus, S Pillai, A Raventos, A Gaidon, S Hochreiter, J Hu, M Ozay, Y Zhang, T Okatani, J Hur, S Roth, Y Kuznietsov, M Proesmans, L Van Gool, X Luo, J Huang, R Szeliski, K Matzen, J Kopf, V Aliev, P Polyakov, A Korinevskaya, D Maslov, U Sharma, H Wang, M Menze, A Geiger, C Heipke, J Schnberger, J Frahm, Z Wang, A Bovik, H Sheikh, E SimoncelliOpenalex Unknown 2022 · DOI · OpenAlex

Abstract

Depth estimation is a critical task for computer vision applications like autonomous driving. While self-supervised approaches—which rely on camera motion rather than ground truth depth maps—have advanced significantly, most still operate on a single-frame basis. This work explores integrating recurrent blocks and attention mechanisms to exploit temporal information from previous frames. By proposing new neural network architectures for monocular depth estimation, the authors demonstrate on the KITTI dataset that these modifications effectively leverage temporal context to improve prediction quality.

Introduction

Estimating 3D depth from a single 2D image is an inherently ill-posed problem, as multiple 3D configurations can result in the same 2D projection. Traditional self-supervised methods address this by comparing adjacent video frames to learn geometry. However, by predicting depth frame-by-frame, these models ignore the rich temporal "memory" available in video sequences.

This paper introduces a novel pipeline that incorporates ConvGRU recurrent blocks and self-attention mechanisms. By maintaining a hidden state of the scene's geometry over time, the model produces more consistent and accurate depth maps in real-time, effectively bridging the gap between independent frame analysis and continuous temporal understanding.

Methodology: Beyond Single-Frame Estimation

The authors build upon the Monodepth2 framework, a standard in self-supervised learning that uses photometric consistency between a target frame and source frames to supervise the network.

The standard self-supervised depth estimation pipeline, utilizing a Depth Net and Pose Net to perform view synthesis and calculate photometric loss.

To enhance this baseline, the following architectural modifications were introduced:

  1. Recurrent ConvGRU Blocks: Convolutional Gated Recurrent Units are integrated into the decoder at four different scales. These blocks pass hidden states from previous timestamps to the current one, allowing the network to "remember" spatial features.
  2. Self-Attention Fusion: A dedicated Fusion block combines current frame features with temporal hidden states. The authors utilize Residual Self-Attention units to capture long-range spatial dependencies, which helps the model understand global scene structure better than standard local convolutions.
  3. Two-Stage Training: The complexity of recurrent training is managed via a two-stage strategy: initial training with zero-initialized hidden states, followed by refinement using learned temporal context.

Experimental Results

The models were evaluated on the KITTI dataset (Eigen split), the industry standard for autonomous driving benchmarks.

Performance and Efficiency

The proposed "Model #2" (incorporating ConvGRU and Self-Attention Fusion) achieved superior results in Root Mean Square Error (RMSE) compared to the Monodepth2 baseline. Crucially, the model maintains high efficiency, running at 124 ± 16 FPS on an RTX6000 GPU. This makes it significantly more practical for real-world robotics than heavy supervised models like DORN, which often run at much lower frame rates (approx. 15 FPS).

Qualitative Improvements

Visual analysis indicates that the recurrent approach is particularly effective at:

  • Moving Objects: Better depth prediction for cyclists and cars.
  • Distant Regions: Reduced variance in far-off areas where single-frame models often struggle.
  • Object Boundaries: Sharper, more accurate edges for complex or angular shapes.

Discussion and Future Work

The study confirms that temporal consistency is vital for robust depth estimation. By aggregating history over a 10-frame window, the model resolves ambiguities caused by lighting changes or sensor noise that typically trip up single-frame models.

However, limitations remain. The model is constrained by GPU memory to 12-frame sequences, and like most self-supervised methods, it struggles with the "static camera" problem where moving objects violate the basic ego-motion assumption. Future research may involve integrating cost volumes or feature-based losses to further narrow the performance gap with fully supervised LiDAR-based systems.

Summary updated

Abstract

Depth estimation has been an essential task for many computer vision applications, especially in autonomous driving, where safety is paramount.Depth can be estimated not only with traditional supervised learning but also via a self-supervised approach that relies on camera motion and does not require ground truth depth maps.Recently, major improvements have been introduced to make self-supervised depth prediction more precise.However, most existing approaches still focus on single-frame depth estimation, even in the self-supervised setting.Since most methods can operate with frame sequences, we believe that the quality of current models can be significantly improved with the help of information about previous frames.In this work, we study different ways of integrating recurrent blocks and attention mechanisms into a common self-supervised depth estimation pipeline.We propose a set of modifications that utilize temporal information from previous frames and provide new neural network architectures for monocular depth estimation in a self-supervised manner.Our experiments on the KITTI dataset show that proposed modifications can be an effective tool for exploiting temporal information in a depth prediction pipeline.
BibTeX
@inproceedings{makarov2022peer,
  author = {Ilya Makarov and Maria Bakhanova and Sergey Nikolenko and Olga Gerasimova and K Cho and B Van Merrienboer and C Glc ehre and D Bahdanau and F Bougares and H Schwenk and Y Bengio and F Gers and J Schmidhuber and F Cummins and C Godard and O Mac Aodha and M Firman and G Brostow and V Guizilini and R Ambrus and S Pillai and A Raventos and A Gaidon and S Hochreiter and J Schmidhuber and J Hu and M Ozay and Y Zhang and T Okatani and J Hur and S Roth and Y Kuznietsov and M Proesmans and L Van Gool and X Luo and J Huang and R Szeliski and K Matzen and J Kopf and I Makarov and V Aliev and O Gerasimova and I Makarov and V Aliev and O Gerasimova and P Polyakov and I Makarov and A Korinevskaya and I Makarov and A Korinevskaya and V Aliev and I Makarov and A Korinevskaya and V Aliev and I Makarov and A Korinevskaya and V Aliev and I Makarov and D Maslov and O Gerasimova and V Aliev and A Korinevskaya and U Sharma and H Wang and D Maslov and I Makarov and D Maslov and I Makarov and M Menze and A Geiger and M Menze and C Heipke and A Geiger and M Menze and C Heipke and A Geiger and J Schnberger and J Frahm and Z Wang and A Bovik and H Sheikh and E Simoncelli},
  title = {{Peer Review \#2 of "Self-supervised recurrent depth estimation with attention mechanisms (v0.1)"}},
  booktitle = {Openalex Unknown},
  year = {2022},
  doi = {10.7287/peerj-cs.865v0.1/reviews/2},
}
Figure