Rebalancing gradient to improve self-supervised co-training of depth, odometry and optical flow predictions

Marwane Hariat, Antoine Manzanera, David Filliat

Introduction

Self-supervised monocular depth estimation typically relies on the static scene assumption, where image warping between frames is attributed solely to camera motion. However, dynamic objects like moving vehicles or pedestrians violate this assumption, leading to high reconstruction errors that degrade depth map quality. While joint training of depth, pose, and optical flow networks has been proposed to handle these dynamic regions, a fundamental intrinsic bias exists: optical flow is naturally easier for networks to learn than complex 3D geometry. This often leads to a competitive scenario where the flow network dominates early on, "starving" the depth network of training data and resulting in degenerate outputs like infinite-depth "black stains."

The CoopNet Approach: From Competition to Cooperation

To address these instabilities, the authors introduce CoopNet, a framework that replaces competition with a cooperative training strategy. Instead of forcing networks to compete for pixels, CoopNet dynamically rebalances gradients to ensure each network specializes in the regions it is best suited for.

Quantile-Based Error Splitting

The core of CoopNet is a dynamic rebalancing mechanism based on the distribution of the difference ($\Delta$) between depth-based and flow-based reconstruction errors. Using the $P^2$ algorithm, the model tracks the quantiles of this distribution in real-time throughout the 30-epoch training cycle.

Quantile-based distribution of reconstruction errors used to define the agreement area for rigid scene learning.

As illustrated in the figure above, the training process identifies an Agreement Area—a tight neighborhood around the mean error where both depth and flow networks produce similar results. The depth and pose networks are exclusively trained on pixels within this area, ensuring they learn from reliable, static regions of the scene. This allows the stronger network (flow) to act as a filter for the weaker one (depth), similar to teacher-student techniques.

Specialized Training and Regularization

While the depth network focuses on the agreement area, the optical flow network is trained on the entire image. However, higher weights are assigned to the "tails" of the error distribution—pixels where the networks disagree. This encourages the flow network to specialize in modeling moving objects. Additionally, a flow regularization term ($\Delta_{flow}$) is introduced to ensure geometric consistency in textureless areas where photometric losses are uninformative.

Experiments and Results

CoopNet was evaluated on the KITTI and Cityscapes benchmarks, focusing on depth estimation and camera odometry.

Qualitative comparison of depth maps generated by CoopNet on urban scenes.

Performance Metrics

  • Depth Estimation: On the KITTI dataset, CoopNet achieved an Absolute Relative (Abs Rel) error of 0.113, surpassing established baselines such as Monodepth2 (0.115) and SGDepth (0.117).
  • Odometry: The method demonstrated significant improvements in camera trajectory accuracy. On KITTI Sequence 09, it reduced translation error to 8.42%, a substantial improvement over the 17.84% error of the Zhou et al. baseline.
  • Qualitative Sharpness: As seen in the visual results, CoopNet produces significantly sharper depth maps with fewer "bleeding" edges around thin objects and high-texture regions compared to competitive methods.

Conclusion

CoopNet demonstrates that managing the intrinsic bias between different self-supervised tasks is crucial for stable training. By shifting from a competitive to a cooperative paradigm via quantile-based gradient rebalancing, the model achieves state-of-the-art performance in depth and odometry without the need for external supervision like pre-trained semantic segmentation. While challenges remain in homogeneous areas, future work may explore combining this strategy with explicit residual motion correction to further refine the modeling of dynamic scene elements.

Summary updated

Abstract

We present CoopNet, an approach that improves the cooperation of co-trained networks by dynamically adapting the apportionment of gradient, to ensure equitable learning progress. It is applied to motion-aware self-supervised prediction of depth maps, by introducing a new hybrid loss, based on a distribution model of photo-metric reconstruction errors made by, on the one hand the depth + odometry paired networks, and on the other hand the optical flow network. This model essentially assumes that the pixels from moving objects (that must be discarded for training depth and odometry), correspond to those where the two reconstructions strongly disagree. We justify this model by theoretical considerations and experimental evidences. A comparative evaluation on KITTI and CityScapes datasets shows that CoopNet improves or is comparable to the state-of-the-art in depth, odometry and optical flow predictions.
BibTeX
@inproceedings{hariat2023rebalancing,
  author = {Marwane Hariat and Antoine Manzanera and David Filliat},
  title = {{Rebalancing gradient to improve self-supervised co-training of depth, odometry and optical flow predictions}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00132},
}
Figure