Self-Assessed Generation: Trustworthy Label Generation for Optical Flow and Stereo Matching in Real-world

Han Ling, Yinghui Sun, Quansen Sun, Ivor W. Tsang, Yuhui ZhengarXiv 2024 · OpenReview

TL;DR

This paper introduces Self-Assessed Generation (SAG), a novel data-driven framework for generating high-quality, self-supervised training data for optical flow and stereo matching tasks. SAG leverages advanced 3D reconstruction techniques and a unique "self-assessment" mechanism to create accurate labels from real-world RGB images, significantly improving generalization performance without altering existing network architectures or loss functions.

What Problem Does This Paper Address?

Training robust optical flow and stereo matching models for real-world applications is hampered by the high cost and difficulty of creating large-scale, accurately labeled datasets. Existing self-supervised methods often produce fuzzy results, struggle with complex scenarios like occlusions, and are limited to training lightweight networks. This paper tackles the challenge of generating reliable, cost-effective training data that bridges the gap between synthetic and real-world performance for these crucial computer vision tasks.

Methods

SAG is a unified framework with several key components:

  • 3D Scene Reconstruction: It utilizes state-of-the-art techniques like NeRF (Neural Radiance Fields) and 3DGS (3D Gaussian Splatting) to reconstruct 3D scenes from user-provided monocular RGB images or videos.
  • Data Generation: Based on the reconstructed 3D scenes and controllable camera poses, SAG generates synthetic RGB image pairs and corresponding optical flow or stereo labels.
  • Self-Assessment Mechanism: This is a core innovation. SAG quantifies the confidence of the generated labels using metrics like reconstruction confidence (RC), geometric consistency (GC), and visual structural similarity (VSS). This allows for the filtering of inaccurate or defective generated data.
  • 3D Flight Foreground Rendering Pipeline: To address the lack of realistic motion and occlusion in generated data, SAG incorporates a pipeline that renders and overlays 3D foreground objects, encouraging networks to learn challenging scenarios.

Datasets

  • Training Data: The paper describes collecting and constructing approximately 300 real-world scenes (indoor still life, outdoor courtyards). These scenes are used to generate datasets like GS58 (approximately 58,800 image pairs) and GS15 (15,800 image pairs).
  • Evaluation Datasets: Standard benchmarks for optical flow and stereo matching are used, including KITTI (K15, K12, Kraw), Sintel, and Middlebury (Midd-A, Midd-T). Pre-training datasets like Flyingthings3D (T), Flyingchairs (C, C2), Driving (D), and Monkaa (M) are also mentioned.

Key Insights and Ideas

  • Data-Driven Self-Supervision is Key: The paper strongly advocates for a data-driven approach to self-supervised learning, where the quality and accuracy of the generated labels are paramount, rather than relying solely on indirect loss functions.
  • "Self-Assessment" for Trustworthy Labels: The novel confidence metrics (RC, GC, VSS) are crucial for filtering out generated data that might otherwise introduce noise or errors, leading to more robust models.
  • 3DGS Outperforms NeRF for Generation: The study finds that 3DGS offers significantly better geometric consistency and much faster rendering times compared to NeRF for generating training data, making it a more practical choice.
  • Median Depth is Superior to Mean Depth: The paper demonstrates that using median depth for label generation is more robust to reconstruction artifacts and multiple surfaces compared to the traditional mean depth.
  • 3D Foreground Injection Boosts Performance: Adding rendered 3D foregrounds significantly improves the ability of models to handle occlusions and dynamic elements.

Key Results

  • State-of-the-Art Generalization: SAG-trained models achieve state-of-the-art or highly competitive performance on real-world optical flow and stereo matching benchmarks, particularly in zero-shot generalization scenarios.
  • Significant Improvement on Sintel: SAG reduced optical flow endpoint errors by 48% on the Sintel public test platform compared to previous methods.
  • Cost-Effective and Efficient: SAG allows for the generation of large-scale, high-quality datasets from readily available monocular data at a low cost and without manual intervention.
  • Compatibility: The framework is compatible with various reconstruction methods (NeRF, 3DGS) and can directly train state-of-the-art deep networks without modification.
  • Improved Occlusion Handling: The 3D flight foreground pipeline demonstrably improves the model's ability to handle occluded areas.

Limitations and Future Work

  • Reconstruction Limitations: The current reconstruction methods used in SAG struggle with reflective surfaces (like glass), leading to artifacts.
  • Static Scene Assumption: The current framework is primarily designed for static scenes, limiting its direct application to dynamic driving scenarios.
  • Future Work: The authors suggest that the limitations related to reflective surfaces and dynamic scenes are being actively addressed by ongoing research in NeRF and 3DGS methods, and SAG can benefit from these advancements. They also highlight the potential for SAG to be extended to more complex scenarios and to serve as a foundation for large-scale matching models.

Summary updated

Abstract

A significant challenge facing current optical flow and stereo methods is the difficulty in generalizing them well to the real world. This is mainly due to the high costs required to produce datasets, and the limitations of existing self-supervised methods on fuzzy results and complex model training problems. To address the above challenges, we propose a unified self-supervised generalization framework for optical flow and stereo tasks: Self-Assessed Generation (SAG). Unlike previous self-supervised methods, SAG is data-driven, using advanced reconstruction techniques to construct a reconstruction field from RGB images and generate datasets based on it. Afterward, we quantified the confidence level of the generated results from multiple perspectives, such as reconstruction field distribution, geometric consistency, and structural similarity, to eliminate inevitable defects in the generation process. We also designed a 3D flight foreground automatic rendering pipeline in SAG to encourage the network to learn occlusion and motion foreground. Experimentally, because SAG does not involve changes to methods or loss functions, it can directly self-supervised train the state-of-the-art deep networks, greatly improving the generalization performance of self-supervised methods on current mainstream optical flow and stereo-matching datasets. Compared to previous training modes, SAG is more generalized, cost-effective, and accurate.
BibTeX
@misc{ling2024self,
  author = {Han Ling and Yinghui Sun and Quansen Sun and Ivor W. Tsang and Yuhui Zheng},
  title = {{Self-Assessed Generation: Trustworthy Label Generation for Optical Flow and Stereo Matching in Real-world}},
  year = {2024},
  eprint = {2410.10453},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.10453},
}
Figure