Reusable Architecture Growth for Continual Stereo Matching

Chenghao Zhang, Gaofeng Meng, Bin Fan, Kun Tian, Zhaoxiang Zhang, Shiming Xiang, Chunhong Pan

Introduction

Deep learning models for stereo depth estimation—the process of calculating distance using two cameras—are essential for autonomous navigation. However, these models are notoriously brittle when faced with environmental shifts. A model trained on sunny highways often fails when it encounters a rainy city street. While fine-tuning on new data is a common approach, it leads to catastrophic forgetting, where the model "erases" its previous knowledge to accommodate new information.

To address this, researchers have introduced Reusable Architecture Growth (RAG), a framework designed for continual stereo matching. RAG allows models to learn new environments incrementally, preventing the loss of old skills while maintaining a manageable model size through intelligent resource sharing.

The RAG Framework

The RAG framework operates across three distinct levels to balance stability and plasticity:

Instead of relying on a fixed, one-size-fits-all architecture, RAG employs Neural Architecture Search (NAS) to find the optimal "neural units" (cells) for a specific scene. This search is applied to both the Feature Net, which extracts image details, and the Matching Net, which compares left and right images to determine depth.

2. Network-Level Growth

When the model encounters a new environment, it does not simply add a duplicate network. Instead, it evaluates the architecture layer-by-layer to decide whether to reuse a cell from a previous task or grow a new one. To ensure "zero forgetting," the parameters of previously learned cells are frozen, while new cells are optimized for the current task. This approach results in an Average Reuse Rate (ARR) of approximately 50%, meaning the model effectively leverages existing knowledge rather than redundantly expanding.

3. Scene Router and Inference

At inference time, the system must automatically determine which architectural path to activate. RAG utilizes a Scene Router composed of task-specific autoencoders. By incorporating a scene contrastive loss, the router avoids confusion between similar-looking environments (e.g., cloudy vs. sunny conditions), ensuring the most appropriate specialized path is selected for the input images.

Addressing Data Scarcity: Proxy-Supervision

In many real-world scenarios, ground-truth depth labels are unavailable. RAG overcomes this through Proxy-Supervision. The researchers use style-transferred synthetic data—taking synthetic images with known depth and modifying their appearance to match the target real-world scene. This provides a high-quality starting point for self-supervised learning in new domains.

Experimental Results

The framework was evaluated using major benchmarks including DrivingStereo (weather changes), KITTI Raw, and CityScapes.

Zero Forgetting and High Accuracy

Because RAG freezes the parameters of older tasks, it achieves a Backward Transfer (BWT) of 0.0, meaning there is no performance degradation on previously learned scenes. On the DrivingStereo dataset, RAG achieved a Final Average Error (FAE) of 0.637 EPE (End-Point Error), significantly outperforming established continual learning methods like EWC (1.053) and iCaRL (0.866).

Architecture vs. Weights

A key insight from the study is that in self-supervised settings, searching for the right structure for a scene is more effective than merely adapting the weights of a fixed model. Furthermore, RAG often outperforms "Joint Training" (training on all data simultaneously). This suggests that specialized "local" architectural solutions can be more effective than a single global architecture trying to accommodate highly diverse scenes.

Conclusion and Future Work

Reusable Architecture Growth provides a robust solution for the challenges of environmental adaptation in stereo matching. By combining neural architecture search with a modular growth strategy, RAG ensures zero forgetting and high efficiency. Beyond stereo matching, the authors successfully extended the framework to monocular depth estimation and 3D object detection, demonstrating its versatility. Future research will focus on further compacting reused units and refining the Scene Router to handle entirely unseen environments without prior proxy training.

Summary updated

BibTeX
@article{zhang2024reusable,
  author = {Chenghao Zhang and Gaofeng Meng and Bin Fan and Kun Tian and Zhaoxiang Zhang and Shiming Xiang and Chunhong Pan},
  title = {{Reusable Architecture Growth for Continual Stereo Matching}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3378884},
}
Figure