StereoFactory: A Unified Merging Framework for Robust Stereo Matching

Xianda Guo, Pinhan Fu, Ruilin Wang, Wenke Huang, Mang Ye, Qin ZouarXiv 2026 · arXiv

TL;DR

StereoFactory introduces a two-stage evolutionary framework for post-hoc model merging in stereo matching, bypassing the need for costly joint retraining when new datasets are introduced. By first selecting an optimal subset of source models via a genetic algorithm and then optimizing module-level routing weights using CMA-ES, it significantly reduces cross-domain generalization errors at a fraction of the computational cost.

What Problem Does This Paper Address?

Modern stereo matching systems achieve impressive zero-shot generalization by training foundation models on massive, aggregated datasets. However, this paradigm faces a severe scalability bottleneck: whenever a new data source becomes available, practitioners must retrain the entire model from scratch on the union of all datasets, incurring massive computational overhead.

Model merging offers a post-hoc alternative by combining the weights of independently fine-tuned models without retraining. Unfortunately, existing merging methods suffer from two main flaws:

  1. The "Merge-All" Assumption: They assume that incorporating more models is always better, which often introduces harmful interference between conflicting task vectors.
  2. Monolithic Granularity: They apply a single set of merging coefficients globally across the entire network, ignoring the fact that different functional modules (e.g., feature extraction vs. disparity refinement) specialize in and prefer different knowledge sources.

Methods

The authors propose StereoFactory, a coarse-to-fine evolutionary framework that decouples model merging into two distinct optimization problems:

  1. Stage 1: Evolutionary Subset Search (What to Merge): A Genetic Algorithm (GA) navigates the discrete combinatorial space of available models to select a compact, complementary subset of checkpoints, minimizing interference from detrimental data sources.
  2. Stage 2: Module-Level Routing Optimization (How to Merge): Using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), the framework performs continuous optimization to find block-specific routing weights and structural scaling factors. This allows different modules within the network to favor different knowledge sources and adaptation strengths.
Fig. 2. Overview of the StereoFactory framework. Stage 1 searches for the optimal subset of fine-tuned checkpoints via evolutionary search with validation-set fitness. Stage 2 further refines the merging through architecture-adaptive routing across different architectural components.

Datasets

  • Base Model Pre-training: SceneFlow
  • Source Checkpoints (11 fine-tuning datasets): Sintel, FallingThings, VirtualKITTI2, TartanAir, InStereo2K, UnrealStereo4K, CREStereo, Spring, DynamicReplica, Carla, and the FoundationStereo training dataset.
  • Evaluation Benchmarks: Four standard zero-shot benchmarks were used: ETH3D (also serving as the validation set for the evolutionary search), KITTI 2012, KITTI 2015, and Middlebury.

Key Insights and Ideas

  • More is Not Always Better: Merging all eleven specialized models yields suboptimal results. Selectively merging a smaller subset achieves stronger average generalization because it filters out datasets that introduce conflicting representations.
  • Module-Level Knowledge Specialization: Neural network modules exhibit distinct preferences. For example, downstream disparity refinement benefits from stronger task-specific adaptation weights, whereas matching cost computation modules rely more heavily on the original pretrained base representations.
  • Cross-Architecture Transferability: An optimal subset of datasets discovered on one architecture (e.g., NMRF) transfers to a completely different architecture (e.g., FoundationStereo) with minimal degradation. This indicates that dataset complementarity is largely architecture-independent, allowing a single search to serve multiple deployment targets.

Key Results

  • State-of-the-Art Generalization: StereoFactory consistently achieved the lowest average error across all four benchmarks compared to controlled baselines (including Model Soups, TIES-Merging, and DARE). It reduced average error from 3.80 to 3.30 on the NMRF architecture and from 2.88 to 2.19 on FoundationStereo.
  • Massive Efficiency Gains: The post-hoc search requires only 2.7% to 3.7% of the wall-clock time of conventional joint retraining. For instance, instead of a 128.7-hour joint retrain for FoundationStereo, StereoFactory achieves superior fusion in just 3.5 hours.
Fig. 1. Motivation of StereoFactory. Lower is better for all metrics. Conventional mixed-data training retrains a single model on the union of all source datasets whenever new data sources are introduced. StereoFactory instead keeps independently trained source-specific checkpoints reusable and updates only the post-hoc merging stage through evolutionary selection and adaptive composition.

Limitations and Future Work

  • Validation Set Dependency: The performance of the merged model depends on the representativeness of the validation set used during the evolutionary search. For highly specialized target domains, a domain-aligned validation split is required.
  • Weight-Space Compatibility: The framework assumes that all source checkpoints share the exact same architecture and base initialization.
  • Future Directions: The authors plan to explore automating the block partitioning process, combining adaptive selection with explicit interference-reduction techniques, and extending the framework to other vision tasks.

Summary updated

Abstract

Stereo matching has advanced through foundation models trained on large-scale datasets, yet this paradigm suffers from a scalability bottleneck: incorporating new data requires costly joint retraining. Model merging offers a scalable post-hoc alternative by integrating knowledge from specialized models after source checkpoints are available. However, existing merging methods typically retain all available models or rely on greedy inclusion, which can preserve harmful task-vector interference. We propose StereoFactory, a coarse-to-fine evolutionary framework for adaptive model merging. Stage~1 employs a genetic algorithm to search the combinatorial space of model subsets, determining which models should participate. Stage~2 addresses module-level knowledge specialization (different functional modules exhibit distinct preferences for knowledge sources) through CMA-ES optimization of architecture-adaptive routing over the selected task vectors, with optional module-level scaling. Experiments across two architectures and four benchmarks demonstrate that StereoFactory consistently achieves the best four-benchmark average under the same checkpoint pool, reducing the average error from 3.80 to 3.30 on NMRF and from 2.88 to 2.19 on FoundationStereo relative to the strongest controlled baseline. The post-hoc search requires only 2.7--3.7\% of the corresponding joint-retraining wall-clock time. Analysis reveals that knowledge contributions are inherently module-specific, and selected subsets can transfer across architectures with minimal degradation. Code will be publicly released upon acceptance at: https://github.com/XiandaGuo/StereoFactory.
BibTeX
@misc{guo2026stereofactory,
  author = {Xianda Guo and Pinhan Fu and Ruilin Wang and Wenke Huang and Mang Ye and Qin Zou},
  title = {{StereoFactory: A Unified Merging Framework for Robust Stereo Matching}},
  year = {2026},
  eprint = {2606.17475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.17475},
}
Figure