TL;DR
StereoFactory introduces a two-stage evolutionary framework for post-hoc model merging in stereo matching, bypassing the need for costly joint retraining when new datasets are introduced. By first selecting an optimal subset of source models via a genetic algorithm and then optimizing module-level routing weights using CMA-ES, it significantly reduces cross-domain generalization errors at a fraction of the computational cost.
What Problem Does This Paper Address?
Modern stereo matching systems achieve impressive zero-shot generalization by training foundation models on massive, aggregated datasets. However, this paradigm faces a severe scalability bottleneck: whenever a new data source becomes available, practitioners must retrain the entire model from scratch on the union of all datasets, incurring massive computational overhead.
Model merging offers a post-hoc alternative by combining the weights of independently fine-tuned models without retraining. Unfortunately, existing merging methods suffer from two main flaws:
- The "Merge-All" Assumption: They assume that incorporating more models is always better, which often introduces harmful interference between conflicting task vectors.
- Monolithic Granularity: They apply a single set of merging coefficients globally across the entire network, ignoring the fact that different functional modules (e.g., feature extraction vs. disparity refinement) specialize in and prefer different knowledge sources.
Methods
The authors propose StereoFactory, a coarse-to-fine evolutionary framework that decouples model merging into two distinct optimization problems:
- Stage 1: Evolutionary Subset Search (What to Merge): A Genetic Algorithm (GA) navigates the discrete combinatorial space of available models to select a compact, complementary subset of checkpoints, minimizing interference from detrimental data sources.
- Stage 2: Module-Level Routing Optimization (How to Merge): Using the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), the framework performs continuous optimization to find block-specific routing weights and structural scaling factors. This allows different modules within the network to favor different knowledge sources and adaptation strengths.
Datasets
- Base Model Pre-training: SceneFlow
- Source Checkpoints (11 fine-tuning datasets): Sintel, FallingThings, VirtualKITTI2, TartanAir, InStereo2K, UnrealStereo4K, CREStereo, Spring, DynamicReplica, Carla, and the FoundationStereo training dataset.
- Evaluation Benchmarks: Four standard zero-shot benchmarks were used: ETH3D (also serving as the validation set for the evolutionary search), KITTI 2012, KITTI 2015, and Middlebury.
Key Insights and Ideas
- More is Not Always Better: Merging all eleven specialized models yields suboptimal results. Selectively merging a smaller subset achieves stronger average generalization because it filters out datasets that introduce conflicting representations.
- Module-Level Knowledge Specialization: Neural network modules exhibit distinct preferences. For example, downstream disparity refinement benefits from stronger task-specific adaptation weights, whereas matching cost computation modules rely more heavily on the original pretrained base representations.
- Cross-Architecture Transferability: An optimal subset of datasets discovered on one architecture (e.g., NMRF) transfers to a completely different architecture (e.g., FoundationStereo) with minimal degradation. This indicates that dataset complementarity is largely architecture-independent, allowing a single search to serve multiple deployment targets.
Key Results
- State-of-the-Art Generalization: StereoFactory consistently achieved the lowest average error across all four benchmarks compared to controlled baselines (including Model Soups, TIES-Merging, and DARE). It reduced average error from 3.80 to 3.30 on the NMRF architecture and from 2.88 to 2.19 on FoundationStereo.
- Massive Efficiency Gains: The post-hoc search requires only 2.7% to 3.7% of the wall-clock time of conventional joint retraining. For instance, instead of a 128.7-hour joint retrain for FoundationStereo, StereoFactory achieves superior fusion in just 3.5 hours.
Limitations and Future Work
- Validation Set Dependency: The performance of the merged model depends on the representativeness of the validation set used during the evolutionary search. For highly specialized target domains, a domain-aligned validation split is required.
- Weight-Space Compatibility: The framework assumes that all source checkpoints share the exact same architecture and base initialization.
- Future Directions: The authors plan to explore automating the block partitioning process, combining adaptive selection with explicit interference-reduction techniques, and extending the framework to other vision tasks.
Summary updated