Sharing Knowledge without Sharing Data: Stitches can improve ensembles of disjointly trained models

Arthur Guijt, Dirk Thierens, Ellen M. Kerkhof, Jan Wiersma, Tanja Alderliesten, Peter A. N. BosmanarXiv 2025 · DOI · arXiv

Abstract

Collaborative deep learning in privacy-sensitive domains like healthcare is often hindered by the inability to share raw data and the logistical complexity of synchronous federated learning. This paper introduces "stitch-ensembles," a novel asynchronous framework for combining independently trained models. By learning small translation layers, or "stitches," between intermediate representations, the method enables effective knowledge sharing without data exchange. We demonstrate that stitch-ensembles mitigate the performance degradation typically experienced by data-rich institutions in standard ensembles and achieve results competitive with centralized data pooling.

Introduction

Deep learning thrives on massive, diverse datasets, yet in fields like healthcare, data is frequently siloed across institutions. Strict privacy regulations, such as GDPR and HIPAA, often make the sharing of raw patient data impossible. While several collaborative paradigms exist, they each carry significant trade-offs:

  1. Federated Learning (FL): While FL allows institutions to share weight updates rather than data, it requires synchronous, lockstep coordination between compute clusters, which is often logistically impractical.
  2. Weight Averaging: Averaging the weights of independently trained models is generally ineffective due to permutation symmetries in neural networks; unless models share the exact same initialization and architecture, averaging typically destroys performance.
  3. Deep Ensembles: Simple averaging of final predictions improves generalization but treats models as black boxes. Critically, if a high-performing model from a data-rich institution is ensembled with a mediocre model from a smaller institution, the larger institution often sees its local performance degrade.

This paper addresses the challenge of asynchronous collaboration: how to effectively combine pre-trained models from different institutions to improve generalization without compromising local performance or requiring raw data sharing.

Methodology

The authors propose a framework based on "Model Stitching," introducing two primary innovations to facilitate asynchronous model combination.

Stitch-Ensembles

Rather than combining models only at the output stage, stitch-ensembles integrate models at their intermediate layers. The authors insert a "stitch"—a simple 1x1 convolutional or linear layer—that translates the feature map of Network A into a format compatible with Network B. By averaging this stitched representation with Network B’s original internal features, the models share high-level semantic information mid-computation, allowing for more nuanced integration than simple output averaging.

Double-Batched Training

Traditionally, stitches are trained via direct matching, minimizing the Mean Squared Error (MSE) between the feature maps of two networks. However, this approach is often overly restrictive, as it ignores the fact that subsequent layers may be invariant to certain feature differences (e.g., negative values being zeroed out by a ReLU activation).

To solve this, the authors introduce Double-Batched Training. During the stitching process, the batch size is doubled to pass both the original reference features and the stitched features through the network simultaneously. This allows the stitch to learn which feature differences actually impact the final output, resulting in significantly more robust translation layers that do not degrade the performance of the host network.

Experimental Setup

The approach was evaluated using two pairs of real-world medical image segmentation datasets, simulating collaboration between two distinct institutions:

  • Pair 1 (Organ-at-Risk Segmentation): Cervical cancer MRI scans from the LUMC dataset (185 scans) and the AUMC dataset (68 scans).
  • Pair 2 (Polyp Segmentation): Endoscopy video frames from HyperKvasir (1000 frames) and CVC-ClinicDB (612 frames).

Key Insights and Results

The Data-Rich Penalty

A significant finding of the study is that standard ensembling often "punishes" the institution with the superior dataset. For instance, in the HyperKvasir/CVC-ClinicDB experiments, standard ensembling severely degraded the Dice score and Hausdorff Distance (HD95) for the HyperKvasir model on its own test set. Stitch-ensembles successfully recovered this lost performance, ensuring that collaboration remains strictly beneficial for all parties.

Robustness and Positioning

The double-batched training method proved essential for stability. While traditional direct-matching caused severe performance drops as more stitches were introduced, the double-batched approach maintained high accuracy. Furthermore, the authors identified a "sweet spot" for stitching: placing the translation layers in the later stages of the network (approximately 60-75% through the architecture) provided the optimal balance between local accuracy and global generalization.

Closing the Performance Gap

Stitch-ensembles effectively bridged the gap between basic ensembling and "ideal" scenarios, such as pooling all raw data into a single training set or using synchronous federated learning. This suggests that asynchronous stitching is a viable alternative to more logistically demanding collaborative methods.

Limitations and Future Work

Despite its successes, the method has limitations:

  • Inference Overhead: A stitched network requires running both parent models, increasing memory and compute requirements. Future research may utilize Knowledge Distillation to compress these ensembles into a single model.
  • Task Complexity: The current study focused on binary segmentation. Scaling to multi-class problems may require more complex stitch selection strategies.
  • Label Consistency: The method assumes consistent ground-truth definitions across institutions. If clinical protocols differ significantly, additional multi-task learning components may be necessary.

Conclusion

Stitch-ensembles provide a powerful mechanism for asynchronous collaboration in deep learning. By enabling feature-level knowledge sharing through robust, double-batched translation layers, institutions can improve their models' generalization capabilities without the privacy risks of data sharing or the coordination hurdles of federated learning.

Summary updated

BibTeX
@misc{guijt2025sharing,
  author = {Arthur Guijt and Dirk Thierens and Ellen M. Kerkhof and Jan Wiersma and Tanja Alderliesten and Peter A. N. Bosman},
  title = {{Sharing Knowledge without Sharing Data: Stitches can improve ensembles of disjointly trained models}},
  year = {2025},
  eprint = {2512.17592},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.17592},
}
Figure