TL;DR
Models trained for the same task often have complementary strengths, but traditional collaborative learning forces them to become more similar, destroying the very differences that make them useful. Corroborative Learning (CL) solves this by using one model's internal representations to "corroborate" (reconstruct) the changes in another model's predictions, filtering out noise while preserving unique, task-relevant capabilities.
What Problem Does This Paper Address?
In machine learning, we often have multiple models (different architectures, trained on different data) for the same task. These models usually have "complementary" knowledge—one might excel at recognizing rare entities while another is better at understanding complex syntax.
The standard way to make models learn from each other is through alignment (e.g., Knowledge Distillation or Mutual Learning), where we penalize models for disagreeing. However, the authors identify a "structural trade-off": if you force two models to agree perfectly, you eliminate the diversity that made them complementary in the first place. As shown in Figure 1(a), two models can have nearly identical scores but succeed on very different examples.
The challenge is: How can models improve one another without losing their unique "secret sauce"?
Methods
The authors propose Corroborative Learning (CL). Instead of forcing Model A to match Model B's output, CL asks: "Can Model B's internal state explain the changes Model A is making to its predictions?"
Key components include:
- Task-Response Increments: Each model learns a small additive change ($\Delta$) to its original task scores via a lightweight adapter.
- Source-Excluded Reconstruction: To prevent the models from just memorizing training noise, the data is split into groups. The system fits a relationship between Model B’s features and Model A’s score changes on most groups, then evaluates how well that relationship holds on a "held-out" group.
- Directional Regularization: If a change in Model A's scores can be reconstructed by Model B across different data groups, it's considered "corroborated" and allowed. If it can't be reconstructed, it's treated as noise and suppressed.
- Independent Inference: Crucially, the collaborator model is only used during training. At test time, each model remains completely independent, with no extra computational cost.
Datasets
The authors evaluated CL across a wide range of tasks and architectures:
- NLP Benchmarks: Re-TACRED and CrossRE (relation extraction), MNLI (natural language inference), Re-DocRED (document-level RE), and ACE05/WikiEvents (event argument extraction).
- Vision Benchmarks: ImageNet-1K and ImageNetV2.
- Models: A diverse pool including LUKE, RoBERTa, KnowPrompt, ResNet-50, ConvNeXt, and Vision Transformers (ViT).
Key Insights and Ideas
- Reconstructability vs. Similarity: The core "aha" moment is that a collaborator doesn't need to agree with the target to be useful; it just needs to provide a "second opinion" that can predict which changes are likely to generalize.
- Filtering Noise: The mathematical analysis shows that CL acts as a "directional regularizer." It selectively shrinks updates in directions that lack cross-model support, which effectively suppresses idiosyncratic errors (noise) while keeping task-relevant signals.
- Effective with Less Data: CL shines in low-label scenarios. When only 10% of training labels are available, the performance gains from collaboration are even more pronounced, as the models rely more on each other to distinguish signal from noise.
Key Results
- Broad Improvements: CL improved performance in 86.1% of the 72 directed model pairs tested.
- Superiority over Baselines: It consistently outperformed standard techniques like Knowledge Distillation (KD) and Deep Mutual Learning (DML). In a head-to-head "pairwise win" count across all tasks, CL won 827 out of 960 comparisons against ten other methods.
- Cross-Domain Success: The method generalized well from NLP to computer vision, showing that the principle of corroboration is fundamental to pattern analysis, not just language.
- Efficiency: While training takes about 2.2x longer than independent training, the inference time is identical to the base model, making it highly practical for deployment.
Limitations and Future Work
- Training Overhead: The cross-group reconstruction process is computationally intensive during the training phase.
- Pairwise Focus: The current study focuses on pairs of models. The authors note that extending this to "model zoos" (collaborating with many models simultaneously) or developing automatic ways to select the best "peer" for a given target are logical next steps.
- Group Sensitivity: The effectiveness of the "source-excluded" mechanism depends on how the data is partitioned into groups, which currently requires some domain knowledge or metadata.
Summary updated