Assisted Learning: A Framework for Multi-Organization Learning

Xun Xian, Xinran Wang, Jie Ding, Reza Ghanadan

TL;DR

Assisted Learning (AL) is a novel decentralized framework enabling organizations to collaborate on machine learning tasks without sharing private data, proprietary models, or even the specific nature of their tasks. By iteratively exchanging only task-specific residuals and statistics, participants can achieve "near-oracle" performance—matching the accuracy they would have reached if all their private data had been centralized in a single database.

The Challenge of Siloed Data

In the era of big data, valuable information is often siloed across different organizations. For example, a hospital (Alice) might hold patient health records, while a clinical lab (Bob) holds specialized test results for those same patients. To build the most accurate predictive model, Alice needs Bob’s data.

However, two major hurdles prevent this collaboration:

  1. Privacy and Security: Strict regulations and proprietary interests prevent organizations from sharing raw data.
  2. Model Secrecy: Organizations often treat their specific machine learning algorithms as proprietary "black boxes" and refuse to reveal their internal logic.

Existing solutions fall short. Federated Learning typically requires a central coordinator and often assumes all participants use the same model architecture. Secure Multi-party Computation can be computationally intractable (too slow or expensive). The authors propose Assisted Learning to bridge this gap, focusing specifically on "vertically partitioned" data—where different organizations hold different features for the exact same set of individuals.

How Assisted Learning Works

The authors introduce a two-stage system that is entirely "model-free." This means Alice and Bob can use completely different types of models (e.g., Alice uses a Random Forest while Bob uses a Neural Network).

Stage I: The Learning Protocol

This stage relies on an iterative feedback loop:

  1. Alice starts by training a model on her local data.
  2. She calculates her "residuals" (the errors her model made) and sends only these residuals to Bob.
  3. Bob treats Alice's residuals as his new "labels." He trains his own model to predict these errors using his private features, then sends his fitted values back to Alice.
  4. Alice incorporates Bob's feedback to refine her predictions. This cycle repeats until the model converges.

Stage II: The Prediction Protocol

When a new patient arrives, Alice queries Bob’s model through an API. Bob provides a prediction based solely on his features, which Alice then combines with her own model's output to produce the final, highly accurate result.

Datasets and Evaluation

The researchers evaluated their framework using two primary sources:

  1. Friedman1: A standard synthetic regression benchmark used to test how well the framework handles non-linear relationships.
  2. MIMIC3 (Medical Information Mart for Intensive Care III): A massive real-world clinical database. They specifically looked at the "Length of Stay" (LOS) task, treating different data tables (Lab, Output, Input) as if they belonged to separate, non-sharing organizations.
MIMIC3 Assisted Learning Setup

Key Insights and Results

  • The "Oracle" Target: The most striking insight is that AL can converge to "oracle performance"—the theoretical maximum accuracy possible if all data were merged into a single dataset. In synthetic tests (Friedman1), AL's error rate quickly converged to this oracle level across linear regression, gradient boosting, and neural networks.
  • Real-World Success: On the MIMIC3 medical benchmark, AL significantly outperformed "Stacking" (a common ensemble method). In one configuration, AL achieved an error score of 108.8, while the best stacking method lagged behind at 114.2.
  • Model Agnosticism: Unlike many collaborative frameworks, AL doesn't care what's inside the assisting organization's "black box." This makes it highly practical for a real-world market where companies use diverse, proprietary tools.
  • Residuals as Information: By sharing residuals rather than raw data or gradients, the framework minimizes the "leakage" of sensitive information while still providing the necessary signal for the other party to help.
  • The Complexity Trade-off: The number of "rounds of assistance" acts similarly to model complexity. Too few rounds lead to underfitting; too many can eventually lead to overfitting, mirroring the classical bias-variance tradeoff.

Limitations and Future Work

While promising, the framework has areas for future exploration:

  • Adversarial Risks: A malicious participant could potentially use "model extraction" techniques (repeatedly querying the API) to try and steal a partner's private model logic.
  • Task Scope: The current paper focuses heavily on regression (predicting continuous numbers). Extending this to classification and clustering is a key area for future research.
  • Peer Selection & Incentives: The paper assumes Alice already knows Bob can help. In a real-world "learning market," Alice would need a way to statistically test which organizations are worth collaborating with. Furthermore, concrete incentive structures for organizations to assist each other still need to be developed.

Summary updated

Abstract

In an increasing number of AI scenarios, collaborations among different organizations or agents (e.g., human and robots, mobile units) are often essential to accomplish an organization-specific mission. However, to avoid leaking useful and possibly proprietary information, organizations typically enforce stringent security constraints on sharing modeling algorithms and data, which significantly limits collaborations. In this work, we introduce the Assisted Learning framework for organizations to assist each other in supervised learning tasks without revealing any organization's algorithm, data, or even task. An organization seeks assistance by broadcasting task-specific but nonsensitive statistics and incorporating others' feedback in one or more iterations to eventually improve its predictive performance. Theoretical and experimental studies, including real-world medical benchmarks, show that Assisted Learning can often achieve near-oracle learning performance as if data and training processes were centralized.
BibTeX
@inproceedings{xian2020assisted,
  author = {Xun Xian and Xinran Wang and Jie Ding and Reza Ghanadan},
  title = {{Assisted Learning: A Framework for Multi-Organization Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=2qAmyDhKjkU},
}
Figure