Abstract
Multi-party learning provides solutions for training joint models with decentralized data under legal and practical constraints. However, traditional multi-party learning approaches are confronted with obstacles such as system heterogeneity, statistical heterogeneity, and incentive design. How to deal with these challenges and further improve the efficiency and performance of multi-party learning has become an urgent problem to be solved. In this paper, we propose a novel contrastive multi-party learning framework for knowledge refinement and sharing with an accountable incentive mechanism. Since the existing naive model parameter averaging method is contradictory to the learning paradigm of neural networks, we simulate the process of human cognition and communication, and analogy multi-party learning as a many-to-one knowledge sharing problem. The approach is capable of integrating the acquired explicit knowledge of each client in a transparent manner without privacy disclosure, and it reduces the dependence on data distribution and communication environments. The proposed scheme achieves significant improvement in model performance in a variety of scenarios, as we demonstrated through experiments on several real-world datasets.
Introduction
In the era of strict privacy regulations like GDPR, training AI models on data scattered across different organizations—often referred to as "data silos"—presents a significant challenge. Multi-party learning, or Federated Learning (FL), was designed to address this by allowing participants to train models locally and share only model updates with a central server.
However, the industry standard, Federated Averaging (FedAvg), faces three critical hurdles:
- Statistical Heterogeneity: Non-IID (non-identical) data distributions across clients often lead to poor global model performance.
- System Heterogeneity: Synchronous systems are often bottlenecked by "stragglers" (slow devices).
- Lack of Interpretability: Averaging millions of numerical weights is a "black box" process that obscures what is actually being learned from each participant.
Methods: The Contrastive Knowledge Sharing Framework
This paper introduces a "bionic" approach inspired by human communication. Rather than swapping raw model parameters, the framework allows clients to share "refined knowledge" in the form of feature distributions. The local model is decomposed into three functional modules:
- Cognitive Module (The Analyzer): An encoder that uses Contrastive Learning to extract high-order features, pulling representations of the same class closer in latent space while pushing different classes apart.
- Descriptive Module (The Memory): A generator that learns to mimic the distribution of features produced by the Cognitive Module. Clients send this generator to the server instead of raw data or weights.
- Discriminative Module (The Judge): A classifier that predicts labels based on the extracted features.
The Workflow
Instead of averaging weights, the server receives the Descriptive Modules (generators) from clients. It uses these to create synthetic feature embeddings and trains a central Discriminative Module. This aggregated knowledge is then redistributed to clients to unify their local cognition, avoiding the "permutation problem" where averaging different neural network parameters results in incoherent models.
Experimental Evaluation
The framework was evaluated on five diverse datasets: MNIST & FEMNIST (handwritten digits), CIFAR-10 (objects), 20 Newsgroups (text classification), and SST-5 (fine-grained sentiment analysis).
Key Results
As shown in the performance metrics, the proposed method consistently outperforms traditional FedAvg in both convergence speed and final accuracy across various data types.
- Superior Speed: In environments with 80% stragglers, the method reached 70% accuracy 3.45x faster than FedAvg.
- Robustness to Non-IID Data: On the difficult SST-5 dataset, the framework maintained strong performance where FedAvg struggled to converge due to uneven data partitioning.
- Security against Poisoning: The server can measure contribution quality by analyzing the dispersion of features from a client's generator. In label-flipping attacks, this filtering kept accuracy stable (dropping only ~1%), while FedAvg’s performance dropped by over 7%.
Discussion and Future Work
The framework's success stems from its move away from parameter averaging toward sharing feature descriptions, mimicking how a teacher imparts concepts to a student. This provides a built-in incentive mechanism where the server can accountably reward high-quality contributions.
However, limitations remain. The system currently faces a "cold-start" problem for new participants with minimal local data. Future research will focus on deeper theoretical analysis of the loss functions, unsupervised encoding techniques to further enhance privacy, and optimizing performance in extreme cases of statistical heterogeneity.
Summary updated