Learning to Collaborate in Decentralized Learning of Personalized Models

Shuangtong Li, Tianyi Zhou, Xinmei Tian, Dacheng Tao

Introduction

In distributed machine learning, the traditional objective has been to achieve consensus, where all participating devices converge on a single global model. While effective for homogeneous data, this approach falters in decentralized learning (DL) environments where data is non-identically and independently distributed (non-IID). For instance, a mobile device's image classifier should ideally reflect its user's specific preferences rather than a generic average. This paper proposes a paradigm shift from forced consensus to "learning to collaborate," enabling nodes to autonomously build personalized models through dynamic collaboration.

Problem Statement

Standard DL algorithms often struggle with data heterogeneity. When nodes with disparate tasks—such as one classifying birds and another classifying vehicles—are forced to agree on a single model, the result is often a "jack of all trades, master of none" that underperforms on local tasks. Existing personalization techniques frequently rely on central coordination or static network topologies. The core challenge addressed here is: how can decentralized nodes independently identify beneficial collaborators and adapt their communication strategies to optimize local performance?

Methodology: L2C and Meta-L2C

The authors introduce two frameworks to facilitate autonomous collaboration: L2C (Learning to Collaborate) and meta-L2C.

Overview of the L2C and Meta-L2C frameworks for decentralized personalized learning.

L2C (Learning to Collaborate)

In L2C, mixing weights ($\alpha$) for model aggregation are treated as learnable parameters rather than fixed values. After performing a local update, a node evaluates the resulting model on a local validation set. It then adjusts its mixing weights to prioritize updates from neighbors that successfully reduced its local validation loss. This feedback loop allows nodes to discover which neighbors provide the most relevant information for their specific task.

Meta-L2C

To accommodate dynamic networks where nodes or tasks may join frequently, the authors developed meta-L2C. This version employs an attention mechanism where an encoder maps model updates into compact representations. A dot-product attention module then calculates mixing weights based on the similarity between these representations. This allows the system to generalize to unseen tasks by identifying potential collaborators based on model update characteristics rather than historical performance alone.

Both methods incorporate topology pruning. Nodes eventually cease communication with neighbors assigned low collaboration scores, resulting in a sparse, communication-efficient network structure.

Experimental Setup

The frameworks were evaluated on three computer vision benchmarks: CIFAR-10, CIFAR-100, and MiniImageNet. These datasets were partitioned across 100 nodes to simulate non-IID environments. For example, in the CIFAR-10 setup, each node was restricted to seeing only two of the ten available classes. This partitioning ensures that while some task overlap exists, the global consensus model remains suboptimal for individual nodes.

Results and Discussion

The proposed methods demonstrate significant improvements in both accuracy and convergence speed compared to state-of-the-art baselines.

Performance comparison of L2C and meta-L2C against baselines on test accuracy and training loss.

Accuracy and Convergence

On CIFAR-10, meta-L2C achieved a test accuracy of 92.10%, outperforming standard decentralized SGD (83.01%) and personalized Federated Learning methods like Ditto (87.32%). Furthermore, L2C and meta-L2C reached these higher accuracy levels in significantly fewer communication rounds, highlighting their efficiency.

Fairness and Efficiency

The methods also improved performance for the "worst 10%" of nodes, ensuring that nodes with rare or difficult tasks still benefit from the network. Additionally, the topology pruning mechanism allowed the models to maintain peak performance while pruning up to 90% of network edges, drastically reducing communication overhead.

Conclusion

L2C and meta-L2C offer a robust solution for decentralized personalization by treating model aggregation as a learnable attention problem. By allowing nodes to choose their collaborators dynamically, these frameworks overcome the limitations of consensus-based learning on heterogeneous data. Future research will focus on reducing the computational overhead of meta-L2C encoders and enhancing security against potential malicious nodes in the network.

Summary updated

BibTeX
@inproceedings{li2022learning,
  author = {Shuangtong Li and Tianyi Zhou and Xinmei Tian and Dacheng Tao},
  title = {{Learning to Collaborate in Decentralized Learning of Personalized Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00954},
}
Figure