AutoRank: MCDA Based Rank Personalization for LoRA-Enabled Distributed Learning

Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, Xin Chen, Albert Y. ZomayaarXiv 2024 · DOI · arXiv

Introduction

Distributed learning, particularly Federated Learning (FL), faces significant hurdles due to the inherent heterogeneity of edge devices. Participants often possess varying computational resources and, more critically, non-IID (Independent and Identically Distributed) data. While Low-Rank Adaptation (LoRA) has become a standard for efficient model fine-tuning by updating only low-rank matrices, current implementations typically enforce a uniform rank across all clients. This "one-size-fits-all" approach leads to a "double imbalance": under-fitting for clients with complex data and over-fitting for those with simpler datasets.

Comparison of standard distributed learning versus LoRA-enabled distributed learning, highlighting the matrix decomposition approach.

The AutoRank Framework

AutoRank addresses this imbalance by automating rank selection through Multi-Criteria Decision Analysis (MCDA). The framework treats the assignment of LoRA ranks as a decision problem based on the specific data characteristics of each participant.

Data Complexity Estimation

The core of AutoRank lies in its ability to quantify data complexity without requiring centralized access to raw data. It utilizes three primary metrics:

  1. Dataset Learning Difficulty: Calculated using the information entropy of local training loss over initial epochs. High entropy indicates more challenging data.
  2. Distribution Estimation: A metric combining label uniformity and sample size.
  3. Gini-Simpson Index: A measure of label diversity robust against rare classes.

Automated Weighting and Ranking

To synthesize these metrics, AutoRank employs the CRITIC method to objectively weight criteria based on their correlation and contrast. Finally, the TOPSIS algorithm aggregates these weighted metrics to assign a final "rank scale ratio" to each participant, determining the size of their local LoRA matrices relative to a global baseline.

Theoretical Motivation and Performance

The necessity of personalized ranking is rooted in the bias-variance tradeoff. As model complexity (rank) increases, bias typically decreases while variance increases. AutoRank dynamically finds the "sweet spot" for each client, minimizing total error by scaling rank with data complexity.

Left: Test accuracy vs. communication rounds showing the superiority of heterogeneous ranking (HetRank) in non-IID settings. Right: Theoretical bias-variance curve illustrating the optimal model complexity.

Key Results

Empirical evaluations on benchmarks like MNIST, CIFAR-10, and CINIC-10 demonstrate that AutoRank consistently outperforms homogeneous baselines:

  • Higher Accuracy: On the MNIST dataset, AutoRank achieved 96.15% accuracy, surpassing the best baseline (95.87%).
  • Resource Efficiency: Compared to standard homogeneous settings, the heterogeneous rank assignment reduced communication and computation costs by approximately 45%.
  • Faster Convergence: As shown in the performance curves, the heterogeneous approach (HetRank) reaches target accuracy levels in significantly fewer communication rounds compared to standard Federated Averaging (FedAvg) in non-IID scenarios.

Conclusion

AutoRank provides a robust, automated solution for the "double imbalance" problem in LoRA-enabled distributed learning. By leveraging MCDA to personalize model complexity, it achieves a superior balance between accuracy and resource efficiency. Future work aims to integrate hardware constraints, such as memory and power limits, into the ranking criteria to further enhance real-world deployability on resource-constrained edge devices.

Summary updated

BibTeX
@misc{chen2024autorank,
  author = {Shuaijun Chen and Omid Tavallaie and Niousha Nazemi and Xin Chen and Albert Y. Zomaya},
  title = {{AutoRank: MCDA Based Rank Personalization for LoRA-Enabled Distributed Learning}},
  year = {2024},
  eprint = {2412.15553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.15553},
}
Figure