Federated Communication-Efficient Multi-Objective Optimization

Baris Askin, Pranay Sharma, Gauri Joshi, Carlee Joe-Wong

Abstract

We study a federated version of multi-objective optimization (MOO), where a single model is trained to optimize multiple objective functions. MOO has been extensively studied in the centralized setting but is less explored in federated or distributed settings. We propose FedCMOO, a novel communication-efficient federated multi-objective optimization (FMOO) algorithm that improves the error convergence performance of the model compared to existing approaches. Unlike prior works, the communication cost of FedCMOO does not scale with the number of objectives, as each client sends a single aggregated gradient, obtained using randomized SVD (singular value decomposition), to the central server. We provide a convergence analysis of the proposed method for smooth non-convex objective functions under milder assumptions than in prior work. In addition, we introduce a variant of FedCMOO that allows users to specify a preference over the objectives in terms of a desired ratio of the final objective values. Through extensive experiments, we demonstrate the superiority of our proposed method over baseline approaches.

Introduction

In many real-world machine learning applications, models must satisfy multiple, often conflicting, objectives simultaneously. For instance, a molecular property predictor might need to estimate a dozen different chemical constants, or a facial recognition system might need to maintain high accuracy across diverse demographic groups. This is the domain of Multi-Objective Optimization (MOO).

As data privacy becomes paramount, Federated Learning (FL) has emerged as a standard for training models on decentralized data. However, bringing MOO into the FL setting introduces two critical challenges: communication explosion and local objective drift. Traditional federated MOO methods require clients to upload separate gradients for every task, leading to bandwidth costs that scale linearly with the number of objectives. Furthermore, if clients optimize tasks independently before aggregation, the local models tend to drift toward specific task solutions, hindering global convergence. This paper introduces FedCMOO to address these bottlenecks.

The FedCMOO Framework

FedCMOO (Federated Communication-efficient Multi-Objective Optimization) optimizes the Pareto-stationary solution through three primary innovations:

1. Randomized SVD-based Jacobian Approximation

To find an optimal descent direction that balances all tasks, the server typically requires the full Jacobian matrix (gradients for all $M$ tasks). FedCMOO avoids this by using a randomized Singular Value Decomposition (SVD) approach. Clients compress their local Jacobian into a few principal components. This ensures that the upload cost remains $\Theta(d)$—the same as standard single-objective federated learning—regardless of whether there are 2 or 40 objectives.

2. Weighted Local Training

To combat local objective drift, FedCMOO moves the task-balancing logic to the client side. The server provides a set of weights, and clients perform local SGD on the weighted sum of all objectives. This keeps the local updates aligned with the global compromise path, significantly improving stability and convergence speed.

Comparison of mean test accuracy across varying local iterations, demonstrating FedCMOO's robustness to local drift compared to FSMGDA.

3. Preference-Aware Optimization (FedCMOO-Pref)

In many scenarios, users have specific priorities. FedCMOO-Pref allows users to define a desired ratio of final objective values. The algorithm incorporates this by minimizing the KL divergence between the current loss distribution and the target preference distribution, ensuring the final model respects user-defined priorities.

Experimental Evaluation

The researchers evaluated FedCMOO against state-of-the-art baselines like FSMGDA across several benchmarks, including MultiMNIST, CelebA (up to 40 attributes), and the QM9 molecular dataset (11 chemical properties).

Key Results

  • Communication Efficiency: On the QM9 dataset, FedCMOO reached a lower mean test loss while using approximately 10% of the total upload data required by FSMGDA. This confirms that the randomized SVD approximation successfully decouples communication cost from the number of tasks.
  • Robustness to Local Computation: As shown in the MultiMNIST and MNIST+FMNIST experiments, FedCMOO maintains high accuracy even as the number of local iterations increases. In contrast, the baseline FSMGDA suffers from significant performance degradation due to local drift.
  • Theoretical Soundness: The authors prove that FedCMOO achieves a convergence rate of $O(1/\sqrt{T})$ for non-convex smooth functions, matching the best-known rates for single-objective federated learning.
FedCMOO demonstrates superior convergence and communication efficiency on the QM9 dataset compared to the FSMGDA baseline.

Conclusion

FedCMOO represents a significant step forward for federated multi-task and multi-objective learning. By leveraging randomized SVD for Jacobian approximation and employing weighted local training, it eliminates the linear scaling of communication costs and mitigates the destructive effects of local drift. Future work may explore variance reduction techniques to further stabilize training and investigate strategies for handling tasks with varying difficulty levels.

Summary updated

Abstract

We study a federated version of multi-objective optimization (MOO), where a single model is trained to optimize multiple objective functions. MOO has been extensively studied in the centralized setting but is less explored in federated or distributed settings. We propose FedCMOO, a novel communication-efficient federated multi-objective optimization (FMOO) algorithm that improves the error convergence performance of the model compared to existing approaches. Unlike prior works, the communication cost of FedCMOO does not scale with the number of objectives, as each client sends a single aggregated gradient, obtained using randomized SVD (singular value decomposition), to the central server. We provide a convergence analysis of the proposed method for smooth non-convex objective functions under milder assumptions than in prior work. In addition, we introduce a variant of FedCMOO that allows users to specify a preference over the objectives in terms of a desired ratio of the final objective values. Through extensive experiments, we demonstrate the superiority of our proposed method over baseline approaches.
BibTeX
@inproceedings{askin2025federated,
  author = {Baris Askin and Pranay Sharma and Gauri Joshi and Carlee Joe-Wong},
  title = {{Federated Communication-Efficient Multi-Objective Optimization}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=kiVz2xXjEm},
}

Loading related content...

Figure