Abstract
We study a federated version of multi-objective optimization (MOO), where a single model is trained to optimize multiple objective functions. MOO has been extensively studied in the centralized setting but is less explored in federated or distributed settings. We propose FedCMOO, a novel communication-efficient federated multi-objective optimization (FMOO) algorithm that improves the error convergence performance of the model compared to existing approaches. Unlike prior works, the communication cost of FedCMOO does not scale with the number of objectives, as each client sends a single aggregated gradient to the central server. We provide a convergence analysis of the proposed method for smooth and non-convex objective functions under milder assumptions than in prior work. In addition, we introduce a variant of FedCMOO that allows users to specify a preference over the objectives in terms of a desired ratio of the final objective values. Through extensive experiments, we demonstrate the superiority of our proposed method over baseline approaches.
Introduction
In modern machine learning, models are often required to satisfy multiple, sometimes conflicting, objectives simultaneously. For instance, a recommendation system might aim for both high accuracy and high diversity. This challenge is known as Multi-Objective Optimization (MOO). When applied to Federated Learning (FL)—where data resides on decentralized edge devices—MOO faces significant hurdles, primarily due to communication constraints and the "tug-of-war" between tasks.
Existing federated MOO methods often suffer from a communication bottleneck, requiring clients to transmit separate updates for every objective. Furthermore, "local objective drift"—where clients optimize tasks independently before server-side fusion—can lead to poor convergence. To address these issues, we introduce FedCMOO, a communication-efficient framework that decouples the number of objectives from the communication cost and mitigates local drift.
The FedCMOO Framework
FedCMOO introduces two primary innovations to enable efficient and effective federated multi-task learning:
- Communication-Efficient Weighting: Instead of transmitting $M$ gradients for $M$ objectives, FedCMOO employs a subroutine called
ApproxGramJacobian. Clients use a randomized SVD-based approach to compress their Jacobian matrices. The server then utilizes these compressed representations to estimate the Gram matrix, which provides sufficient information to calculate optimal task weights without full gradient transmission. This reduces communication cost from $\Theta(Md)$ to $\Theta(d)$, where $d$ is the model size. - Joint Local Training: After the server determines the optimal weights, it broadcasts a single set of weights to the clients. Clients then perform local updates on the weighted sum of all objectives. This ensures that local training remains aligned with the global Pareto-optimal direction, drastically reducing local drift compared to methods that train tasks separately.
- Preference-Aware Optimization (FedCMOO-Pref): A variant that allows users to specify desired ratios between objective values (e.g., prioritizing one task's loss over another), providing fine-grained control over the final model's behavior.
Experimental Evaluation
We evaluated FedCMOO against state-of-the-art baselines like FSMGDA across several benchmarks, including MultiMNIST, CelebA (with 40 facial attribute tasks), and QM9 (a molecular property dataset with 11 objectives).
Performance and Accuracy
FedCMOO demonstrates significant improvements in both accuracy and convergence speed. On the CelebA dataset, FedCMOO significantly outperformed FSMGDA, which struggled to converge due to high local drift across its 40 tasks.
Efficiency and Convergence
In the QM9 regression tasks, FedCMOO achieved a lower error rate in just 100 rounds than the baseline reached in 400 rounds. Crucially, when considering the "Total Upload Size," FedCMOO is orders of magnitude more efficient because its communication overhead remains constant regardless of the number of objectives.
Conclusion and Future Work
FedCMOO offers a scalable path for training complex, multi-task models on edge devices without prohibitive communication costs. By approximating the Gram matrix, the server can find a descent direction without needing full task-specific gradients. Empirical results show that training on a weighted aggregate locally is a powerful fix for the drift issues that plague previous federated MOO algorithms.
While the randomized SVD adds some local computational overhead, the trade-off for massive communication savings is highly favorable. Future work will explore how task-specific characteristics influence alignment with user preference vectors and investigate further variance reduction techniques to stabilize training in highly heterogeneous environments.
Summary updated