Variance Reduction Can Improve Trade-Off in Multi-Objective Learning
Lists
Only you see this
Abstract
Many machine learning problems today have multiple objective functions, which are often tackled by the multi-objective learning (MOL) framework. Albeit many encouraging results are obtained by MOL algorithms, a recent theoretical study [1] revealed that these gradient-based MOL methods (e.g., MGDA, CAGrad) all reflect an inherent trade-off between optimization convergence speeds and conflict-avoidance abilities. To this end, we develop an improved stochastic variance-reduced multi-objective gradient correction method for MOL, achieving the ${\mathcal{O}}\left({{\varepsilon ^{ - 1.5}}}\right)$ sample complexity. In addition, our proposed method simultaneously improves the theoretical guarantees for conflict avoidance and convergence rate compared to prior stochastic gradient-based MOL methods in the non-convex setting. We further validate the effectiveness of the proposed method empirically using popular multi-task learning (MTL) benchmarks.
BibTeX
@inproceedings{fernando2024variance,
author = {Heshan Devaka Fernando and Lisha Chen and Songtao Lu and Pin-Yu Chen and Miao Liu and Subhajit Chaudhury and Keerthiram Murugesan and Gaowen Liu and Meng Wang and Tianyi Chen},
title = {{Variance Reduction Can Improve Trade-Off in Multi-Objective Learning}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2024},
doi = {10.1109/icassp48485.2024.10446038},
}Loading related content...