Variance Reduction Can Improve Trade-Off in Multi-Objective Learning

Heshan Devaka Fernando, Lisha Chen, Songtao Lu, Pin-Yu Chen, Miao Liu, Subhajit Chaudhury, Keerthiram Murugesan, Gaowen Liu, Meng Wang, Tianyi Chen

Abstract

Many machine learning problems today have multiple objective functions, which are often tackled by the multi-objective learning (MOL) framework. Albeit many encouraging results are obtained by MOL algorithms, a recent theoretical study [1] revealed that these gradient-based MOL methods (e.g., MGDA, CAGrad) all reflect an inherent trade-off between optimization convergence speeds and conflict-avoidance abilities. To this end, we develop an improved stochastic variance-reduced multi-objective gradient correction method for MOL, achieving the ${\mathcal{O}}\left({{\varepsilon ^{ - 1.5}}}\right)$ sample complexity. In addition, our proposed method simultaneously improves the theoretical guarantees for conflict avoidance and convergence rate compared to prior stochastic gradient-based MOL methods in the non-convex setting. We further validate the effectiveness of the proposed method empirically using popular multi-task learning (MTL) benchmarks.
BibTeX
@inproceedings{fernando2024variance,
  author = {Heshan Devaka Fernando and Lisha Chen and Songtao Lu and Pin-Yu Chen and Miao Liu and Subhajit Chaudhury and Keerthiram Murugesan and Gaowen Liu and Meng Wang and Tianyi Chen},
  title = {{Variance Reduction Can Improve Trade-Off in Multi-Objective Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446038},
}

Loading related content...

Figure