AutoLoss: Learning Discrete Schedules for Alternate Optimization

Haowen Xu, Hao Zhang, Zhiting Hu, Xiaodan Liang, Ruslan Salakhutdinov, Eric P. XingarXiv 2018 · OpenReview

TL;DR

This paper introduces AutoLoss, a meta-learning framework that automatically learns optimal schedules for iterative and alternating optimization problems. By framing schedule selection as a reinforcement learning task, AutoLoss can dynamically adapt to the training process, leading to improved convergence quality and speed across various machine learning tasks.

What Problem Does This Paper Address?

Many complex machine learning problems, such as training Generative Adversarial Networks (GANs) or multi-task learning models, involve optimizing multiple objectives or parameter sets in an alternating fashion. The effectiveness of these optimization processes heavily relies on the "schedule" – deciding which objective to optimize and for how long at each step. Traditionally, these schedules are hand-designed heuristics or determined through extensive, often inefficient, hyperparameter searches. This manual approach is suboptimal, time-consuming, and may not generalize well to different models or datasets. The paper addresses the challenge of automatically learning these crucial optimization schedules in a data-driven manner.

Methods

The core of AutoLoss is a meta-learning framework that uses reinforcement learning (RL) to learn the optimization schedule. The framework treats the optimization process as a sequential decision-making problem. A "controller" (a neural network) observes the current state of the task model's optimization (e.g., training progress, loss values, gradient magnitudes) and decides which task objective and parameter set to update next. This decision is made by sampling from a learned policy.

The controller is trained using policy gradient methods (specifically, REINFORCE and its variants) to maximize a reward signal, which is typically the final performance metric of the task model (e.g., accuracy, inception score, perplexity). The framework is designed to be generic, allowing it to be applied to various iterative and alternate optimization scenarios by defining the task-specific loss functions, parameters, action space, state features, and reward function.

Datasets

The paper evaluates AutoLoss on the following tasks and datasets:

  • d-ary Quadratic Regression and MLP Classification: Synthesized datasets were used, designed to specifically test the model's ability to combat overfitting with L1 regularization.
  • Image Generation using GANs: The MNIST dataset was used for training DCGANs. For transferability experiments, the CIFAR-10 dataset was also used.
  • Multi-task Neural Machine Translation (NMT):
    • German-to-English translation: WIT corpus (subset).
  • German Named Entity Recognition (NER): GermEval 2014 NER Shared Task data.
  • German POS Tagging: Tiger Corpus.

No new benchmarks were introduced; instead, existing datasets and tasks were adapted to the AutoLoss framework.

Key Insights and Ideas

  • Optimization Schedules are Learnable: The paper strongly suggests that effective optimization schedules are not arbitrary but can be learned from data, leading to significant improvements in convergence quality.
  • Meta-Learning for Optimization: AutoLoss demonstrates the power of meta-learning by training a "controller" that learns how to optimize, rather than just optimizing a specific model. This controller can then guide the optimization of new, unseen task models.
  • Dynamic Adaptation is Key: Unlike fixed heuristic schedules, AutoLoss can dynamically adjust its decisions based on the real-time state of the optimization process, allowing it to adapt to complex dynamics like overfitting or imbalanced objective importance.
  • Generalizability of Learned Schedules: A trained AutoLoss controller can be transferred to different task models (even with different architectures) and different datasets, showcasing its ability to learn generalizable optimization strategies.
  • Beyond Linear Combinations: AutoLoss is particularly effective for problems where objectives cannot be easily combined linearly (like GANs), offering a more flexible approach than traditional hyperparameter tuning.

Key Results

  • Improved Convergence Quality: AutoLoss consistently achieved better convergence quality across all four tested tasks (regression, classification, GANs, NMT) compared to strong baselines, including manually designed schedules and grid search.
  • For regression and classification with L1 regularization, AutoLoss effectively detected and combatted overfitting, outperforming baselines that either lacked regularization or used fixed linear combinations.
  • For GANs, AutoLoss achieved higher Inception Scores (IS) and faster per-epoch convergence than fixed ratio schedules (e.g., 1:K). For example, it improved converged IS by 0.5 over GAN 1:1 and was nearly 3x faster.
  • For NMT, AutoLoss achieved the lowest validation perplexity (3.54), surpassing single-task baselines and heavily tuned multi-task schedules.
  • Efficiency and Sample Efficiency: AutoLoss demonstrated significant sample efficiency, especially in comparison to grid search methods. For regression, it achieved better convergence with fewer training runs. For complex tasks like GANs and NMT, controller training often occurred concurrently with task model training, avoiding the need for extensive repeated experiments.
  • Transferability:
    • A trained controller generalized to new GAN architectures, outperforming baselines in 16 out of 20 tested architectures.
  • A controller trained on MNIST for GANs could guide GAN training on CIFAR-10, achieving comparable or better results than fixed schedules.

Limitations and Future Work

  • Computational Overhead: While generally efficient, the controller's feature extraction and inference add some overhead. For tasks that converge very quickly, AutoLoss might require multiple training instances to accumulate enough supervision, similar to grid search.
  • Reward Sparsity: For computationally heavy tasks requiring many iterations, the reward signal can be sparse and delayed, necessitating techniques like intermediate rewards or longer training horizons.
  • Future Work: The authors suggest that AutoLoss can be extended to other iterative processes beyond optimization, such as sampling methods. They also envision its application to more complex scenarios with more intricate action spaces and state representations. The generalizability of the learned controllers to entirely new domains and problem types remains an exciting avenue for future research.

Summary updated

Abstract

Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence. In this paper, we present AutoLoss, a meta-learning framework that automatically learns and determines the optimization schedule. AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives. We apply AutoLoss on four ML tasks: d-ary quadratic regression, classification using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT). We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks. The trained AutoLoss controller is generalizable -- it can guide and improve the learning of a new task model with different specifications, or on different datasets.
BibTeX
@misc{xu2018autoloss,
  author = {Haowen Xu and Hao Zhang and Zhiting Hu and Xiaodan Liang and Ruslan Salakhutdinov and Eric P. Xing},
  title = {{AutoLoss: Learning Discrete Schedules for Alternate Optimization}},
  year = {2018},
  eprint = {1810.02442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.02442},
}
Figure