Q-Learning Scheduler for Multi Task Learning Through the use of Histogram of Task Uncertainty

Kourosh Meshgi, Maryam Sadat Mirzaei, Satoshi SekineACL 2022

TL;DR

This paper introduces a novel Q-learning-based scheduler (QLS) for multi-task learning (MTL) that dynamically learns an optimal task ordering strategy. By using a "histogram of task uncertainty" as a state representation, QLS effectively balances task learning, prevents catastrophic forgetting, and improves overall MTL performance compared to existing methods.

What Problem Does This Paper Address?

Multi-task learning (MTL) aims to improve model generalization by training on multiple tasks simultaneously, sharing parameters to reduce redundancy and leverage inter-task relationships. However, simply training tasks in a uniform or arbitrary order can lead to suboptimal performance. This is because tasks have varying complexities, sample sizes, and interdependencies, which can cause issues like:

  • Redundant computations: Training easy tasks when harder ones are more critical.
  • Catastrophic forgetting: When training on one task negatively impacts performance on another.
  • Task imbalances: Leading to skewed gradient updates.
  • Plateaus and overfitting: Simpler tasks might be overtrained while harder ones struggle.

Existing scheduling methods often rely on heuristics, fixed strategies, or simple signals like validation loss. These approaches can be too simplistic to capture the complex dynamics of MTL, especially when dealing with heterogeneous tasks or intricate inter-task relationships. The paper argues for a more adaptive and intelligent scheduling approach that can learn optimal policies from experience.

Methods

The core of the proposed method is a Deep Q-Learning Scheduler (QLS). Here's a breakdown of its key components:

  • State Representation: Histogram of Task Uncertainty: Instead of relying solely on accuracy or loss, QLS uses a novel state representation called the "histogram of task uncertainty." This captures what the model doesn't know for each task. Specifically, it uses margin uncertainty (based on the difference between the top two predicted probabilities for a class) calculated on validation data for each task. These individual histograms are concatenated to form the overall state.
  • Reinforcement Learning (RL) Framework: QLS frames task scheduling as an RL problem.
  • Agent: The Q-learning agent.
  • State ($S_t$): The histogram of task uncertainties.
  • Action ($a_t$): Selecting which task to train next (represented by a one-hot vector).
  • Reward ($r_t$): Defined as the negative sum of losses across all tasks on the validation set. A large negative reward is given if the MTL's average accuracy drops significantly below that of a single-task baseline, acting as a strong penalty for poor scheduling.
  • Policy: The agent learns a policy to choose actions that maximize cumulative future rewards.
  • Q-Learning Algorithm: The agent uses Q-learning to estimate the expected future reward for taking a specific action in a given state. The Q-values are updated using the Bellman equation: $Q(S_t, a_t) = r(S_t, a_t) + \gamma Q(S_{t+1}, a_{t+1})$.
  • Implementation Details: The paper uses an LSTM-based network for MTL and GloVe embeddings. Q-learning parameters like discount factor ($\gamma$) and learning rate are set to fixed values, and Q-values are initialized randomly. They experimented with different uncertainty measures and found margin uncertainty to be most effective.

Datasets

The paper evaluates the proposed QLS on several benchmark datasets across different scenarios:

  • Multi-Domain Text Classification: 16 different product review datasets (e.g., Books, Electronics, DVD, Kitchen, Apparel, Camera, Health, Music, Toys, Video, Baby, Magazines, Software, Sports, IMDB, MR) were used. These datasets represent variations of the same task (sentiment classification) across different domains.
  • Multi-Task Text Classification (Simple Relations): Three related NLP tasks: Part-of-Speech (POS) tagging, chunking, and Named Entity Recognition (NER). These tasks are considered related but do not directly benefit from each other's outputs.
  • Multi-Task Text Classification (Complex Relations): Neural Machine Translation (NMT) as the target task, with POS tagging and dependency parsing as auxiliary tasks. This scenario involves more complex inter-task relationships where auxiliary tasks can be prerequisites or provide beneficial features. Datasets used include WMT'14 parallel corpus for translation, Penn Treebank for POS tagging, and TIGER tree-bank for dependency parsing.

Key Insights and Ideas

  • Task Uncertainty as a Rich State Signal: Moving beyond simple loss or accuracy, using task uncertainty provides a more nuanced view of the model's internal state, revealing what it "doesn't know" and thus where learning is most needed.
  • Learning Optimal Scheduling Policies: Framing scheduling as an RL problem allows the system to learn complex, dynamic strategies that adapt to the evolving MTL landscape, rather than relying on static or heuristic rules.
  • Handling Task Imbalances and Forgetting: The QLS, by learning from delayed rewards and considering the overall state, is shown to be effective in mitigating catastrophic forgetting and balancing the learning progress across tasks with varying difficulties.
  • Discovering Latent Features: The RL approach can implicitly discover useful inter-task relationships and learn features that are beneficial across multiple tasks, even when these relationships are not explicitly defined.

Key Results

  • Multi-Domain Text Classification: QLS consistently outperformed various baseline scheduling methods (Uniform, Hand Crafted, Random, Greedy, Loss Exponentiation, Homoscedastic Uncertainty, Self Paced, Focal Loss, DTP, Grad Norm, Adaptive) across 16 domains, achieving higher average accuracy. For instance, in Table 2, QLS achieved an average accuracy of 91.5%, outperforming the next best adaptive scheduler at 91.0%.
  • Multi-Task Text Classification (Simple Relations): QLS also showed superior performance in this scenario, outperforming other methods.
  • Multi-Task Text Classification (Complex Relations): For NMT, QLS achieved a BLEU score of 21.7, a significant improvement over the baseline single-task NMT (19.30 BLEU) and other scheduling methods. This highlights its effectiveness in complex scenarios.
  • Generalization: The paper demonstrates that QLS improves the generalization of emergent shared features.

Limitations and Future Work

  • Computational Cost: Training an RL agent for scheduling can be computationally more expensive than static or heuristic methods, although the paper argues that this upfront cost leads to better overall performance and potentially faster inference by avoiding suboptimal exploration.
  • Balanced vs. Imbalanced Tasks: The authors note that while QLS excels at handling task difficulty imbalances, it might not perform best in scenarios with perfectly balanced tasks or where complex tasks heavily benefit from abundant related easy tasks.
  • Future Work: The paper suggests incorporating QLS with transformer architectures, which are becoming increasingly prevalent, to benchmark the added value of such scheduling. They also mention exploring other uncertainty measures and state representations.

Summary updated

BibTeX
@inproceedings{meshgi2022q,
  author = {Kourosh Meshgi and Maryam Sadat Mirzaei and Satoshi Sekine},
  title = {{Q-Learning Scheduler for Multi Task Learning Through the use of Histogram of Task Uncertainty}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure