Temporally Correlated Task Scheduling for Sequence Learning

Xueqing Wu, Lewen Wang, Yingce Xia, Weiqing Liu, Lijun Wu, Shufang Xie, Tao Qin, Tie‐Yan Liu

Abstract

Sequence learning tasks are often accompanied by multiple temporally correlated auxiliary tasks, such as varying translation latencies or different prediction horizons in time-series forecasting. While these auxiliary tasks can provide valuable signals, effectively leveraging them remains a challenge. We propose Temporally Correlated Task Scheduling (TCTS), a framework that introduces a learnable scheduler to adaptively select auxiliary tasks during training. By employing bi-level optimization, the scheduler learns to prioritize tasks that most effectively improve the model's performance on the primary objective. Our experiments in simultaneous machine translation and stock trend forecasting demonstrate that TCTS significantly outperforms standard multi-task and curriculum learning baselines.

Introduction

In many sequence learning applications, the "main" task is part of a broader family of related tasks. For instance, in Simultaneous Machine Translation (SiMT), the goal is to translate speech or text in real-time. This primary task is related to various "wait-k" auxiliary tasks, where the model waits for k words before starting. Similarly, in Stock Trend Forecasting, predicting tomorrow's price (the main task) is inherently linked to predicting prices for subsequent days.

Traditional approaches like Multi-Task Learning (MTL) treat all tasks equally, while Curriculum Learning (CL) uses fixed, hand-designed schedules. However, the optimal auxiliary task for training often depends on the model's current state and the specific data batch. TCTS addresses this by learning an adaptive scheduling policy.

Methodology: The TCTS Framework

The TCTS framework consists of two primary components: a sequence learning model ($f$) and a learnable scheduler ($\phi$).

The TCTS framework architecture, illustrating the interaction between the model parameters, the scheduler, and the selection of auxiliary tasks.

Bi-Level Optimization

The system is trained using a bi-level optimization strategy:

  1. Inner Loop: The model parameters are updated to minimize training loss on an auxiliary task selected by the scheduler.
  2. Outer Loop: The scheduler is updated via the REINFORCE algorithm. Its reward is based on the model's performance improvement on the main task's validation set. This ensures the scheduler learns to pick tasks that provide the most beneficial gradients for the primary goal.

For SiMT, the authors also introduced a unidirectional/incremental encoder, optimizing computational complexity from $O(L^3)$ to $O(L^2)$, which significantly enhances inference speed.

Experimental Evaluation

The method was evaluated on two distinct domains:

  • Simultaneous Machine Translation: Tested on IWSLT (En→De, En→Vi, En→Zh) and WMT (En→De) datasets.
  • Stock Trend Forecasting: Evaluated using the CSI300 dataset, containing historical transaction data for the 300 largest stocks in the Chinese A-share market from 2008 to 2020.

Key Results and Insights

TCTS consistently outperformed standard baselines, including fixed "wait-k" models, MTL, and CL. In translation tasks, TCTS achieved BLEU score improvements of 1 to 3 points across various latency settings. In stock forecasting, it significantly improved the RankIC metric and reduced Mean Square Error compared to standalone GRUs and advanced models like Graph Attention Networks (GAT).

Comparison of TCTS (Ours) against baselines like Wait-k, MTL, and CL on the En→Zh translation task, showing superior BLEU scores across different latency (AL) levels.

Analysis of the Scheduler

  • Dynamic Hardness: The scheduler typically begins by sampling tasks uniformly but eventually focuses on "harder" auxiliary tasks (e.g., higher latency or longer-term predictions) as the model matures.
  • State-Driven Decisions: Ablation studies show the scheduler relies more on "network status" (e.g., current validation loss) than on raw data features (e.g., sequence length), suggesting that knowing the model's current proficiency is critical for effective scheduling.

Conclusion

TCTS provides a robust, automated way to leverage auxiliary tasks in sequence learning. By treating task selection as a learnable policy, it avoids the pitfalls of rigid curriculum design. While the bi-level optimization introduces a 20-30% training overhead, the performance gains justify the cost. Future work may explore the automated discovery of auxiliary tasks and application to other domains like streaming speech recognition or weather forecasting.

Summary updated

Abstract

Sequence learning has attracted much research attention from the machine learning community in recent years. In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks, which are different in terms of how much input information to use or which future step to predict. For example, (i) in simultaneous machine translation, one can conduct translation under different latency (i.e., how many input words to read/wait before translation); (ii) in stock trend forecasting, one can predict the price of a stock in different future days (e.g., tomorrow, the day after tomorrow). While it is clear that those temporally correlated tasks can help each other, there is a very limited exploration on how to better leverage multiple auxiliary tasks to boost the performance of the main task. In this work, we introduce a learnable scheduler to sequence learning, which can adaptively select auxiliary tasks for training depending on the model status and the current training data. The scheduler and the model for the main task are jointly trained through bi-level optimization. Experiments show that our method significantly improves the performance of simultaneous machine translation and stock trend forecasting.
BibTeX
@inproceedings{wu2021temporally,
  author = {Xueqing Wu and Lewen Wang and Yingce Xia and Weiqing Liu and Lijun Wu and Shufang Xie and Tao Qin and Tie‐Yan Liu},
  title = {{Temporally Correlated Task Scheduling for Sequence Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=FynX8He2_uL},
}
Figure