Abstract
Sequence learning tasks are often accompanied by multiple temporally correlated auxiliary tasks, such as varying translation latencies or different prediction horizons in time-series forecasting. While these auxiliary tasks can provide valuable signals, effectively leveraging them remains a challenge. We propose Temporally Correlated Task Scheduling (TCTS), a framework that introduces a learnable scheduler to adaptively select auxiliary tasks during training. By employing bi-level optimization, the scheduler learns to prioritize tasks that most effectively improve the model's performance on the primary objective. Our experiments in simultaneous machine translation and stock trend forecasting demonstrate that TCTS significantly outperforms standard multi-task and curriculum learning baselines.
Introduction
In many sequence learning applications, the "main" task is part of a broader family of related tasks. For instance, in Simultaneous Machine Translation (SiMT), the goal is to translate speech or text in real-time. This primary task is related to various "wait-k" auxiliary tasks, where the model waits for k words before starting. Similarly, in Stock Trend Forecasting, predicting tomorrow's price (the main task) is inherently linked to predicting prices for subsequent days.
Traditional approaches like Multi-Task Learning (MTL) treat all tasks equally, while Curriculum Learning (CL) uses fixed, hand-designed schedules. However, the optimal auxiliary task for training often depends on the model's current state and the specific data batch. TCTS addresses this by learning an adaptive scheduling policy.
Methodology: The TCTS Framework
The TCTS framework consists of two primary components: a sequence learning model ($f$) and a learnable scheduler ($\phi$).
Bi-Level Optimization
The system is trained using a bi-level optimization strategy:
- Inner Loop: The model parameters are updated to minimize training loss on an auxiliary task selected by the scheduler.
- Outer Loop: The scheduler is updated via the REINFORCE algorithm. Its reward is based on the model's performance improvement on the main task's validation set. This ensures the scheduler learns to pick tasks that provide the most beneficial gradients for the primary goal.
For SiMT, the authors also introduced a unidirectional/incremental encoder, optimizing computational complexity from $O(L^3)$ to $O(L^2)$, which significantly enhances inference speed.
Experimental Evaluation
The method was evaluated on two distinct domains:
- Simultaneous Machine Translation: Tested on IWSLT (En→De, En→Vi, En→Zh) and WMT (En→De) datasets.
- Stock Trend Forecasting: Evaluated using the CSI300 dataset, containing historical transaction data for the 300 largest stocks in the Chinese A-share market from 2008 to 2020.
Key Results and Insights
TCTS consistently outperformed standard baselines, including fixed "wait-k" models, MTL, and CL. In translation tasks, TCTS achieved BLEU score improvements of 1 to 3 points across various latency settings. In stock forecasting, it significantly improved the RankIC metric and reduced Mean Square Error compared to standalone GRUs and advanced models like Graph Attention Networks (GAT).
Analysis of the Scheduler
- Dynamic Hardness: The scheduler typically begins by sampling tasks uniformly but eventually focuses on "harder" auxiliary tasks (e.g., higher latency or longer-term predictions) as the model matures.
- State-Driven Decisions: Ablation studies show the scheduler relies more on "network status" (e.g., current validation loss) than on raw data features (e.g., sequence length), suggesting that knowing the model's current proficiency is critical for effective scheduling.
Conclusion
TCTS provides a robust, automated way to leverage auxiliary tasks in sequence learning. By treating task selection as a learnable policy, it avoids the pitfalls of rigid curriculum design. While the bi-level optimization introduces a 20-30% training overhead, the performance gains justify the cost. Future work may explore the automated discovery of auxiliary tasks and application to other domains like streaming speech recognition or weather forecasting.
Summary updated