Adaptive Task Scheduling via Multi Armed Bandits

This direction is split into other directions: dynamic task scheduling. Its last version stays readable and citable here.

Version 1 of 131 papers38 conceptsfrozen 2026-10-04

In traditional machine learning, we typically feed data to a model in a random order or follow a fixed, human-designed curriculum. However, humans don't learn this way. A student doesn't master calculus by looking at random equations; they follow a path where the difficulty of the material scales with their current understanding.

Recent research has converged on a powerful mechanism to replicate this: Automated Curriculum Learning (ACL). Instead of a static schedule, these systems use a "teacher" algorithm—often framed as a Multi-Armed Bandit (MAB)—to dynamically select the most informative tasks or data points at every step of training.

The Learning Progress Signal

The core intuition behind this direction is that the best training data is neither too easy (already mastered) nor too hard (impossible to grasp). As Alex Graves and colleagues demonstrated in Automated Curriculum Learning for Neural Networks, the model should focus on the "Zone of Proximal Development"—the frontier where it is making the most Learning Progress.

In this framework, the training process is treated as a game:

  1. The Arms: Each "arm" of the bandit represents a different task, data subset, or difficulty level (e.g., masking ratio or sequence length).
  2. The Reward: The bandit receives a reward based on how much the model improved after training on that selection. This is often measured by the rate of error reduction or "local prediction gain."

By maximizing this reward, the bandit automatically discovers an optimal path through complex task sets. Ziping Xu and Ambuj Tewari provided a theoretical "stamp of approval" for this in On the Statistical Benefits of Curriculum Learning, proving that prioritizing tasks with high local prediction gain can match optimal statistical rates for task selection.

Balancing Exploration and Exploitation

A major challenge in multi-task learning is "negative transfer," where training on a secondary task actually hurts performance on the primary goal. To solve this, researchers like Xiangheng He use non-stationary bandits with Thompson Sampling in Task Selection and Assignment for Multi-modal Multi-task Dialogue Act Classification with Non-stationary Multi-armed Bandits.

This approach allows the scheduler to "forget" tasks that are no longer useful. As the model matures, the bandit shifts its focus:

  • Early Training: It explores a wide variety of easy, auxiliary tasks to build stable representations.
  • Late Training: It exploits the specific tasks that provide the most significant validation gains for the target objective.

This dynamic interleaving is more efficient than mastering tasks one by one. In robotics, Hanne Say and Erhan Oztop showed in Energy Weighted Learning Progress Guided Interleaved Multi-Task Learning that this "sweet spot" scheduling leads to faster convergence and better skill transfer than fixed methods.

Finding the Peak Learning Signal

How do we know exactly when a task is at the right difficulty? Xinyu Zhang proposes a simple heuristic in Gradient-Aware Scheduling: Coupling Curriculum and Staleness for Async Reinforcement Learning: the peak learning signal often occurs when the model's success rate is approximately 50%. If the model succeeds 100% of the time, it learns nothing new; if it succeeds 0% of the time, the gradient is too noisy to be useful.

This principle is being applied to the most advanced frontiers of AI, including Large Language Models. In Self-Evolving Curriculum for LLM Reasoning, Minsu Kim and Jian Tang use a bandit to treat problem categories as arms. By using "absolute advantage" as a proxy for learning gain, the system ensures the LLM always tackles reasoning problems that are just challenging enough to drive improvement, leading to superior out-of-distribution generalization.

Beyond Accuracy: Efficiency and Personalization

The benefits of bandit-guided curricula extend beyond just final accuracy:

By replacing rigid training schedules with adaptive, reward-driven agents, this research direction moves us toward AI systems that learn with the fluidity and efficiency of biological intelligence.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_adaptive_task_scheduling_via_multi_armed_062961a4,
  title = {Adaptive Task Scheduling via Multi Armed Bandits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-task-scheduling-via-multi-armed-bandits/rdr_062961a450014185962e2568f834761a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_adaptive_task_scheduling_via_multi_armed_062961a4_v1,
  title = {Adaptive Task Scheduling via Multi Armed Bandits},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/adaptive-task-scheduling-via-multi-armed-bandits/rdr_062961a450014185962e2568f834761a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}