Online Switching Control as Learning with Experts

Masataka Nakamura, Kaito Ito, Hideaki IshiiarXiv 2026 · arXiv

TL;DR

This paper introduces a novel algorithm for online switching control in nonlinear systems that achieves the theoretically optimal regret bound of $O(\sqrt{T \log N})$. By combining the classic Hedge algorithm with "lazy sampling" and a "dwell-time filter," the authors successfully balance the need to learn the best controller in real-time with the physical requirement to maintain system stability.

What Problem Does This Paper Address?

In many real-world control scenarios—like industrial robotics or power grid management—the environment is unpredictable. "Adaptive switching control" (or supervisory control) handles this by maintaining a library of candidate controllers and switching between them as conditions change.

However, most traditional research in this area focuses on asymptotic stability (will the system eventually settle down?) rather than non-asymptotic performance (how much cost do we incur while learning?). Recently, the field of "online control" has tried to bridge this gap using regret analysis, which measures the difference between the cost incurred by the learner and the cost of the single best controller chosen in hindsight.

Previous attempts at online switching control often assumed "bandit feedback" (you only see the cost of the controller you actually used), which leads to a relatively slow learning rate. This paper tackles the full-information feedback setting, where the learner can observe the entire cost function and reconstruct disturbance effects after each step. The challenge is to exploit this extra information to achieve optimal regret without switching so frequently that the physical system becomes unstable.

Methods

The authors propose a three-component strategy to solve this problem:

  1. Counterfactual Cost Reconstruction: Using a "replayability" assumption, the learner uses historical data to simulate what would have happened if they had used any of the other candidate controllers. This reduces the complex control problem into a standard "learning with experts" problem.
  2. Lazy Sampling (Maximal Coupling): Standard learning algorithms like Hedge might suggest a new probability distribution for controllers at every time step. To avoid jittery, frequent switching, the authors use "lazy sampling." This technique ensures the controller only changes when the underlying probability distribution shifts significantly, minimizing the total number of switches.
  3. Dwell-Time Filter: Even with lazy sampling, a series of quick switches could theoretically destabilize a nonlinear system. The authors add a deterministic "dwell-time filter" that enforces a minimum period ($\tau$) for which a controller must remain active. This ensures the system state remains uniformly bounded (stable).

Datasets

The paper does not use a standard public dataset. Instead, it evaluates the algorithm using numerical simulations of a discrete-time system:

  • System Dynamics: A 2D system with four candidate controllers.
  • Disturbances: Non-stochastic, bounded disturbances.
  • Cost Functions: Time-varying quadratic costs designed to "stress test" the switching mechanism by encouraging frequent changes.
  • Baseline: Compared against BAPS (Bandit-based Adaptive Policy Selection), a state-of-the-art method for the bandit feedback setting.

Key Insights and Ideas

  • The Power of "What If": The core "aha" moment is that in a full-information setting, you don't need to actually run a controller to know how it would have performed. By reconstructing the "counterfactual" trajectories, you can learn much faster than bandit-based methods.
  • Stability vs. Optimality: Usually, adding constraints (like a minimum dwell time) hurts performance. The authors prove that their switching suppression mechanism is "gentle" enough that it doesn't degrade the regret order. You get stability "for free" in terms of the big-O complexity.
  • Matching the Lower Bound: The achieved regret of $O(\sqrt{T \log N})$ matches the theoretical lower bound for the standard experts problem, proving that their approach is as efficient as possible for this setting.

Key Results

  • Regret Performance: The proposed method achieves a regret of $O(\sqrt{T \log N})$, significantly outperforming the $O(T^{2/3})$ regret typically found in bandit-feedback literature.
  • Numerical Superiority: In simulations, the proposed algorithm achieved much lower mean regret than the BAPS baseline.
  • Stability Verification: In "stress tests" where the cost function was specifically designed to trick the algorithm into switching too fast, the dwell-time filter was shown to be essential. Without it, the system state norm exploded; with it, the state remained bounded while still maintaining low regret.

Limitations and Future Work

  • Known Dynamics: The current framework assumes the transition map (the physics of the system) is known. While the authors note that full knowledge isn't strictly necessary (only the ability to "replay" disturbances), applying this to a completely unknown system would require integrating system identification.
  • Finite Candidate Set: The algorithm selects from a finite set of $N$ controllers. It does not currently handle continuous policy spaces or the generation of new controllers on the fly.
  • Future Directions: The authors suggest extending this work to unknown systems where counterfactual trajectories must be estimated rather than calculated exactly, potentially using techniques from robust control or Gaussian processes.

Summary updated

Abstract

This paper addresses the problem of online switching control for discrete-time nonlinear systems where a controller is sequentially selected from a finite set of candidates. The goal is to generate an adaptive sequence of controllers to minimize the cumulative cost under unknown disturbances and cost functions. Specifically, we consider a full-information feedback setting where realized disturbance effects and cost functions are available after each selection. Using counterfactual costs computed from historical data, we reduce the online switching control problem to the experts problem with switching costs. Then, we propose an adaptive switching algorithm based on the Hedge algorithm with a switching suppression mechanism. Notably, the proposed algorithm achieves an expected regret of $O(\sqrt{T\log N})$, where $T$ is the time horizon and $N$ is the number of candidate controllers, while ensuring uniform boundedness of the state. This matches the optimal regret order for the standard experts problem.
BibTeX
@misc{nakamura2026online,
  author = {Masataka Nakamura and Kaito Ito and Hideaki Ishii},
  title = {{Online Switching Control as Learning with Experts}},
  year = {2026},
  eprint = {2611.18352},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18352},
}
Figure