When controlling a physical system with unknown dynamics—such as an autonomous vehicle navigating changing terrain or a robotic limb lifting an unfamiliar payload—a controller must satisfy two competing mandates. First, it must regulate the system to minimize operational costs or track a desired trajectory. Second, it must learn the system's underlying physics by actively perturbing it.
This tension is known as the dual control problem. If a controller focuses entirely on immediate performance, it risks operating on inaccurate model assumptions, leading to long-term instability or suboptimality. Conversely, if it injects excessive exploratory signals to identify the system's parameters, it incurs immediate performance costs and risks destabilizing the system.
Why Optimal Control Destroys Data Quality
To understand why this trade-off is so difficult, consider how an adaptive controller behaves when it attempts to minimize regret—the cumulative performance gap between the adaptive controller and an omniscient controller that knows the true system dynamics from the start (Abbasi-Yadkori et al.; Dean et al.).
Intuitively, one might hope that a controller could simply implement a "passive" learning strategy: execute the best estimated control policy, observe the natural variations in the state transitions, and refine the model over time. However, Ziemann et al. and Ziemann demonstrated a fundamental mathematical barrier to this approach: optimal control policies are frequently uninformative.
When a controller successfully regulates a system, it actively dampens variations and forces the state to remain within a narrow, low-dimensional subspace. Because the system is prevented from deviating, the incoming data becomes ill-conditioned. The controller cannot observe how the system behaves outside of this tight trajectory, making it impossible to uniquely identify the true physical parameters. This creates a sharp "phase transition" in learning efficiency: if the optimal policy suppresses too much excitation, achieving fast (logarithmic) regret becomes mathematically impossible, forcing the system into a much slower ($\sqrt{T}$) learning rate (Ziemann et al.).
The Circular Trap of Active Learning
To break out of this subspace, the controller must intentionally inject exploratory perturbations—often referred to as persistent excitation (Solowjow et al.; Topcu et al.). However, designing these exploratory inputs introduces a circular dependency.
As Wagenmaker et al. point out, to construct an optimal exploratory signal that maximizes information gain without destabilizing the system, you must already know the system's parameters. Without prior knowledge, exploratory inputs act as blind noise relative to the optimal path, directly increasing cumulative regret (Tewari et al.). Furthermore, in closed-loop systems, the feedback gains themselves can actively resist these exploratory movements, treating them as disturbances to be suppressed rather than informative probes (Watson et al.).
Algorithmic Solutions to the Dual Problem
Modern adaptive control frameworks resolve this circularity through three primary paradigms that explicitly balance exploration and regulation.
1. Uncertainty-Aware Cost Functions
Rather than treating system identification as a separate preprocessing step, some approaches embed model uncertainty directly into the control objective. Sbárbaro demonstrated that by minimizing a joint cost function that penalizes both tracking error and model prediction variance, a dual control strategy naturally emerges. When uncertainty is high, the controller deliberately executes probing actions to shrink the model's confidence bounds; as the model becomes precise, the controller automatically transitions to aggressive tracking.
2. Active Probing and Dual MPC
In Model Predictive Control (MPC), this balance is achieved by splitting the controller's optimization horizon or explicitly planning actions to elicit environmental responses. For example, Dual Stochastic MPC frameworks split control into an active learning phase and a performance phase (Arcari et al.).
A physical intuition for this is found in autonomous driving: Knaup et al. developed a framework where an autonomous vehicle executes subtle, strategic maneuvers ("probing") near a highway on-ramp to force surrounding human drivers to react. By observing these reactions, the vehicle rapidly identifies whether a driver is aggressive or cooperative, resolving model ambiguity far faster than passive observation (Schuurmans et al.).
3. Optimism and Randomized Exploration
Other methods adapt multi-armed bandit principles to continuous dynamical systems. The "Optimism in the Face of Uncertainty" (OFU) principle constructs a confidence set of possible systems and selects the control policy that yields the best possible performance across that set (Abbasi-Yadkori et al.; Lale et al.).
Alternatively, Thompson Sampling approaches randomly sample a system model from a posterior distribution and execute its corresponding optimal policy (Ouyang et al.). However, Abeille et al. highlight a critical constraint unique to control: while standard bandits can switch strategies instantly, control systems require infrequent policy updates. Changing control strategies too rapidly introduces transient dynamics that can destabilize physical hardware, forcing a strict mathematical trade-off between frequent sampling for optimism and infrequent updates for physical stability.
The Information-Theoretic Limit
Ultimately, the dual control problem is bounded by fundamental physical laws. By framing adaptive control through information theory, Raginsky established that a controller's capacity to stabilize an uncertain system is strictly limited by the Shannon information capacity of the control loop. An adaptive agent cannot manipulate a system more effectively than the rate at which it can extract information from it, proving that active exploration is not just an algorithmic preference, but a mathematical prerequisite for robust control.
Key Foundational Work
NONLINEAR ADAPTIVE CONTROL USING NONPARAMETRIC GAUSSIAN PROCESS PRIOR MODELS (2002) — contributed 2 concept elements to this direction.
Bayes-Adaptive POMDPs (2007) — contributed 1 concept elements to this direction.