Lookahead planning algorithms, such as Monte Carlo Tree Search (MCTS), have achieved historic breakthroughs in discrete environments like Chess and Go. These algorithms work by constructing a search tree, simulating future sequences of moves, and using statistics (like visit counts and average rewards) to decide which paths are worth exploring deeper.
However, when these algorithms encounter real-world domains like robotics, autonomous driving, or resource allocation, they run into a fundamental mathematical barrier: the curse of continuity.
In a continuous action space, the branching factor of the search tree is infinite. If you sample actions purely at random, the probability of selecting the exact same real-valued action twice is zero. As a result, standard tree search cannot aggregate statistics for individual action nodes. The search tree expands infinitely wide rather than deep, causing the planner to fail to look into the future.
To make lookahead planning work in continuous spaces, researchers must tame this infinite branching factor. Modern approaches solve this problem through three core mechanisms: action shortlisting, space partitioning and smoothing, and gradient-based trajectory optimization.
Action Shortlisting: Generating Discrete Candidates
The most direct way to handle an infinite action space is to filter it down to a small, high-quality shortlist of discrete candidate actions at each step. If a generator can reliably propose a handful of promising actions, traditional tree search can proceed as normal.
A common strategy is to train a deep neural network to act as this action generator. For example, Riviere utilizes neural network priors to propose coordinated maneuvers for multi-robot planning within a search tree. Similarly, Deglurkar et al. combine offline-trained generative models with online MCTS to handle high-dimensional vision tasks under uncertainty.
Infinite Actions ──> [ Learned Generator ] ──> Shortlist (a1, a2, a3) ──> Standard Tree Search
However, standard policy networks are typically trained to output the single average best action, which can lead to a shortlist of highly redundant candidates. To maintain search efficiency, the shortlist must be diverse.
To address this, Ahmad et al. introduced a marginal utility objective for training action generators. Instead of optimizing for the average value of all proposed actions, their framework explicitly optimizes for the added value that each new candidate action brings to the existing shortlist. This naturally forces the generator to cover different modes of the action space (e.g., proposing both a sharp left turn and a hard brake in a driving scenario).
By formalizing this selection process, frameworks like Sampled MuZero, introduced by Hubert et al., demonstrate that lookahead planning can mathematically excel via sample-based policy iteration, even when full action enumeration is impossible.
Space Partitioning and Smoothing: Sharing Information
Instead of picking a few isolated actions, another family of methods structures the continuous space so that nearby actions can share information. This leverages the physical intuition of smoothness: a steering angle of $10.0^\circ$ will yield a nearly identical outcome to $10.1^\circ$.
One way to exploit this is through statistical smoothing. Couëtoux et al. developed Continuous RAVE (cRAVE), an algorithm that applies Gaussian smoothing across the action space. When the tree evaluates the outcome of a specific continuous action, it updates the statistics of all "neighboring" actions in the tree, allowing data to aggregate even if the exact same action value is never visited twice.
Action Sampled (10.0°) ──> Updates statistics for [ 9.9° <─── 10.0° ───> 10.1° ] via Gaussian Smoothing
Alternatively, planners can dynamically slice the continuous space into regions. Rather than using a rigid, pre-defined grid—which suffers from the curse of dimensionality—algorithms can partition space adaptively based on where high rewards are found.
- Voronoi Partitioning: Kim et al. proposed VOOT (Voronoi Optimistic Optimization applied to Trees). Instead of axis-aligned splits, it uses implicit Voronoi cells to dynamically subdivide the continuous action space, focusing computational resources on the most promising regions while maintaining theoretical regret bounds.
- Progressive Widening: This technique limits the number of active action branches at a node, only allowing new action samples to be added as the node receives more total visits. Jamgochian combined progressive widening with dual ascent to handle continuous spaces in safety-critical constrained environments, while Bianchi et al. demonstrated that seeding this widening process with boundary values significantly accelerates convergence.
Gradient-Based Planning: Exploiting Differentiability
The third paradigm bypasses discrete branching entirely. If an agent learns a transition model of the world using a differentiable neural network, the entire lookahead plan becomes a differentiable sequence of operations. Instead of searching a discrete tree, the planner can use gradient descent to backpropagate through the physics of its own mind.
Initial Actions ──> [ Differentiable World Model ] ──> Predicted Reward
▲ │
└─────────────────── Backpropagate Gradients ────────┘
Wu et al. pioneered this with TF-Plan. By exploiting the mathematical structure of ReLU-based neural networks, they showed that an agent can compute exact gradients from the predicted future reward all the way back to the initial action inputs.
Similarly, Zhu et al. introduced Policy Optimization with Model Planning (POMP). Their framework utilizes first-order Taylor expansions of the Bellman equation to iteratively refine continuous action sequences. By treating planning as trajectory optimization over a smooth landscape, these gradient-based methods turn the infinite nature of continuous spaces from a curse into an advantage, navigating the space via smooth vectors rather than blind combinatorial guessing.
Summary
Planning with an infinite branching factor requires moving away from rigid, discrete choices. Whether by using learned generators to propose diverse shortlists (Ahmad et al.), sharing statistics across continuous neighborhoods (Couëtoux et al.), or backpropagating gradients directly through world models (Wu et al.), these techniques allow agents to retain the long-horizon foresight of tree search in a smooth, continuous world.
Key Foundational Work
Applying Online Search Techniques to Continuous-State Reinforcement Learning (1998) — contributed 1 concept elements to this direction.
A General Manipulation Task Planner (2002) — contributed 1 concept elements to this direction.
Computational Complexity of Planning with Discrete Time and Continuous State Variables (2002) — contributed 1 concept elements to this direction.