When we need to optimize an expensive black-box function or design a sequence of scientific experiments, we rely on an acquisition function to tell us where to sample next. Traditional acquisition functions—such as Expected Improvement or basic Information Gain—are notoriously myopic (short-sighted). They evaluate the utility of the immediate next step as if it were the final action in the campaign.
This short-sightedness frequently traps optimization algorithms in local optima or causes active learning agents to over-exploit familiar regions. To truly maximize the efficiency of an experimental budget, we must transition from greedy heuristics to non-myopic, multi-step lookahead planning.
The Trap of Greedy Exploration
To understand why myopic acquisition functions fail, imagine an autonomous robot mapping environmental phenomena, a scenario studied by Singh. If the robot uses a greedy, one-step-ahead strategy, it will only move toward locations that yield high immediate information. If a massive cluster of information is separated from the robot by a data-barren "valley," a myopic agent will never cross it. Because the very next step into the valley yields zero immediate reward, the agent remains trapped in a local optimum.
As noted by González et al. and Jiang et al., greedy strategies fundamentally ignore how a current choice alters the value of future observations. A non-myopic strategy evaluates the cumulative utility across an entire sequence of actions. It can justify traveling through low-value regions—or executing experiments with low immediate returns—because doing so uncovers massive information payoffs in subsequent stages (Iollo et al.).
The Computational Curse of Foresight
If looking ahead multiple steps yields strictly better results, why has the field historically relied on myopic heuristics? The bottleneck is computational.
Determining the mathematically optimal sequence of future samples requires solving a recursive Bellman equation. To evaluate a single candidate point, you must simulate every possible experimental outcome at that point, update your model (e.g., a Gaussian Process), select the next optimal point under that simulated future, and repeat this process recursively across the planning horizon. As Foster et al. point out, this nested simulation structure causes computational complexity to grow exponentially with the number of lookahead steps, rendering exact sequential planning intractable for real-time applications.
To make non-myopic planning practical, researchers have developed three core mechanisms that bypass this exponential recursion.
Mechanism 1: One-Shot Joint Optimization
Rather than solving a deeply nested, recursive decision tree step-by-step, we can collapse the entire multi-step horizon into a single, joint optimization problem.
Jiang et al. introduced this "one-shot" formulation for Bayesian Optimization. Instead of sequentially optimizing the next point given the past, the one-shot approach optimizes the current point and all future points simultaneously over a simulated tree of outcomes. By leveraging reparameterization tricks and clever linear algebra, the authors make the entire simulated future path differentiable. This allows standard gradient-based optimizers to update the entire multi-step trajectory at once, bypassing recursive nesting entirely.
Recursive (Intractable):
[Optimize Step 1] -> Simulate Outcome -> [Optimize Step 2] -> Simulate Outcome -> [Optimize Step 3]
One-Shot (Differentiable):
[Optimize Step 1, Step 2, Step 3 Jointly via Simulated Gradients]
A closely related approximation involves treating sequential steps as a simultaneous batch. In frameworks like BINOCULARS, developed by Jiang et al., the agent approximates the long-term value of the remaining budget by calculating a single, high-dimensional optimal batch of points as if they were all going to be sampled at once. The agent then executes just the first point of that batch. This reduction drops the computational complexity from exponential to polynomial time while still preserving a long-term planning horizon.
Mechanism 2: Amortization via Reinforcement Learning
Another way to eliminate the runtime cost of lookahead planning is to shift the computational burden entirely offline through amortization. Instead of computing a lookahead tree during the live experiment, we can train a neural network policy to "learn" non-myopic behavior from simulated data.
This approach frames sequential experimental design as a reinforcement learning (RL) problem. During offline training, a policy network (such as the Transformer Neural Decision Process proposed by Huang et al.) is tasked with maximizing a discounted or cumulative reward over a fixed horizon.
Because the policy is rewarded based on the final outcome of the entire sequence, it naturally learns to avoid greedy traps. As demonstrated by Foster et al., these amortized policies automatically discover sophisticated, non-myopic strategies—such as executing broad, space-filling exploration early on, followed by highly targeted local refinement at the end of the budget. Once trained, this policy network can ingest a history of experimental results and output the next optimal design in milliseconds.
Mechanism 3: Macro-Actions and Trajectory Filtering
When dealing with massive search spaces or complex dynamical systems, even one-shot or amortized methods require structural constraints to keep the trajectory space manageable.
One approach is to restrict the granularity of the action space. Ling introduced the concept of macro-actions to Gaussian Process optimization. Instead of planning individual steps, the algorithm selects pre-defined sequences of actions executed together. This allows the system to tractably look ahead up to 20 steps by drastically reducing the branching factor of the future decision tree.
For environments with complex, non-linear dynamics, tracking the long-term information trajectory requires robust state estimation. Researchers such as Iqbal et al. and Abdulsamad et al. address this by integrating nested particle filters into the experimental design loop. These filters track the evolution of probability distributions over long horizons, ensuring that the non-myopic planner optimizes for the true cumulative information gain of the entire system rather than getting distracted by immediate, noisy observations.