Automated experiments choose what to measure next with an acquisition function, a score that ranks candidate measurements by how useful they are expected to be. Most acquisition functions are myopic: they act as if the next measurement were the last one. In high-throughput instruments this setup hides a second decision. The software that decides what to measure, the control plane, is kept separate from the system that moves and processes the raw detector bytes, the data plane. A measurement's information does not reach the decision-maker until its payload has crawled through that data plane. If the planner ships a full detector frame now, every later result waits behind it in the queue. If it ships only a cheap summary, the result arrives quickly, but the rich auxiliary signal that would have sharpened the model for future decisions is lost. Today's planners ignore this trade-off.
We propose a planner that chooses where to measure and how much of the result to transport. It looks far ahead over the data-plane queue, which is an engineered and nearly deterministic system, and only one or two steps ahead over the unknown objective. The bet is that this asymmetric horizon captures most of the benefit of non-myopic planning under bandwidth limits. It should also avoid the compounding surrogate error that makes deep lookahead brittle. Finally, it should identify exactly when myopic acquisition is enough: namely, when the data plane is not congested.
Core Claim
Non-myopic acquisition has two known weaknesses. First, deep lookahead over a learned surrogate of the objective "hallucinates" futures. Second, gains plateau after three or four steps. Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees folds the nested lookahead tree into one gradient-based optimization over fantasized posterior samples, and it reports that plateau. BINOCULARS and COPEx both flag compounding model error. Our claim is that in data-plane-limited experiments, the part of the future that most needs long-horizon reasoning is not the objective. It is the transport queue, and the queue can be simulated exactly.
We give the planner a state made of two parts. The first is the posterior over the objective, built from results that have actually arrived. The second is the data-plane backlog $b_t$, the bytes still waiting at decision step $t$. The planner chooses a design $x_t$ and a payload level $m_t$. A summary is the scalar outcome computed at the source worker. A full payload is that scalar plus a high-dimensional auxiliary trace $h(x_t)$. The backlog evolves as
$$b_{t+1} = \max\big(0,\; b_t + s(m_t) - B\,\Delta_t\big),$$
where $s(m_t)$ is the payload size, $B$ is the effective data-plane bandwidth (including the destination bottleneck), and $\Delta_t$ is the control-plane step duration. A result therefore becomes visible to the posterior only after a delay of roughly $b_t/B$. The budget is wall-clock time $T$ rather than a count of evaluations. A full payload costs no extra measurement, but it pushes back the arrival of every later result.
Positioning Against Prior Work
The motivation comes from two papers. Few-Shot Design Optimization by Exploiting Auxiliary Information shows that high-dimensional side data substantially speeds up Bayesian optimization. Its examples are tactile maps from robotic grasps and full learning curves. Its method uses a Transformer Neural Process, a neural model that predicts outcomes from a handful of example observations. However, its policy is myopic Probability of Improvement, and it treats $h(x)$ as free and co-located with the planner. The Mamba Data Worker system in A high-throughput big-data orchestration and processing system for the High Energy Photon Source separates control and data planes at a synchrotron beamline. It ingests 6K × 6K frames at 19 Hz and supports reconfiguring data destinations mid-experiment. Yet its scan steering is manual or heuristic. We supply the missing decision rule, and we use that system's ability to reroute data as the actuator for $m_t$.
Two papers supply our methods. We reuse the batch-to-sequential reduction from BINOCULARS: plan an optimal batch over the remaining horizon, execute only the first action, then replan. We extend the batch from designs $x$ to pairs $(x, m)$ and roll the queue forward exactly. For the objective part, we reuse the auxiliary-data surrogate from the few-shot paper.
Two papers constrain our design. Standard Acquisition Is Sufficient for Asynchronous Bayesian Optimization finds that myopic UCB or LogEI, updated with whatever results have completed, matches more elaborate asynchronous schemes. This is a direct threat, because delayed results are a form of asynchrony. Our hypothesis is that this finding holds while the queue is stable and fails under congestion, because there the payload choice itself controls how stale the posterior becomes. Cost-aware Bayesian Optimization via the Pandora's Box Gittins Index handles heterogeneous costs at near-EI overhead. It assumes the cost $c(x)$ is a fixed function of the design. Our cost depends on the current backlog and spills over onto later steps, which a per-point index cannot express.
The architecture of Scalable Video Conferencing Using SDN Principles influences ours. It runs fast, simple forwarding in switch hardware and keeps rare complex logic in software. We likewise put only the trivial summary computation in the data plane and keep all planning in the control plane.
Research Program
We will build three pieces.
The first is a queue-aware lookahead planner. Its objective horizon is $k_f \in \{1, 2\}$, using one-shot multi-step fantasies. Its queue horizon $k_q$ runs up to the remaining time budget. Optimization is joint over continuous $x$ and binary $m$. We enumerate payload patterns over the horizon, using a small set of structured patterns such as "full every $j$ steps" plus a local search over them. This sidesteps the limitation that reparameterization-based lookahead cannot handle discrete choices.
The second is a value estimate for the auxiliary trace. A full payload adds no immediate improvement, so the planner credits it only with the expected reduction in posterior uncertainty over the next $k_f$ decisions that will see it. Those are the decisions arriving after the delay $b_t/B$.
The third is a data-plane simulator calibrated to the published Mamba Data Worker rates. It uses the ZeroMQ-style multi-stage worker design, includes a destination bottleneck parameter, and replays real benchmark data.
The analysis question is when the planner beats myopic acquisition. We will characterize this regime by the load ratio $\rho = \mathbb{E}[s(m)]/(B\,\Delta)$, where $\rho = 1$ means the queue can just keep up with always-full payloads. We expect myopic sufficiency below $\rho \approx 1$ and growing gains above it.
Baselines
The naive reference is myopic asynchronous LogEI that always ships full payloads and fantasizes pending results, run through the same queue simulator. Its counterpart always ships summaries. Together they bracket the payload decision. They are strong because Standard Acquisition Is Sufficient for Asynchronous Bayesian Optimization argues that exactly this setup is hard to beat. No reported result was found on Lacuna for that paper's numerical scores, so we will rerun it ourselves.
The strongest existing method on the auxiliary-data benchmarks is the Transformer Neural Process with myopic Probability of Improvement from Few-Shot Design Optimization by Exploiting Auxiliary Information. With free data transport, it reports solving 68.5% of gripper-design tasks within 30 trials, against 59.3% for the best prior baseline. On LCBench it reports 89.7% against 82.1%. We will run it unchanged through our simulator with always-full payloads, using the same surrogate weights as ours. In the unconstrained limit ($\rho \to 0$) we must reproduce its numbers; otherwise our setup is faulty. "Beating it" means a higher solved-task fraction at a fixed wall-clock budget when $\rho \geq 1$, with matched performance below that. The comparison is fair only if the extra planning compute is charged to wall-clock time, which we will do.
We also compare against the following methods. First is Pandora's Box Gittins Index, run from PandoraBayesOpt. Each pair $(x, m)$ is treated as an option whose cost is the expected queueing delay under the current backlog, recomputed every step. Second is BINOCULARS over $x$ alone, with a fixed payload policy. Third is a hand-tuned periodic policy that ships full payloads every $j$ steps, with $j$ tuned on held-out tasks using the same tuning budget as our planner.
The central ablation runs our planner deep on both the queue and the objective ($k_f = k_q$). This isolates the claim that depth should be spent on the known dynamics. A second ablation sets $k_q = 1$, which isolates the value of queue foresight.
Why not just…?
Why not just use myopic asynchronous acquisition with fantasized pending points? This is the most serious threat. The asynchronous-sufficiency result says posterior updates on partial results already supply diversity. That argument does not cover the case where the planner's own payload choice decides how much data is pending. Our load-ratio sweep tests this directly. If myopic LogEI with the best tuned fixed payload ratio matches our planner across all values of $\rho$, the claim is wrong, and we will report it as a negative result.
Why not just divide acquisition value by payload cost, or use the Gittins index? The Pandora's Box paper itself documents that dividing Expected Improvement by cost is unstable and suffers from vanishing gradients. The Gittins index with a backlog-dependent cost is a real competitor, and we include it. Its blind spot is the spillover: a full payload's cost falls mostly on later measurements, whose results arrive late. That spillover should matter most under bursty congestion. We test this by injecting congestion from destination-storage stalls, the bottleneck noted in The Modern Research Data Portal. We expect a clear gap there and little gap under steady load.
Why not just buy more bandwidth and always ship everything? When bandwidth is plentiful, the problem disappears, and our method should reduce to the few-shot baseline. The Modern Research Data Portal shows, however, that sender-side decoupling does not fix destination bottlenecks. HEPS-scale data volumes (over 250 TB per day) make congestion the normal operating condition. We report the bandwidth at which our method's advantage vanishes, so practitioners can judge whether the regime applies to them.
Proposed Experiments
We run four testbeds, all at matched wall-clock budgets with 20 seeds each. The first is the gripper-design dataset, where the 16×16 tactile time series is the full payload and the scalar reward is the summary. The second is LCBench, where the full learning curve is the payload and the final validation score is the summary. The third is a set of deceptive synthetic functions (Eggholder, Shubert, Dropwave) with synthetic auxiliary gradients attached. On these, BINOCULARS reports that 12-step variants beat myopic EI, but the file gives no exact scores. The fourth is a simulated X-ray fluorescence raster-mapping task driven by Mamba Data Worker rates. We note that the file lists no public beamline dataset, so this testbed is simulation only.
The metrics are solved-task fraction and simple regret against wall-clock time, plus posterior staleness, measured as the mean age of the newest incorporated result.
Three checks decide the outcome. First, the advantage over myopic asynchronous LogEI should be near zero for $\rho < 1$ and grow with $\rho$; if it is flat, the mechanism is wrong. Second, the shallow-objective, deep-queue planner should match or beat the deep-both ablation at a fraction of the compute. Its margin should widen when we deliberately misspecify the surrogate by training it on a shifted object subset. Third, in the gripper task, the payload choices our planner makes should concentrate on full payloads early and in regions of high uncertainty. If instead they look like a fixed periodic schedule, the tuned periodic baseline should already be enough, and the planner adds nothing.
References
- Efficient Nonmyopic Bayesian Optimization via One-Shot Multi-Step Trees
- BINOCULARS for efficient, nonmyopic sequential experimental design
- Constrained Bayesian Experimental Design via Online Planning
- Few-Shot Design Optimization by Exploiting Auxiliary Information
- A high-throughput big-data orchestration and processing system for the High Energy Photon Source
- Standard Acquisition Is Sufficient for Asynchronous Bayesian Optimization
- Cost-aware Bayesian Optimization via the Pandora's Box Gittins Index
- Scalable Video Conferencing Using SDN Principles
- The Modern Research Data Portal: a design pattern for networked, data-intensive science