CAP: Learning to Optimize from Sparse Observations of Related Tasks

Lin Fu, Siwei Fu, Dong NiarXiv 2026 · arXiv

TL;DR

CAP (Candidate Acquisition Policy) is a novel framework that learns adaptive query policies for black-box optimization by training on "practice functions" synthesized from sparse historical records. By bridging the gap between static past data and interactive reinforcement learning, CAP achieves state-of-the-art performance on complex engineering benchmarks with extremely limited evaluation budgets.

What Problem Does This Paper Address?

In fields like drug discovery or aerospace engineering, evaluating a single design can be incredibly expensive, costing thousands of dollars in materials or hours of supercomputer time. This is the realm of black-box optimization, where we want to find the best configuration with as few trials as possible.

While Bayesian Optimization (BO) is the standard tool for this, it often starts every new task from scratch. We frequently have "records" from past, related experiments, but these are often sparse (only a few data points per task) and fixed (we can't go back and re-run those old experiments). Existing reinforcement learning (RL) methods for optimization usually require an interactive simulator to train a policy, which we don't have. The authors ask: How can we turn a handful of static, historical records into a smart, adaptive strategy for a brand-new task?

Methods

The authors propose CAP (Candidate Acquisition Policy), which operates in three main stages:

  1. Generating Practice Functions: Since they can't interact with the original historical tasks, they fit a joint Gaussian Process model to all available historical records. This model captures both the shared structure across tasks and the unique variations of each. They then sample "practice functions" from this model to create a synthetic, interactive training environment.
  2. Policy Learning via RL: Using these practice functions, they train an actor-critic policy using Proximal Policy Optimization (PPO).
  3. Features: The policy doesn't just look at raw data; it uses a frozen, pretrained TabPFN (a transformer-based model for tabular data) to provide predictive features (mean and uncertainty) for potential query locations.
  4. Architecture: A dual-stream attention mechanism processes the "context" (what we've seen so far) and the "candidates" (where we might go next) to decide the best move.
  5. Objective: Unlike standard BO which often looks only one step ahead, CAP is trained to maximize the final improvement at the end of a fixed budget.
  6. Deployment: Once trained, the policy is frozen. On a new task, it simply takes in observations and suggests the next query, adapting its behavior based on the feedback it receives in real-time.
Figure 1: Overview of CAP. Fixed historical records fit an approximate distribution of coherent practice functions on which complete trajectories train a candidate-selection policy. At each step, the menu and represented observations inform a frozen, externally pretrained TabPFN and the actor–critic; only the selected response updates the observation context. Full trajectories yield PPO returns, while responses of all practice candidates supply auxiliary labels qt only to the training loss. At deployment, one frozen policy per historical archive and budget condition uses current-task feedback and pays for only the selected target query.

Datasets

The researchers evaluated CAP against a wide array of baselines:

  • Synthetic Families: Five task families (Hartmann6, Ackley6, Levy6, Ackley20, Levy20) ranging from 6 to 20 dimensions. Each family provided 20 historical tasks for training and 100 target tasks for testing.
  • Engineering Benchmarks:
    • BOPTEST: A 6D building-control simulation focused on optimizing HVAC setpoints for energy efficiency and comfort.
  • SU2: A 10D aerodynamic design task involving the shape optimization of an RAE2822 airfoil under varying flight conditions.

Key Insights and Ideas

  • Bootstrapping Interactivity: The most significant "aha" moment is that you don't need a perfect simulator to train a high-performing RL policy. A "good enough" generative model built from sparse data can provide sufficient "practice" to teach a policy how to adapt to new feedback.
  • Separation of Prediction and Policy: By using a frozen, general-purpose predictor (TabPFN) for features, the model can focus its learning capacity entirely on the decision-making logic (the policy) specific to the task family.
  • Fixed-Budget Awareness: Because the policy is trained on full trajectories with a known end-point, it learns when to explore and when to exploit more effectively than "greedy" acquisition functions like Expected Improvement (EI).

Key Results

CAP consistently outperformed both traditional BO methods and state-of-the-art transfer learning approaches.

  • Synthetic Performance: CAP achieved the lowest median final regret on 4 out of 5 synthetic families. In high-dimensional (20D) tasks, it showed a particularly strong ability to navigate complex landscapes where traditional methods struggled.
  • Real-World Engineering: On the BOPTEST and SU2 benchmarks, CAP reached better designs faster than any other reported method. For instance, in the SU2 airfoil task, CAP's median final objective was significantly lower than established methods like TuRBO or FSBO.
Table 1: Final median normalized simple regret on 100 target tasks per family (lower is better; best in bold). PFNs4BO supports at most 18 dimensions and is omitted in 20D.

Limitations and Future Work

While powerful, CAP has a few constraints:

  • Task Relatedness: The method assumes that new tasks share some underlying structure with the historical archive. It is not designed to be a "universal" optimizer for completely unrelated problems.
  • Candidate Generation: Currently, the policy learns how to select from a menu of candidates (a mix of global Sobol points and local samples), but it doesn't learn how to generate those candidate locations itself.
  • Archive Size: The paper focuses on small archives (20 tasks). Future work could explore how the policy scales or changes when thousands of historical tasks are available.

Summary updated

Abstract

In experiment-driven black-box optimization, each evaluation can consume materials, equipment time, and other resources, limiting the number of experiments available for a new task. Records from completed related optimization campaigns could help, but learning a sequential query policy remains challenging when task-specific experience consists of only a few sparse histories. We propose CAP (Candidate Acquisition Policy), which models shared structure and task-specific variation in fixed historical records to sample interactive practice functions. On these functions, CAP learns a feedback-conditioned query policy aimed at final optimization performance under a fixed evaluation budget. At deployment, its parameters remain fixed while each new observation informs the next query. Across five synthetic task families spanning 6 and 20 dimensions and two engineering simulation benchmarks, BOPTEST and SU2, CAP attains the lowest median final-budget value among the reported methods on four synthetic families and both engineering benchmarks. These results indicate that sparse records from completed related tasks can support adaptive query decisions on new tasks, making more effective use of limited, costly experimental evaluations.
BibTeX
@misc{fu2026cap,
  author = {Lin Fu and Siwei Fu and Dong Ni},
  title = {{CAP: Learning to Optimize from Sparse Observations of Related Tasks}},
  year = {2026},
  eprint = {2610.41173},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41173},
}
Figure