TL;DR
CAP (Candidate Acquisition Policy) is a novel framework that learns adaptive query policies for black-box optimization by training on "practice functions" synthesized from sparse historical records. By bridging the gap between static past data and interactive reinforcement learning, CAP achieves state-of-the-art performance on complex engineering benchmarks with extremely limited evaluation budgets.
What Problem Does This Paper Address?
In fields like drug discovery or aerospace engineering, evaluating a single design can be incredibly expensive, costing thousands of dollars in materials or hours of supercomputer time. This is the realm of black-box optimization, where we want to find the best configuration with as few trials as possible.
While Bayesian Optimization (BO) is the standard tool for this, it often starts every new task from scratch. We frequently have "records" from past, related experiments, but these are often sparse (only a few data points per task) and fixed (we can't go back and re-run those old experiments). Existing reinforcement learning (RL) methods for optimization usually require an interactive simulator to train a policy, which we don't have. The authors ask: How can we turn a handful of static, historical records into a smart, adaptive strategy for a brand-new task?
Methods
The authors propose CAP (Candidate Acquisition Policy), which operates in three main stages:
- Generating Practice Functions: Since they can't interact with the original historical tasks, they fit a joint Gaussian Process model to all available historical records. This model captures both the shared structure across tasks and the unique variations of each. They then sample "practice functions" from this model to create a synthetic, interactive training environment.
- Policy Learning via RL: Using these practice functions, they train an actor-critic policy using Proximal Policy Optimization (PPO).
- Features: The policy doesn't just look at raw data; it uses a frozen, pretrained TabPFN (a transformer-based model for tabular data) to provide predictive features (mean and uncertainty) for potential query locations.
- Architecture: A dual-stream attention mechanism processes the "context" (what we've seen so far) and the "candidates" (where we might go next) to decide the best move.
- Objective: Unlike standard BO which often looks only one step ahead, CAP is trained to maximize the final improvement at the end of a fixed budget.
- Deployment: Once trained, the policy is frozen. On a new task, it simply takes in observations and suggests the next query, adapting its behavior based on the feedback it receives in real-time.
Datasets
The researchers evaluated CAP against a wide array of baselines:
- Synthetic Families: Five task families (Hartmann6, Ackley6, Levy6, Ackley20, Levy20) ranging from 6 to 20 dimensions. Each family provided 20 historical tasks for training and 100 target tasks for testing.
- Engineering Benchmarks:
- BOPTEST: A 6D building-control simulation focused on optimizing HVAC setpoints for energy efficiency and comfort.
- SU2: A 10D aerodynamic design task involving the shape optimization of an RAE2822 airfoil under varying flight conditions.
Key Insights and Ideas
- Bootstrapping Interactivity: The most significant "aha" moment is that you don't need a perfect simulator to train a high-performing RL policy. A "good enough" generative model built from sparse data can provide sufficient "practice" to teach a policy how to adapt to new feedback.
- Separation of Prediction and Policy: By using a frozen, general-purpose predictor (TabPFN) for features, the model can focus its learning capacity entirely on the decision-making logic (the policy) specific to the task family.
- Fixed-Budget Awareness: Because the policy is trained on full trajectories with a known end-point, it learns when to explore and when to exploit more effectively than "greedy" acquisition functions like Expected Improvement (EI).
Key Results
CAP consistently outperformed both traditional BO methods and state-of-the-art transfer learning approaches.
- Synthetic Performance: CAP achieved the lowest median final regret on 4 out of 5 synthetic families. In high-dimensional (20D) tasks, it showed a particularly strong ability to navigate complex landscapes where traditional methods struggled.
- Real-World Engineering: On the BOPTEST and SU2 benchmarks, CAP reached better designs faster than any other reported method. For instance, in the SU2 airfoil task, CAP's median final objective was significantly lower than established methods like TuRBO or FSBO.
Limitations and Future Work
While powerful, CAP has a few constraints:
- Task Relatedness: The method assumes that new tasks share some underlying structure with the historical archive. It is not designed to be a "universal" optimizer for completely unrelated problems.
- Candidate Generation: Currently, the policy learns how to select from a menu of candidates (a mix of global Sobol points and local samples), but it doesn't learn how to generate those candidate locations itself.
- Archive Size: The paper focuses on small archives (20 tasks). Future work could explore how the policy scales or changes when thousands of historical tasks are available.
Summary updated