ICICLE: In-Context Inference for Curve-Learning Extrapolation

Laurits Fredsgaard, Michael Riis Andersen, Mikkel N. SchmidtarXiv 2026 · arXiv

TL;DR

ICICLE is a new transformer-based surrogate model for "freeze-thaw" Bayesian optimization that predicts how partially trained models will perform. By using a set-based encoder, it handles any number of hyperparameters and outperforms larger, meta-trained models while being trained entirely on synthetic data.

What Problem Does This Paper Address?

Training modern deep learning models is expensive. To save compute, researchers use freeze-thaw Bayesian optimization (BO): they start many training runs, "freeze" the ones that look unpromising, and "thaw" (continue) the ones that show potential.

To do this effectively, you need a surrogate model that can look at a partial learning curve and predict its final value. Previous state-of-the-art models like FT-PFN used "Prior-Data Fitted Networks" (PFNs)—transformers trained on synthetic data to perform Bayesian inference in a single forward pass. However, FT-PFN had a major limitation: it used a fixed-size encoder (10 slots). If your search space had 11 hyperparameters, it wouldn't work; if it had 2, it was inefficient.

The authors of ICICLE set out to create a surrogate that is architecture-agnostic regarding the number of hyperparameters and more robust to the "messiness" of real-world training (noise, crashes, and plateaus).

Methods

ICICLE (In-Context Inference for Curve-Learning Extrapolation) introduces several key innovations:

  1. Factored Set Encoder: Instead of a fixed-width vector, ICICLE treats hyperparameters as a set of tokens. Each token is a combination of the hyperparameter's value and a unique "identity tag." This allows the model to process a variable number of hyperparameters (up to 128) using the same weights.
  2. Attention-Based Pooling: It uses "Pooling by Multihead Attention" (PMA) to compress the variable-sized set of hyperparameters into a single configuration embedding.
  3. Enhanced Synthetic Prior: The model is trained entirely on synthetic data generated by a Bayesian Neural Network (BNN) prior. The authors improved this prior to include:
  4. Typed Hyperparameters: Continuous, ordinal, and nominal types.
  5. Divergence Modeling: Simulating training runs that suddenly crash or plateau.
  6. Heavy-Tailed Noise: Using Student-t distributions to better mimic real-world observation noise.
  7. Compact Architecture: Despite its flexibility, ICICLE is tiny. It has only 1.28 million parameters—just 8.7% of the size of its predecessor, FT-PFN.
Figure 1: ICICLE’s freeze–thaw Bayesian-optimization loop. (1) Observed curve prefixes form the context; later fidelities are queries. (2) Feature self-attention and attention pooling encode each configuration once, using one value–identity token per hyperparameter. (3) Observations combine the gathered configuration embedding with separate value and fidelity encodings; queries omit the value encoding. (4) Each query attends independently to the context, and a 1,000-bin head predicts its value distribution. The optimizer advances a configuration and adds the observation to the context. Appendix B.2 gives the architecture details.

Datasets

The model was trained purely on synthetic data sampled on the fly during training. It was then evaluated on several major HPO benchmarks:

  • LCBench: 35 classification tasks.
  • PD1: 28 tasks involving various architectures and datasets.
  • TaskSet: 24 tasks focused on optimizer behavior.
  • Meta-Album: A massive 86-task benchmark.
  • ARLBench: 42 reinforcement learning tasks.

Key Insights and Ideas

  • Zero-Shot Generalization: ICICLE outperforms Quick-Tune on the Meta-Album benchmark even though Quick-Tune was meta-trained on real Meta-Album data, while ICICLE only saw synthetic BNN curves. This suggests that a sufficiently diverse synthetic prior can replace expensive meta-learning on real datasets.
  • Flexibility via Random Identities: By using task-random identity vectors for hyperparameters, the model learns to distinguish between different inputs without needing them to be in a specific "slot" or order.
  • Efficiency: The smaller model size (1.28M parameters) doesn't hurt performance. In fact, it makes the model fast enough to run inference on a CPU via ONNX without needing heavy deep-learning frameworks like PyTorch or JAX.

Key Results

  • Meta-Album Performance: ICICLE achieved a lower mean normalized regret (0.103) compared to Quick-Tune (0.133). The advantage was most significant on "extended" (16-hour) training budgets.
  • Benchmark Consistency: On LCBench, PD1, and TaskSet, ICICLE's performance was nearly identical to the much larger FT-PFN, proving that the compact architecture is sufficient for curve extrapolation.
  • Low Overhead: The amortized optimization overhead is less than 3 seconds per run, making it practically negligible compared to the hours spent training the actual models.
Comparison of mean normalized regret on Meta-Album tasks.

Limitations and Future Work

  • Hyperparameter Limit: While flexible, the current implementation is limited to 128 hyperparameters due to the specific identity-tag construction used.
  • Resolution at the Limit: The authors note that as regret gets very low, the resolution of the benchmarks makes it harder to distinguish between the top-performing methods.
  • Live Evaluation: Most evaluations were performed against pre-computed benchmark tables; testing ICICLE in a "live" training environment for Reinforcement Learning (like ARLBench) remains future work.

The authors have released the code and a pre-trained model, including a scheduler for Ray Tune, making it easy for practitioners to drop ICICLE into their existing pipelines.

Summary updated

Abstract

Freeze-thaw Bayesian optimization allocates training budget by predicting how partially observed learning curves will continue. LC-PFN first applied prior-data fitted networks (PFNs) to learning-curve extrapolation, and FT-PFN extended this approach to configuration-aware freeze-thaw optimization with a ten-slot hyperparameter encoder. We introduce In-Context Inference for Curve-Learning Extrapolation (ICICLE), an in-context surrogate that accepts a variable number of hyperparameters through a shared set encoder and is trained on a synthetic prior covering typed hyperparameters, heavy-tailed noise, quantization, and divergence. ICICLE has 1.28 million parameters (8.7% of FT-PFN's count) and produces similar mean-regret trajectories to FT-PFN on LCBench, PD1, and TaskSet. On the 86-task Meta-Album benchmark with 33 hyperparameters, ICICLE reaches lower mean normalized regret than Quick-Tune overall and on extended, with no clear difference on mini or micro. It does so without pre-training or meta-training on Meta-Album tasks, whereas Quick-Tune is meta-trained on related Meta-Album tasks. Our release samples synthetic training tasks on the fly and provides a freeze-thaw scheduler for Ray Tune and an ONNX model for inference without JAX or PyTorch.
BibTeX
@misc{fredsgaard2026icicle,
  author = {Laurits Fredsgaard and Michael Riis Andersen and Mikkel N. Schmidt},
  title = {{ICICLE: In-Context Inference for Curve-Learning Extrapolation}},
  year = {2026},
  eprint = {2611.24469},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24469},
}
Figure