TL;DR
ICICLE is a new transformer-based surrogate model for "freeze-thaw" Bayesian optimization that predicts how partially trained models will perform. By using a set-based encoder, it handles any number of hyperparameters and outperforms larger, meta-trained models while being trained entirely on synthetic data.
What Problem Does This Paper Address?
Training modern deep learning models is expensive. To save compute, researchers use freeze-thaw Bayesian optimization (BO): they start many training runs, "freeze" the ones that look unpromising, and "thaw" (continue) the ones that show potential.
To do this effectively, you need a surrogate model that can look at a partial learning curve and predict its final value. Previous state-of-the-art models like FT-PFN used "Prior-Data Fitted Networks" (PFNs)—transformers trained on synthetic data to perform Bayesian inference in a single forward pass. However, FT-PFN had a major limitation: it used a fixed-size encoder (10 slots). If your search space had 11 hyperparameters, it wouldn't work; if it had 2, it was inefficient.
The authors of ICICLE set out to create a surrogate that is architecture-agnostic regarding the number of hyperparameters and more robust to the "messiness" of real-world training (noise, crashes, and plateaus).
Methods
ICICLE (In-Context Inference for Curve-Learning Extrapolation) introduces several key innovations:
- Factored Set Encoder: Instead of a fixed-width vector, ICICLE treats hyperparameters as a set of tokens. Each token is a combination of the hyperparameter's value and a unique "identity tag." This allows the model to process a variable number of hyperparameters (up to 128) using the same weights.
- Attention-Based Pooling: It uses "Pooling by Multihead Attention" (PMA) to compress the variable-sized set of hyperparameters into a single configuration embedding.
- Enhanced Synthetic Prior: The model is trained entirely on synthetic data generated by a Bayesian Neural Network (BNN) prior. The authors improved this prior to include:
- Typed Hyperparameters: Continuous, ordinal, and nominal types.
- Divergence Modeling: Simulating training runs that suddenly crash or plateau.
- Heavy-Tailed Noise: Using Student-t distributions to better mimic real-world observation noise.
- Compact Architecture: Despite its flexibility, ICICLE is tiny. It has only 1.28 million parameters—just 8.7% of the size of its predecessor, FT-PFN.
Datasets
The model was trained purely on synthetic data sampled on the fly during training. It was then evaluated on several major HPO benchmarks:
- LCBench: 35 classification tasks.
- PD1: 28 tasks involving various architectures and datasets.
- TaskSet: 24 tasks focused on optimizer behavior.
- Meta-Album: A massive 86-task benchmark.
- ARLBench: 42 reinforcement learning tasks.
Key Insights and Ideas
- Zero-Shot Generalization: ICICLE outperforms Quick-Tune on the Meta-Album benchmark even though Quick-Tune was meta-trained on real Meta-Album data, while ICICLE only saw synthetic BNN curves. This suggests that a sufficiently diverse synthetic prior can replace expensive meta-learning on real datasets.
- Flexibility via Random Identities: By using task-random identity vectors for hyperparameters, the model learns to distinguish between different inputs without needing them to be in a specific "slot" or order.
- Efficiency: The smaller model size (1.28M parameters) doesn't hurt performance. In fact, it makes the model fast enough to run inference on a CPU via ONNX without needing heavy deep-learning frameworks like PyTorch or JAX.
Key Results
- Meta-Album Performance: ICICLE achieved a lower mean normalized regret (0.103) compared to Quick-Tune (0.133). The advantage was most significant on "extended" (16-hour) training budgets.
- Benchmark Consistency: On LCBench, PD1, and TaskSet, ICICLE's performance was nearly identical to the much larger FT-PFN, proving that the compact architecture is sufficient for curve extrapolation.
- Low Overhead: The amortized optimization overhead is less than 3 seconds per run, making it practically negligible compared to the hours spent training the actual models.
Limitations and Future Work
- Hyperparameter Limit: While flexible, the current implementation is limited to 128 hyperparameters due to the specific identity-tag construction used.
- Resolution at the Limit: The authors note that as regret gets very low, the resolution of the benchmarks makes it harder to distinguish between the top-performing methods.
- Live Evaluation: Most evaluations were performed against pre-computed benchmark tables; testing ICICLE in a "live" training environment for Reinforcement Learning (like ARLBench) remains future work.
The authors have released the code and a pre-trained model, including a scheduler for Ray Tune, making it easy for practitioners to drop ICICLE into their existing pipelines.
Summary updated