TL;DR
Researchers have developed the first workflow that uses LLM agents to automatically tune the complex configuration options of SMT solvers. By "reading" solver documentation, inspecting source code, and analyzing past experiment logs, these agents achieved significant performance gains—including a 33.1% increase in solved problems for nonlinear integer arithmetic—outperforming even the best human-tuned entries in specific competition categories.
What Problem Does This Paper Address?
Satisfiability Modulo Theories (SMT) solvers are the heavy lifters behind software verification, symbolic execution, and formal methods. Modern solvers like cvc5 are incredibly versatile but notoriously complex, exposing hundreds of configuration options (flags) that control preprocessing, search heuristics, and internal "portfolios" of strategies.
Finding the optimal configuration for a specific workload is often considered a "black art." While traditional automated tuning methods (like SMAC or ParamILS) exist, they are purely data-driven and "blind" to the semantic meaning of the options. They don't know why a flag exists or how it relates to the solver's internal logic. This paper explores whether LLMs can bridge this gap by using their reasoning capabilities to make informed, hypothesis-based configuration proposals.
Methods
The authors introduced a workflow built on the Humanize framework, featuring several novel components:
- Agent-Guided Loop: General-purpose LLM agents (specifically OpenAI’s Codex and Anthropic’s Claude) were tasked with proposing configuration edits. Unlike traditional tuners, these agents were given access to the solver’s documentation, source code, and a persistent history of previous experiment results.
- Flame Chase: To prevent the search from getting stuck, the system used an alternating "Flame Chase" flow where two different LLMs took turns proposing changes, building on the state left by the previous agent.
- Deterministic Harness: A rigorous "harness" acted as the gatekeeper. It measured every proposal using BenchExec, validated the solver's output models using Dolmen (to ensure the solver wasn't just "guessing" correctly), and only retained changes that showed a strict improvement in a specific competition Division.
- Bounded Interface: Agents could only edit specific "argument sections" for their assigned logic divisions, ensuring the search remained focused and manageable.
Datasets
The study utilized the benchmarks from the 2025 and 2026 SMT-COMP ModelValidation tracks, covering 13 Divisions and 36 different logics (e.g., bit-vectors, floating-point arithmetic, arrays).
- Training Set: 15,252 problems sampled from the competition sets, specifically targeting "difficult" problems that timed out in baseline runs.
- Evaluation Set: The complete official competition selections (19,094 problems for 2026 and 59,750 for 2025). The authors carefully separated "test sets" (problems not seen during training) to measure true generalization.
Key Insights and Ideas
- Unlocking "Hidden" Capabilities: One of the most striking findings was that tuning didn't just make the solver faster; it made existing algorithms accessible. For example, some default configurations had internal time limits that prevented effective strategies from ever starting, or used options that were incompatible with the "model generation" mode required for validation. The agents successfully identified and removed these barriers.
- Hypothesis-Based Search: The agents weren't just throwing darts. Analysis of the experiment traces showed that 64.3% of the agents' hypotheses were based on prior measurements. They were effectively "learning" from the logs, connecting accumulated evidence to future proposals.
- The "Pipe Blocking" Discovery: The agents (and subsequent source analysis) identified a subtle bug where a forked worker process would block because its output log exceeded 64 KiB, filling up a pipe. Tuning the portfolio to disable certain workers bypassed this hardware-level bottleneck.
- Family Trade-offs: The paper highlights that configuration is a zero-sum game. A setting that helps one benchmark family (like "ITS" problems) might significantly hurt another (like "SAT14"). The agents had to navigate these trade-offs to find a net-positive configuration.
Key Results
The tuning campaign lasted roughly nine weeks and recorded 4,545 proposals. The results were impressive:
- Generalization: On the 2026 test set, the tuned configurations achieved a 7.8% geometric-mean gain in validated problems across all divisions.
- Massive Wins in NIA: In Nonlinear Integer Arithmetic (NIA), the tuned solver validated 640 problems compared to the official cvc5’s 481—a 33.1% increase.
- Beating the Field: In the Floating Point division, the agent-tuned configurations exceeded the 24-second solved-count of every individual participant in both the 2025 and 2026 competitions.
- Correctness: Despite the aggressive tuning, the process produced zero wrong answers or wrong models, thanks to the integrated validation harness.
Limitations and Future Work
- Computational Cost: The campaign required approximately 24,220 slot hours. While effective, this is a significant resource investment.
- Measurement Noise: The authors noted that small gains (1-3 problems) are often hard to distinguish from random noise on shared hardware, making it difficult for agents to distinguish "lucky" gains from robust improvements.
- Fixed Rules: The agents were constrained by the rules of the harness. They could diagnose problems (like measurement noise) but lacked the authority to change the evaluation criteria or the "stopping rules" of the search.
- Future Directions: The authors suggest that future work could involve agents that not only tune flags but also synthesize new strategy programs or modify the solver's source code directly to resolve the bottlenecks they identify.
Summary updated