Agent-Guided Configuration Tuning for SMT Solvers

Cong Wang, Qiaoyi Zheng, Zhaoguo Wang, Fan LongarXiv 2026 · arXiv

TL;DR

Researchers have developed the first workflow that uses LLM agents to automatically tune the complex configuration options of SMT solvers. By "reading" solver documentation, inspecting source code, and analyzing past experiment logs, these agents achieved significant performance gains—including a 33.1% increase in solved problems for nonlinear integer arithmetic—outperforming even the best human-tuned entries in specific competition categories.

What Problem Does This Paper Address?

Satisfiability Modulo Theories (SMT) solvers are the heavy lifters behind software verification, symbolic execution, and formal methods. Modern solvers like cvc5 are incredibly versatile but notoriously complex, exposing hundreds of configuration options (flags) that control preprocessing, search heuristics, and internal "portfolios" of strategies.

Finding the optimal configuration for a specific workload is often considered a "black art." While traditional automated tuning methods (like SMAC or ParamILS) exist, they are purely data-driven and "blind" to the semantic meaning of the options. They don't know why a flag exists or how it relates to the solver's internal logic. This paper explores whether LLMs can bridge this gap by using their reasoning capabilities to make informed, hypothesis-based configuration proposals.

Methods

The authors introduced a workflow built on the Humanize framework, featuring several novel components:

  • Agent-Guided Loop: General-purpose LLM agents (specifically OpenAI’s Codex and Anthropic’s Claude) were tasked with proposing configuration edits. Unlike traditional tuners, these agents were given access to the solver’s documentation, source code, and a persistent history of previous experiment results.
  • Flame Chase: To prevent the search from getting stuck, the system used an alternating "Flame Chase" flow where two different LLMs took turns proposing changes, building on the state left by the previous agent.
  • Deterministic Harness: A rigorous "harness" acted as the gatekeeper. It measured every proposal using BenchExec, validated the solver's output models using Dolmen (to ensure the solver wasn't just "guessing" correctly), and only retained changes that showed a strict improvement in a specific competition Division.
  • Bounded Interface: Agents could only edit specific "argument sections" for their assigned logic divisions, ensuring the search remained focused and manageable.

Datasets

The study utilized the benchmarks from the 2025 and 2026 SMT-COMP ModelValidation tracks, covering 13 Divisions and 36 different logics (e.g., bit-vectors, floating-point arithmetic, arrays).

  • Training Set: 15,252 problems sampled from the competition sets, specifically targeting "difficult" problems that timed out in baseline runs.
  • Evaluation Set: The complete official competition selections (19,094 problems for 2026 and 59,750 for 2025). The authors carefully separated "test sets" (problems not seen during training) to measure true generalization.

Key Insights and Ideas

  • Unlocking "Hidden" Capabilities: One of the most striking findings was that tuning didn't just make the solver faster; it made existing algorithms accessible. For example, some default configurations had internal time limits that prevented effective strategies from ever starting, or used options that were incompatible with the "model generation" mode required for validation. The agents successfully identified and removed these barriers.
  • Hypothesis-Based Search: The agents weren't just throwing darts. Analysis of the experiment traces showed that 64.3% of the agents' hypotheses were based on prior measurements. They were effectively "learning" from the logs, connecting accumulated evidence to future proposals.
  • The "Pipe Blocking" Discovery: The agents (and subsequent source analysis) identified a subtle bug where a forked worker process would block because its output log exceeded 64 KiB, filling up a pipe. Tuning the portfolio to disable certain workers bypassed this hardware-level bottleneck.
  • Family Trade-offs: The paper highlights that configuration is a zero-sum game. A setting that helps one benchmark family (like "ITS" problems) might significantly hurt another (like "SAT14"). The agents had to navigate these trade-offs to find a net-positive configuration.

Key Results

The tuning campaign lasted roughly nine weeks and recorded 4,545 proposals. The results were impressive:

  • Generalization: On the 2026 test set, the tuned configurations achieved a 7.8% geometric-mean gain in validated problems across all divisions.
  • Massive Wins in NIA: In Nonlinear Integer Arithmetic (NIA), the tuned solver validated 640 problems compared to the official cvc5’s 481—a 33.1% increase.
  • Beating the Field: In the Floating Point division, the agent-tuned configurations exceeded the 24-second solved-count of every individual participant in both the 2025 and 2026 competitions.
  • Correctness: Despite the aggressive tuning, the process produced zero wrong answers or wrong models, thanks to the integrated validation harness.

Limitations and Future Work

  • Computational Cost: The campaign required approximately 24,220 slot hours. While effective, this is a significant resource investment.
  • Measurement Noise: The authors noted that small gains (1-3 problems) are often hard to distinguish from random noise on shared hardware, making it difficult for agents to distinguish "lucky" gains from robust improvements.
  • Fixed Rules: The agents were constrained by the rules of the harness. They could diagnose problems (like measurement noise) but lacked the authority to change the evaluation criteria or the "stopping rules" of the search.
  • Future Directions: The authors suggest that future work could involve agents that not only tune flags but also synthesize new strategy programs or modify the solver's source code directly to resolve the bottlenecks they identify.

Summary updated

Abstract

SMT solvers expose many configuration choices that determine how well their existing algorithms perform on a workload. We present, to our knowledge, the first workflow that uses general-purpose LLM agents to tune SMT solver configurations. Agents read solver documentation, source code, and persistent experiment records to propose option changes, and a deterministic harness measures each proposal, validates the returned models, and selects changes separately for each competition Division. Over roughly nine weeks, our tuning campaign recorded 4,545 proposals for cvc5 across 13 Divisions and 36 logics. We evaluate the tuned configurations on the 2026 and 2025 SMT-COMP ModelValidation benchmarks under a 24-second cutoff. On problems not used for training, they validate more problems than official cvc5 in ten of the 13 Divisions in 2026 and in eight in 2025, with geometric-mean gains of 7.8% and 6.8% across Divisions. On the complete official selections, which include training problems, the tuned floating-point counts exceed those of every individual competition participant in both years. Further studies show how configuration changes make existing algorithms effective by changing access to strategies, scheduling, and option compatibility, and how benchmark families trade off against one another. The agents' experiment traces contain hypothesis-based proposals in every tuning group, and 64.3% of the extracted hypotheses draw on prior measurements. These results show that agent-guided tuning can specialize a mature SMT solver and that accumulated experimental evidence informs sustained search.
BibTeX
@misc{wang2026agent,
  author = {Cong Wang and Qiaoyi Zheng and Zhaoguo Wang and Fan Long},
  title = {{Agent-Guided Configuration Tuning for SMT Solvers}},
  year = {2026},
  eprint = {2611.20484},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20484},
}

Loading related content...

Figure