The Database as Fact-Checker for LLM Tuning Advice

Databases such as PostgreSQL and MySQL expose hundreds of configuration settings ("knobs") for memory, disk writes, and query planning. Choosing their values has long been a job for human administrators working from rules of thumb. Recent systems ask a large language model (LLM) to read the manuals and forums and suggest which knobs matter and what values to try. The suggestions then seed a numerical search. The weak point is hallucination, meaning confident but false statements. The LLM may invent a knob, get its unit wrong, claim it takes effect without a restart when it does not, or assert that raising it helps when, on this workload and this hardware, it hurts. The best current pipeline checks only whether a suggested knob exists and has the right type. That pipeline still leaves 6% of its extracted knowledge wrong, and it is confined to offline replicas because bad suggestions can crash the server (GPTuner).

This proposal treats every piece of LLM tuning advice as a testable claim and uses the database itself to test it. Most advice about knobs predicts something the database can show cheaply. The catalog shows whether a knob exists. A throwaway instance shows whether it boots. The query planner's cost-only output (EXPLAIN, which shows the plan without running the query) shows whether a knob changes the plan. A short single-query probe shows whether a sort stops spilling to disk. Only claims that survive these checks become priors for the optimizer. The central bet is that the harmful hallucinations in tuning are rarely errors about world facts. More often they are claims that are true in general but false for this engine version, workload, or machine. Only execution can catch those, and catching them should pay for itself in fewer wasted and unsafe tuning trials.

Core Claim

LLM knob advice can be broken into typed claims. We use five types: existence and type, unit and valid range, restart requirement, direction of effect on an observable quantity under a stated condition, and pairwise interaction. A ladder of increasingly expensive checks run against the database settles each claim as supported, refuted, or untestable at the budget. We predict three things. First, the first two claim types are where the LLM is usually right and cheap catalog checks already work. Second, direction-of-effect claims are where the residual errors live. Third, filtering those claims by execution reduces both tuning iterations and the number of trials that crash or degrade the server, at the same total wall-clock budget, compared with catalog-only verification.

This differs from the closest work in what gets verified. GPTuner validates suggestions against the pg_settings catalog and takes a majority vote over an ensemble of prompts. That checks static metadata and model self-agreement, not behavior. ANDROMEDA grounds the LLM in retrieved manuals and live telemetry, but retrieval cannot tell you whether a manual's general advice holds for your workload. The file also reports that ANDROMEDA confuses knobs with similar descriptions, such as sql_notes versus log_warnings. A behavioral probe distinguishes such knobs directly, because they change different observables.

Research Program

Claim extraction. The first component turns free-text LLM advice into claim records. An example record: "raising work_mem to at least 64MB removes the external merge sort in query Q7." Each record carries a knob, a proposed value or direction, a predicted observable, and a condition. For open-source models we constrain extraction to a fixed schema, since Li et al. report frequent formatting violations. Once claims are verified, the configuration file is written directly from the records, never re-synthesized by the LLM. This follows the finding in Structured Hallucination in Tool-Using Agents: LLMs that reformat structured data corrupt 11–13% of values while producing valid syntax. We reuse its bypass idea for configuration output.

Verification ladder. The ladder has four rungs, ordered by cost:

  1. Catalog lookup, which costs milliseconds.
  2. Apply-and-boot on a scratch instance, which costs seconds and catches invalid values and restart requirements.
  3. Planner-only checks using EXPLAIN, which run no queries and cover planner knobs such as page-cost ratios and join enablement.
  4. Micro-probes. These are single representative queries, drawn from the workload, run on a small data sample with EXPLAIN ANALYZE, checking the specific observable a claim predicts: disk spill, buffer hits, or plan operator.

A claim advances only if cheaper rungs cannot settle it. Every second spent on verification is charged against the tuning budget.

Using verified claims in the tuner. Supported claims shape the search space in the way GPTuner's knob pruning and value ranges do. Refuted claims remove regions. Untestable claims keep a weak prior. The downstream optimizer is held fixed, either GPTuner's coarse-to-fine Bayesian optimization or SMAC. That way any gain is attributable to the knowledge filter and not to the search algorithm.

Hallucination census. The last component is measurement. Across GPT-4o, Claude-3-Opus, Qwen2-7B, and Llama-3, and across PostgreSQL 10 and 14.9, MySQL 5.7 and 8.0, and TiDB 8.2, we record which claim types each model gets wrong. We then trace which of those errors actually change the tuned result. This addresses a contradiction flagged in the file: Li et al. report easy zero-shot transfer across engines, while E2ETune reports engine lock-in. A per-version census of refuted claims shows where parametric knowledge goes stale. The census also tests a claim from the hallucination literature. Siren's Song and Holt et al. argue that errors compound when early false steps are re-ingested. We test whether an early refuted claim, left unfiltered, measurably biases later iterative recommendations.

Baselines

The naive reference points are vendor defaults and plain SMAC. GPTuner reports that defaults give 240 tx/s on MySQL 8.0 TPC-C. Li et al. report that SMAC needed 375 iterations to match what GPT-4-Turbo and Qwen2-7B reached in 9 to 14. SMAC is the standard LLM-free optimizer; the HPO evaluation found it the most stable among Bayesian optimization and reinforcement learning variants, with a 21.17% average improvement. We run it with its knob set chosen by SHAP importance, as that paper recommends.

The strongest existing competitor is GPTuner. It uses the same kind of LLM knowledge, with catalog-level error correction that raised knowledge accuracy from 82% to 94%. It reports 1029 tx/s on MySQL 8.0 TPC-C and a 44.4% latency reduction on TPC-H within 20 iterations. We rerun it on identical hardware, scale factors, LLM, and total wall-clock budget. Our verification time is subtracted from our search iterations, so equal budget means fewer optimizer trials for us.

Centrum is the strongest text-free tuner. It reports 19.2% higher tuned throughput or lower latency than 21 tuners in physical experiments. We run it at the same budget to check whether knowledge is needed at all.

The key ablation is catalog-only verification: our pipeline with rungs 3 and 4 removed. Everything else stays identical, including claim extraction, schema output, and optimizer. The only difference is whether behavioral claims are tested.

Beating the baselines means one of two outcomes at equal wall-clock budget, averaged over at least five seeds and all workloads, with confidence intervals that exclude zero. Either we reach GPTuner's final performance in fewer trials with fewer crash or boot-failure trials, or we reach better final performance in the same time. Matching the catalog-only ablation would mean the core idea failed.

Why not just…?

Why not just use self-consistency or prompt-ensemble voting? GPTuner already does this. Voting catches unstable errors. Agrawal et al. show that hallucinated references become inconsistent under indirect probing. Voting cannot catch a claim that every sample agrees on but that is false for this workload. We test this by comparing ensemble agreement against execution verdicts on the same claims. If high-agreement claims are almost never refuted, voting suffices, and we will report that.

Why not just use retrieval over manuals, as ANDROMEDA does? Manuals state general behavior, not behavior on your hardware and queries, and retrieval inherits ANDROMEDA's confusion of similarly described knobs. We run a retrieval-augmented claim generator as an additional arm. If its refutation rate on direction-of-effect claims is near zero, retrieval makes the execution rungs unnecessary. This is a real threat for claims about existence and ranges, which is why those claims stop at the catalog rung.

Why not just drop the LLM and run Centrum or SMAC? The iteration gap reported by Li et al., 9–14 versus 375, suggests knowledge priors matter when every trial replays a workload. But budgets differ across papers, so this is not settled. Our equal-budget comparison against Centrum settles it on our hardware.

Why not just use a fine-tuned one-shot predictor like E2ETune? It needs a retraining corpus per engine and trusts a surrogate cost model for its labels. Our approach needs no training and checks claims against the real engine. We include E2ETune on PostgreSQL, where its model applies, and report cross-version results only for methods that can run there.

Proposed Experiments

Claim-level accuracy. We start on GPTuner's 150 hand-annotated PostgreSQL knobs and its 50 factually inconsistent summaries, then extend the annotations to direction-of-effect claims for the TPC-H, JOB, and TPC-C workloads. Ground truth for those claims comes from full-workload runs. We report the refutation rate by claim type and model, and the precision and recall of each rung against ground truth. We also report the ladder's cost per claim.

End-to-end tuning. We tune TPC-C, TPC-H at scale factors 1, 10, and 50, JOB, SYSBENCH, and YCSB on the engine versions listed above. We report final throughput or latency, trials to reach 90% of the best configuration found by any method (a target borrowed from EMIT), and counts of crashes, boot failures, and trials that degrade performance below default.

Decisive checks. The first is causal. We inject known-false direction-of-effect claims, taken from the refuted census, into GPTuner's knowledge base and measure the resulting regret. If regret barely moves, behavioral hallucinations do not matter and the proposal's premise fails. The second is the head-to-head against the catalog-only ablation at equal budget. The third is the cross-version test. We take claims generated from parametric memory for PostgreSQL 10 and apply them to 14.9, and vice versa. We predict refutations concentrate in knobs whose semantics changed between versions, which would give the first direct map of where zero-shot LLM tuning knowledge goes stale.