When a database schema changes between versions, tools can list what changed: a column is gone, a new one appeared, a table now has fewer fields. What the list does not say is what the developers meant. A vanished user_email column next to a new email column may be a rename, which keeps the data. Or it may be a drop followed by an unrelated add, which deletes it. Schema modification operators (SMOs) are a small, fixed vocabulary of declarative edits such as rename, drop, split, merge and decompose. Systems built on them, like PRISM, can turn a correct operator sequence into formal mappings and rewrite old queries automatically. But they assume someone has already written the correct sequence. PRISM still left 2.8% of Wikipedia's evolution steps to manual intervention, because the tool could not recover intent where the change destroys information.
We propose treating exactly those steps as questions for a Delphi panel. The Delphi method is an expert-consensus protocol with anonymous independent ratings, several rounds, and statistical feedback between rounds. Our version has two changes. First, the formal SMO machinery decides which operators actually need a judgment: an edit that can be reversed without loss is accepted automatically. Second, the feedback between rounds includes what each candidate operator would break, namely which legacy queries lose their rewriting and which data become unrecoverable. The panel can be LLM agents, each representing one application module's query workload, or human data stewards. The panel only deliberates on the few ambiguous, lossy decisions. It sees consequences computed by a verifier, not just its peers' opinions.
Core Claim
Inferring intended schema-change operators is hard in a narrow place. Most steps are mechanical. A minority are ambiguous and information-destructive, and there the choice decides whether data and queries survive. Our claim is that an operator-level Delphi protocol handles this minority better than the obvious alternatives, at equal compute. The protocol has three parts: a triage step based on invertibility, anonymous independent first drafts, and feedback annotated with formal consequences. We measure "better" as more steps matching developer intent and more legacy queries preserved. We expect this to beat single-model generation, majority voting over samples, and free-form multi-agent debate.
The novelty lies in coupling the two directions in both directions at once. The formal operators make Delphi items executable and checkable, which answers Delphi's usual limitation that its output is natural-language statements. Delphi supplies the intent that the formal system cannot infer. No paper in our search does either. The Delphi applications we found, BRIDGE for health-data governance and VICCCS for diagnostic nomenclature, produce informal playbooks and definitions. The multi-agent consensus studies, Voting or Consensus? and Emergence of Biased Consensus, use question-answering and synthetic choice tasks with no structured artifact.
Research Program
The substrate is the operator-annotated evolution histories from Automating Database Schema Evolution in Information System Upgrades, which reduced upgrades of Wikipedia/MediaWiki, Joomla!, Zen Cart, TikiWiki and others to sequences of atomic operators. These histories give us something rare for consensus research: a ground truth for intent. For each version pair we know the operator sequence the developers' change actually corresponds to.
Task. Given schema versions $S$ and $S'$, sample instance data, and the old and new query workloads split by application module, recover the operator sequence. We also recover the integrity-constraint policy (IGNORE, CHECK or ENFORCE) for each constraint change, following PRISM++.
Triage. A diff-based generator enumerates the candidate operator sequences consistent with $S \to S'$. We compile each candidate into logical dependencies in the PRISM style and test whether it has a quasi-inverse, meaning it can be undone without loss. Components where every candidate is invertible, or where all candidates agree, are auto-accepted. The remaining components become Delphi items. We expect these to be dominated by table-level merges, splits and drops, which EvoSchema found to be the perturbations that most damage downstream Text-to-SQL models.
Protocol. Each panelist is an agent conditioned on one module's queries.
- Round 1. Each panelist drafts its preferred operator for every item without seeing the others. This mirrors the All-Agents Drafting step that Voting or Consensus? found reduced sycophancy.
- Between rounds. Panelists receive the rating distribution and anonymized justifications. They also receive, for every candidate, the verifier's report: the fraction of their module's legacy queries that still rewrite, and the tuples that become unrecoverable.
- Stopping rule. An item is ratified at a pre-registered 67% agreement threshold, borrowed from VICCCS. Rounds are capped at three, because both agent studies report that longer deliberation drifts or collapses.
- Diversity. Panel sampling temperatures are mixed in the 1.3–1.7 range that Okawa et al. found reduced biased lock-in.
Human pilot. A small pilot replaces the agents with practitioners on a subset of items. It tests whether triage reduces the reviewer fatigue that BRIDGE documented as written feedback declined across phases.
Baselines
Naive diff. The do-nothing reference is a diff-based migration that models every change as drop plus add. This is the imperative-script default. It sets the floor for data preservation and query survival. PRISM reports that legacy query support without rewriting is 16%, against 74% worst case with correct operators, on Wikipedia (PRISM). The 74% figure is the ceiling for operator-given rewriting and the natural target. No reported result was found on Lacuna for automatically inferring operator sequences from version pairs.
Single model. One LLM receives the full context, including every module's queries, and emits the operator sequence directly. Its output is checked by the same verifier and gets one repair attempt.
Majority voting. $N$ independent samples from the same model are followed by a majority vote per item, with the same verifier filtering. The number of samples is set so that total tokens equal the Delphi protocol's budget. This is the strongest competitor. Voting or Consensus? found voting beats consensus on reasoning tasks by 13.2% on average (MuSR 54.8% versus 28.4%), and operator choice has a reasoning component.
Free-form debate. Agents conditioned on modules chat openly at equal token budget.
Ablations. Two ablations isolate the ingredients. The first is Delphi without consequence annotations, which is classic peer-statistics feedback. The second is Delphi applied to all components rather than triaged ones.
Fairness. All methods use the same model, prompts drawn from the same template, the same verifier, and the same tuning split (MediaWiki). Evaluation is on held-out systems. Winning means higher accuracy on ambiguous items and higher query survival than voting at equal tokens, with non-overlapping bootstrap intervals over version pairs.
Why not just…?
Why not just have one strong LLM write the migration? LLMATCH reports that monolithic prompting degrades on large enterprise schemas: its hierarchical method reached F1 0.4 on mimic_iii-omop, against 0.2 for ReMatch and 0.04 for COMA. However, that result comes from matching, not evolution. The single-model baseline, given the same verifier, tests the question directly. If it matches Delphi on ambiguous items, the multi-perspective element adds nothing and we will report that.
Why not just sample several proposals and vote? This is the real threat. Kaesberg et al.'s evidence favors voting on reasoning-heavy tasks. Our bet is that ambiguous schema steps are mostly intent questions, closer to their knowledge tasks, where consensus won by 2.8%. We also bet that consequence feedback gives consensus information that voting never uses. The consequence-free ablation separates these two bets. If voting with verifier filtering matches full Delphi, the contribution shrinks to triage plus annotations. We would then recommend the cheaper voting pipeline.
Why not just let agents debate freely? Both agent studies in our search report failures from unstructured interaction: drift over long rounds and snap-to-bias within one or two rounds at low temperature. We include debate at equal budget, and we measure how often a panel ratifies an information-destructive operator that the ground truth does not contain.
Why not just use a schema matcher? Matchers like COMA output correspondences, not operators with executable semantics. Correspondences cannot distinguish rename from drop-plus-add once instance overlap is small. We run a matcher-derived baseline to confirm this.
Proposed Experiments
Main comparison. The main experiment uses Wikipedia plus the other information systems in the Curino testbed, with MediaWiki reserved for tuning. We report four metrics:
- per-item operator accuracy on triaged ambiguous components;
- end-to-end legacy query survival under PRISM-style rewriting, compared against the 16% and 74% reference points;
- the fraction of source tuples recoverable through the inferred sequence's quasi-inverse on real instance data;
- the number of items deliberated per version.
Decisive check 1: consequence feedback. Full Delphi must beat the consequence-free ablation on data preservation. Otherwise the coupling claim fails.
Decisive check 2: voting. Full Delphi must beat voting at equal tokens on ambiguous items. Otherwise we adopt voting.
Decisive check 3: triage. Triage should cut deliberated items by an order of magnitude with no loss of accuracy. We expect a large cut because PRISM automated 97.2% of steps, which implies ambiguity is rare.
Generalization. A held-out test on the Ensembl history, which PRISM++ used for coverage, checks whether triage thresholds and panel settings carry over to a scientific database.
Human pilot. The pilot compares practitioner accuracy and completion rates under triaged versus full-item Delphi.
References
- Graceful database schema evolution: the PRISM workbench
- BRIDGE pilot study: a bilateral regulatory investigation of data governance and exchange
- The Vascular Impairment of Cognition Classification Consensus Study
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Emergence of Biased Consensus in Multi-Agent LLM Debates
- Automating Database Schema Evolution in Information System Upgrades
- Update Rewriting and Integrity Constraint Maintenance in a Schema Evolution Support System: PRISM++
- EVOSCHEMA: TOWARDS TEXT-TO-SQL ROBUSTNESS AGAINST SCHEMA EVOLUTION
- LLMATCH: A Unified Schema Matching Framework with Large Language Models