When code stops matching the design documents that describe it, someone has to decide which one is wrong. Sometimes the code is the problem: a shortcut crossed a module boundary, and the patch should be refactored. Sometimes the document is the problem: the team deliberately changed the design and never wrote it down. Automated drift detectors flag the mismatch but cannot tell these two cases apart. They also produce enough false alarms that developers stop reading them. We propose to treat each flagged mismatch as a small expert-consensus problem. A panel of independently prompted language-model reviewers, each responsible for one module, scores the deviation without seeing who said what. Over two or three rounds they see only the group's median and spread plus rationales that cite actual lines of code or design text. A single "chief editor" agent holds the project's design charter and decides only when the panel has converged.
The central bet is that the panel's disagreement is the most useful output. When the panel stays split after seeing each other's evidence, the case should go to a human. This gives a principled answer to the alert-fatigue and human-in-the-loop problems that the architecture-erosion literature keeps raising.
Core Claim
Architectural drift means the gradual divergence between a system's intended design (blueprints, architecture decision records, specifications) and its actual code. We claim three things about it.
First, adjudicating a drift event is a three-way decision: refactor the code, amend the design record, or accept the patch as a harmless local deviation. It should not be treated as binary violation detection.
Second, a Delphi protocol can reach this decision reliably. The Delphi method is a structured procedure from clinical and policy research in which experts answer anonymously, receive aggregated feedback, and revise over rounds until their answers stabilize. The key requirement is that it be adapted so that only evidence-backed dissent survives between rounds.
Third, the protocol's stability statistics are a better signal for escalating to humans than a single model's confidence or the vote spread of independent samples. The claim holds only if it improves accuracy at a fixed human-review budget, measured at matched token cost.
The editorial-board structure, borrowed from journals where an Editor-in-Chief sets scope and Associate Editors run domain-specific review, plays a narrower role. It is a constraint that stops reviewers from raising concerns that are not tied to the diff or to a clause of the charter. It is not the source of accuracy.
Where This Sits
The problem is well motivated. Understanding Architecture Erosion: The Practitioners' Perceptive traces erosion to "knowledge vaporization," the loss of design rationale when people leave. Software Architecture in Practice: Challenges and Opportunities reports that documentation goes stale almost immediately because nothing links commits back to design. Architecture Without Architects shows that coding agents now make architectural choices implicitly: prompt variants produced systems ranging from 141 to 827 lines of code. The same paper names a "speed-review gap" that human architecture boards cannot close. All three motivate the problem but build no adjudication mechanism.
The closest detection work treats drift as classification. Warnings: Violation Symptoms Indicating Architecture Erosion mines OpenStack and Qt reviews and finds that human reviewers catch erosion: 90% of flagged symptoms were addressed. Towards Automatic Identification of Violation Symptoms of Architecture Erosion automates detection of such comments, with GPT-4o reaching F1 0.851 two-shot. Only 66.7% of surveyed practitioners agreed that the flagged symptoms were real concerns, which is a direct measure of the false-alarm problem. DocChecker and R²ComSync detect and repair mismatches at the method-comment level. All of these answer "is there a mismatch?" We answer "which side should move, and is the machine confident enough to say so?"
Neurosymbolic Architectural Reasoning poses exactly this intended-versus-accidental question as an open research question, alongside low-overhead human validation. It also reports that language models extract overly simplistic formal rules. We take that negative result as a design constraint. Our charter consists of natural-language clauses with identifiers, verified once per project by a human, rather than machine-extracted formal rules.
On the multi-agent side, two systems supply methods and set the bar. CodeAgent arranges review agents in a corporate hierarchy with a supervisor that suppresses conversational drift. It targets vulnerabilities and style rather than design intent, and it has no aggregation of independent judgments. Adversarial Review is the strongest competitor. A reviewer and a critic argue under a constraint that pushback must either cite contradicting code lines or raise a concern the reviewer must answer with evidence. The paper warns that open multi-agent agreement is often "false consensus," and it reports "scope creep" as a failure mode in which speculative concerns trigger out-of-scope changes. We reuse its evidence-citation constraint. We differ by replacing a two-party debate with anonymous multi-reviewer rounds whose statistics feed an explicit escalation rule, and by putting a charter-holding editor in charge of what counts as in scope.
Embodied LLM Agents Learn to Cooperate in Organized Teams shows that designated leadership helps flat LLM teams, which supports having an editor tier. Its evidence comes from household simulation tasks, so it motivates the design rather than validating it here.
Adversarial Review's argument that disagreement is a feature seems to contradict Delphi's pursuit of consensus. We address this head on. The protocol does not force agreement. Persistent dissent that is backed by evidence is itself a reportable outcome, and it triggers escalation. The Delphi literature's own worry about artificial consensus, visible in how clinical panels such as Conservative Management and Biological Treatment Strategies and the behavioral trials priority study handle outlier justification, points the same way.
Research Program
We would build a drift adjudicator in three layers.
Charter. For each project we assemble the design charter from existing architecture documents, specifications, and decision records. It is split into numbered clauses and checked once by a maintainer. This one-time cost is amortized across every later patch, which is how we address Herbold's worry that validation costs as much as writing the architecture by hand.
Panel. For each patch that touches a module boundary, three to five reviewer agents are instantiated. Each owns a module or concern and is drawn from heterogeneous prompts and, where possible, different base models. Each reviewer independently scores the patch on the three-way decision with a 1–9 confidence per option. It also writes a rationale that must cite either diff lines or charter clause identifiers.
Coordinator. Between rounds, a deterministic coordinator runs. It is not a language model, which removes the synthesizer bias the Delphi literature warns about. It performs four steps:
- Compute the median and interquartile range for each option.
- Drop rationales whose citations do not resolve to real lines or clauses.
- Strip reviewer identity and module labels.
- Randomize the order of the remaining rationales before returning them.
Reviewers whose scores fall outside the interquartile range must either revise or restate an evidence-backed justification. Justified dissent is kept as a minority report.
Editor. The editor agent sees only the final-round statistics and rationales. It desk-rejects any concern that cites neither the diff nor a charter clause, which is our scope-creep control. It issues a decision only when the round-to-round change in the interquartile range falls below a threshold and no evidence-backed minority report remains. Otherwise the case goes to a human. When the editor rules for "amend the design," it drafts the charter amendment and an architecture decision record entry. The charter therefore becomes a living document rather than the static output that traditional Delphi studies produce.
Evaluation data. We construct an adjudication benchmark from the OpenStack and Qt review history studied in Warnings. Starting from patches that received architectural-violation comments, we label the outcome from what the project later did:
- Refactor: the patch was revised to remove the violation before merge. This covers the 77.7% of merged-patch cases the paper reports as fixed before merge.
- Amend: the patch was merged together with a change to design documentation or specifications.
- Accept: the patch was merged unchanged, with the concern dismissed.
The charter is always frozen at the pre-patch version, so no model sees post-decision documents. Two annotators verify a stratified subset, and we report Cohen's kappa. The original violation labeling reached 0.857, and we treat anything much lower as a sign that the task is ill-posed.
Baselines
The naive reference points are two rules. One always predicts the majority outcome. The other always flags for refactor, which is effectively what a violation detector implies. Any system that cannot beat always-refactor on macro-F1 across the three outcomes has learned nothing about intent.
The strongest single-model competitor is a frontier model given the same charter clauses, the diff, and the review thread in one prompt. It is prompted few-shot in the style that gave GPT-4o F1 0.851 on violation-comment identification in Towards Automatic Identification of Violation Symptoms. That number is for detection, not three-way adjudication. We will rerun the model on the 606/606 dataset to confirm our prompting reproduces roughly that figure before trusting its adjudication numbers. It is scored for escalation using its verbalized confidence.
The strongest multi-agent competitor is Adversarial Review, reimplemented with its reviewer–critic loop and evidence taxonomy and given the same charter. It reports F1 0.533 on SWE-PRBench review quality, and 75.2% on SWE-bench Verified against 71.6% for zero-shot and 72.6% for the five-agent MARS committee. There is no reported result on Lacuna for any method on a three-way erosion-versus-evolution adjudication task, so on our benchmark every number will be new. We also report on SWE-PRBench so the comparison is anchored to its published figure. CodeAgent is included as a hierarchical-but-not-anonymous reference.
The ablation baseline that matters most is self-consistency voting. We draw the same number of independent reviewer judgments as the panel, with no feedback rounds, and use their vote spread for escalation. Three further ablations each remove one ingredient:
- Show reviewer identities and module roles during feedback, to test whether anonymity matters.
- Replace the deterministic coordinator with a language-model summarizer, to test synthesizer bias.
- Remove the editor's desk-reject step, to test scope control.
Fairness rules apply to every comparison. All systems use the same base models, the same charter text, the same inputs, and prompt-tuning budgets capped at the same number of development-set evaluations. Results are reported at matched total tokens. Adversarial Review is known to cost about 4.5 times a single agent, and our panel will not be cheaper.
Beating the competition means one of two things on the held-out projects. Either higher three-way macro-F1 than both the single model and Adversarial Review, or equal macro-F1 with a strictly better risk-coverage curve. The second means higher accuracy on the auto-decided cases at the same fraction of cases escalated to humans, which is the outcome practitioners care about.
Why not just…?
Why not just give one strong model the design documents and ask? This is the most likely real threat on easy cases, and the single-model baseline exists to measure it. The evidence that it falls short is indirect. Single-agent zero-shot trails structured review on SWE-bench Verified (71.6% versus 75.2%) in Adversarial Review. The 66.7% practitioner agreement in Towards Automatic Identification shows that one model's confident flags are often not real concerns. The decisive comparison is the risk-coverage curve. If verbalized single-model confidence triages as well as panel stability, the panel is not worth its cost, and we will say so.
Why not just sample several independent judgments and vote? This is the alternative most likely to make the Delphi rounds redundant. Round one of our protocol is exactly this. Iteration adds value only if feedback helps correct reviewers more often than it drags them toward wrong majorities. We measure this directly with a flip analysis: among cases where the round-one minority was right, what fraction stays right after feedback, and what fraction flips to wrong? That is a direct measure of false consensus. If voting matches the full protocol at equal tokens, the contribution shrinks to the charter-scoped editor and the three-way task, and we would reframe the work accordingly.
Why not just use Adversarial Review with the charter added? It already enforces evidence-cited disagreement. It is a two-party debate, however, and its authors report scope creep. Its output also has no calibrated notion of when to stop and ask a human. We run it with the charter as a baseline. If it matches our macro-F1, the remaining question is whether our desk-reject rule reduces out-of-scope concerns. We measure that as the fraction of raised concerns citing neither diff nor charter, and as the rate of wrong refactor rulings on cases whose ground truth is accept.
Why not just encode the architecture as rules and run a static conformance checker? Deterministic checkers are sound when rules exist. They cannot represent the intended-evolution outcome, because by construction the rule is what changed. Writing the rules by hand is the cost Neurosymbolic Architectural Reasoning calls prohibitive. Where a project already has dependency rules, we include checker output as an input the reviewers may cite. We then report how many amend cases the checker alone flags as violations. Those are exactly the false alarms we aim to remove.
Proposed Experiments
The primary experiment runs all systems on the OpenStack/Qt adjudication benchmark, with training and prompt development on some projects and testing on held-out projects. We report three-way macro-F1, the risk-coverage curve, the scope-creep rate, and tokens per decision.
Two secondary checks tie the work to published numbers. We compare against the 0.533 F1 of Adversarial Review on SWE-PRBench. We also run a method-level analogue on the Corazza comment-coherence dataset from A New Dataset for Source Code Comment Coherence, where the task becomes deciding whether the code or the comment should change. That dataset reports no model result, while DocChecker reports 72.3% ICCD accuracy on a detection task, as a reference for the detection component.
A small agent-generated stress test adapts the prompt-variant setup of Architecture Without Architects. It checks whether the editor correctly routes large prompt-induced structural changes to amend-or-escalate rather than silently accepting them.
The decisive checks are three.
- Stability versus vote spread. At matched tokens, panel stability must beat self-consistency vote spread on the risk-coverage curve. This is the test of whether Delphi iteration is real.
- Minority retention. Correct round-one minorities must survive feedback at a higher rate in the anonymous condition than in the identity-visible condition. This is the test of whether anonymity counters false consensus in language-model panels as it does in human ones.
- Scope control. The desk-reject rule must lower the scope-creep rate without lowering refactor recall.
If the first check fails, the Delphi component is unjustified and the proposal collapses to a charter-scoped single reviewer. The benchmark and the three-way framing would remain useful contributions.
References
- Understanding Architecture Erosion: The Practitioners' Perceptive
- Software Architecture in Practice: Challenges and Opportunities
- Architecture Without Architects: How AI Coding Agents Shape Software Architecture
- Warnings: Violation symptoms indicating architecture erosion
- Towards Automatic Identification of Violation Symptoms of Architecture Erosion
- DocChecker: Bootstrapping Code Large Language Model for Detecting and Resolving Code-Comment Inconsistencies
- R$$^2$$ComSync: improving code-comment synchronization with in-context learning and reranking
- Neurosymbolic Architectural Reasoning: Towards Formal Analysis through Neural Software Architecture Inference
- CodeAgent: Autonomous Communicative Agents for Code Review
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- Conservative Management and Biological Treatment Strategies: Proceedings of the International Consensus Meeting on Cartilage Repair of the Ankle
- An international, Delphi consensus study to identify priorities for methodological research in behavioral trials in health research
- A New Dataset for Source Code Comment Coherence