Debating Over Contradictions: Reasoner-Grounded Multi-Agent Repair of Ontology Alignments

When two independently built ontologies are merged, the matcher that links their concepts can produce a set of links that each look right and together are logically impossible. An ontology is a formal vocabulary with "is-a" hierarchies and "these two classes share no members" (disjointness) rules. A typical failure is linking Melanoma under Disease in one ontology to Melanoma under Neoplasm in another, where the two parents are declared disjoint. A logic reasoner can prove that such a set of links is contradictory and can name the smallest group of links responsible. It cannot say which link in that group is wrong. Current systems settle this with a confidence heuristic. Language models have so far been used only as single yes/no judges on individual uncertain links, and those judges get worse, not better, when shown the surrounding hierarchy. We propose to make the reasoner's proof of contradiction the thing language-model agents argue about. Each link in a conflicting group is assigned an advocate, the reasoner guarantees that not all advocates can win, and a judge decides which link to give up.

The idea fits in two sentences. Use a description-logic reasoner to find minimal sets of mappings that cannot all be true. Then run a structured debate in which dissent is assigned by the logic rather than by vote count, and give agents ontology context step by step on request instead of all at once. If it works, three problems are handled together. Coherence repair becomes semantic instead of heuristic. The "majority trap" of multi-agent debate is blocked by a formal guarantee. The finding that hierarchical context confuses LLMs is addressed by changing how much context is disclosed, and when.

Core Claim

A conflict set is the right unit for LLM adjudication in ontology alignment, and debate helps on it specifically because the reasoner certifies that the group's default answer ("all links are fine") is false. Formally, given source ontologies $O_1, O_2$ and a candidate alignment $M$, a conflict set is a subset $C \subseteq M$ such that

$$O_1 \cup O_2 \cup C \models A \sqsubseteq \bot \text{ for some class } A, \quad \text{and no proper subset of } C \text{ has this property,}$$

where $A \sqsubseteq \bot$ means class $A$ can have no members (it is unsatisfiable). At least one element of $C$ must be removed, or one source axiom must be blamed. We claim three things. First, asking LLMs "which element of this certified conflict is wrong" gives better repairs than LogMap-style confidence-based removal. It also gives better repairs than asking an LLM oracle about each link separately. Second, structured advocacy over the conflict outperforms a single judge and compute-matched self-consistency on conflicts whose links are lexically strong. These are the cases where one-link-at-a-time judgments tend to accept everything. Third, because adjudication cleans up contradictions, the system can afford a recall-oriented candidate generator. That directly targets the conservatism of frontier models, whose errors are mostly omissions.

Positioning Against Prior Work

The work that motivates the gap comes from the alignment side. Results of the Ontology Alignment Evaluation Initiative 2009 reported that high pairwise precision does not imply a coherent merged structure. Exploring Large Language Models for Ontology Alignment evaluated zero-shot Flan-T5 and GPT-3.5 as pairwise matchers on hard Bio-ML subsets and found that adding parents and children to the prompt hurt performance. OAEI-LLM-T built a taxonomy of LLM matching errors and found that over 57% of GPT-4o and Claude-3-Sonnet errors are missing mappings, while incorrect ones cluster in Align-Down and Disputed categories. Neither study connects LLM judgments to a reasoner. We do, and we treat the context finding as a question about how context is disclosed rather than as a fixed limit.

The closest method is Large Language Models as Oracles for Ontology Alignment (LogMapLLM). It sends LogMap's most uncertain 5–10% of candidates to a single LLM for independent true/false decisions. LogMap's repair then runs downstream, as described in LogMap family participation in the OAEI 2016. We reverse that order. The reasoner runs first to build conflicts, and the LLMs judge links jointly within a conflict, with the logical relationship among those links made explicit. BERTMap pairs a fine-tuned encoder with deterministic symbolic repair. It supplies one of our candidate generators and a repair baseline, but its repair step uses no semantic judgment. A Probabilistic-Logical Framework for Ontology Matching resolves conflicts by joint inference with similarity as soft evidence. We keep its hard-constraint view, but the soft evidence comes from argued LLM judgments rather than similarity scores.

On the debate side, MAD-Logic has agents debate translations into logic programs, first-order logic and SAT, checked by external solvers. It reports gains on deductive puzzles and finds degradation beyond three rounds. We borrow solver grounding and its round budget, but the solver plays a different role. In our setting it does not check an agent's translation. It produces the debate topic and certifies that disagreement is warranted. Can LLM Agents Really Debate? constrains our design. It shows that majorities are entrenched (97%+ stay put), that minorities cave, and that gains are bounded by the strongest agent. We respond to the first two by assigning dissent positions through logic rather than letting them emerge. We treat the third as a real ceiling that the experiments must measure. Examining Inter-Consistency of Large Language Models Collaboration supplies the alternating refute-and-defend format we reuse.

Interaction-based ontology alignment repair with expansion and relaxation is the only prior multi-agent repair work. Its rule-based agents trade classification queries over synthetic ontologies and use "relaxation", meaning deliberately broader answers, to expose wrong mappings hidden behind correct, more specific ones. We carry relaxation into the LLM setting as a second source of conflicts, described below. Our agents argue in natural language over real biomedical ontologies.

The abstraction–flexibility literature supplies a design pattern rather than an object of study. TensorLayer and the declarative dialogue authoring of the Task Completion Platform both keep a simple high-level interface while letting users reach into lower layers on demand. We apply the same principle to what an agent sees. By default it gets a plain-language rendering of the reasoner's justification. It can then request lower layers, such as the parents of a class, its textual definition and synonyms, or the raw axioms, through tool calls. The scientific question is whether disclosure on demand avoids the degradation that a full context dump causes.

Research Program

The pipeline has four stages.

The first stage is a deliberately permissive candidate generator. It combines LogMap candidates, BERTMap scores via KRR-Oxford/BERTMap, and an LLM proposer prompted to favour recall. The point is to counter the omission bias documented in OAEI-LLM-T.

The second stage is conflict extraction. The candidate alignment is loaded together with both ontologies, and a reasoner enumerates minimal conflict sets. For the largest ontologies, full reasoning is intractable, so we use the Horn fragment LogMap already relies on. We report how many conflicts the incomplete fragment misses on the mid-size tracks, where full reasoning is feasible.

To catch mappings that are wrong but cause no contradiction (the shadowing problem), we add relaxation probes. For each accepted mapping we compute new subsumptions it forces inside either source ontology, that is, new is-a relations among that ontology's own classes. We treat these as soft conflicts. Each is put to debate as "this mapping implies X is-a Y within ontology 1; is that acceptable?"

The third stage is adjudication. For a conflict of size $k$, we instantiate $k$ advocates, each obliged to defend one mapping. There is one skeptic whose job is to argue that a source-ontology axiom, rather than any mapping, is at fault, and one judge. Advocates start from the rendered justification and request lower-layer context only through tools. We log every request, which shows what context agents actually find useful.

Debate runs for at most three rounds, following MAD-Logic. Positions are fixed by assignment, not by vote, so the majority-trap dynamics cannot form around "keep everything". The judge outputs a removal decision and a confidence. When conflicts overlap, the final decisions are combined in a small weighted hitting-set step, so that the final alignment is coherent by construction. The weights are the judge confidences.

The fourth stage is analysis. We stratify every result by OAEI-LLM-T error type and by conflict size. We also audit by hand a sample of "errors" against the reference, because Disputed mappings mean the reference itself is incomplete.

Baselines

The do-nothing reference is the permissive candidate alignment with no repair. It shows how much incoherence aggressive generation creates and what recall it buys. We report it at the same candidate pool used by every other system.

The classical repair baseline is LogMap's own logic-based repair applied to the same pool. It removes the lowest-confidence mapping from each conflict. It is the strongest non-LLM competitor because it already guarantees coherence cheaply. Our method must beat it on F-score at equal or better coherence.

The strongest existing method is LogMapLLM from Large Language Models as Oracles for Ontology Alignment. It reports raising LogMap's average Bio-ML F-score from 0.718 to 0.751 and an average Youden's Index of 0.550 for Gemini 2.5 Flash versus about 0.094 for LogMap's internal heuristic. We rerun it with the same LLM backbone as our agents and the same candidate pool, followed by LogMap repair. We give it a token budget equal to our debate's by letting it query additional uncertain mappings. Beating it means a higher Bio-ML F-score than its reproduced value on the same tasks, with zero unsatisfiable classes. Coherence statistics for LogMapLLM are not in the file (no reported result found on Lacuna for unsatisfiable-class counts), so we measure them ourselves.

For context on the hard subsets, we cite the figures from Exploring Large Language Models for Ontology Alignment: Flan-T5-XXL reaches an F-score of 0.721 on NCIT-DOID against BERTMap's 0.628, and BERTMap's 0.552 on SNOMED-FMA (Body) against Flan-T5's best of 0.346. On Anatomy, the bar from Results of the Ontology Alignment Evaluation Initiative 2023 is Matcha's F-measure of 0.941.

Four ablations isolate the ingredients. The single-judge ablation gives one LLM the same conflict set, the same justification and the same tools, and asks which mapping to drop. It isolates the value of debate from the value of conflict framing. The compute-matched self-consistency ablation samples that single judge repeatedly and takes a majority, at a token count equal to the debate's. The full-context ablation gives all agents the complete lineage, definitions and axioms up front. It isolates progressive disclosure. The free-debate ablation lets agents choose positions rather than having them assigned. It tests whether logic-assigned dissent actually prevents conformity.

Why not just…?

Why not just use LogMapLLM and let LogMap repair clean up afterwards? This is the most direct competitor, and it may be good enough when conflicts are rare. Its LLM never sees why two mappings clash. It judges each in isolation, so two individually plausible but jointly impossible links are both accepted, and the heuristic then decides between them. The Youden gap it reports (0.550 vs ~0.094) shows the heuristic is weak at exactly this kind of decision. The strongest-baseline comparison above settles the question. If our gain is confined to tasks with very few conflicts, the honest conclusion is that conflict-grounding matters only under permissive generation, and we will report that.

Why not just ask one strong LLM to pick the culprit from the conflict set? This is the real threat to the debate component. Can LLM Agents Really Debate? shows that debate gains are capped by the strongest agent, so a frontier single judge may match a debating team. The single-judge and self-consistency ablations test this directly at matched tokens. If they match the debate, the contribution reduces to conflict-grounded adjudication with progressive disclosure, which we would still regard as a result, and we would drop debate from the recommended system. We expect debate to help mainly where every mapping in the conflict is lexically strong, because a lone judge tends to keep the most familiar label there. The stratified analysis will show whether that is so.

Why not just give a long-context model the whole hierarchy? Context windows are now large enough to make this possible. The evidence from Exploring Large Language Models for Ontology Alignment, and LogMapLLM's observation that synonyms help more than lineage, suggest that more structure in the prompt hurts. Those results come from older or smaller models, though. The full-context ablation with the same frontier backbone answers this, and the tool-request logs show which layers agents actually pull.

Why not just use joint probabilistic-logical inference over similarity scores? The approach of A Probabilistic-Logical Framework for Ontology Matching resolves conflicts optimally given its scores, but those scores are lexical and structural. Our final hitting-set step is essentially that inference with LLM judgments as weights. We will run the same step with similarity weights, which separates the contribution of better evidence from the contribution of the optimizer.

Proposed Experiments

We evaluate on the OAEI Bio-ML track (NCIT-DOID and SNOMED-FMA, including the hard subsets without exact string matches), Largebio (FMA-NCI, FMA-SNOMED, SNOMED-NCI) for scale, Anatomy, and Conference. All evaluation goes through MELT so that scoring matches OAEI practice. OAEI-LLM-T labels provide the error-type stratification.

Metrics are precision, recall and F-score against the reference. We also report the number of unsatisfiable classes after repair, the number of mappings removed per conflict, the accuracy of conflict-culprit identification on conflicts where the reference marks exactly one mapping as wrong, Youden's Index on the uncertain pool for comparability with LogMapLLM, and total tokens and dollars per task. Running two LLM backbones, one frontier and one open-weight 8B-class model, tests whether assigned dissent helps more when agents are weaker, as the conformity findings predict.

Three checks are decisive. The first is whether conflict-grounded adjudication of any kind beats LogMapLLM on Bio-ML F-score at zero unsatisfiable classes. If it does not, the core claim fails. The second is whether structured advocacy beats the compute-matched single judge on culprit-identification accuracy, particularly on conflicts where all mappings are lexically strong. If it does not, debate is not earning its cost. The third is whether the permissive generator plus adjudication gains recall over a conservative generator without losing precision. This would show that coherence-grounded repair lets the system recover the omissions that dominate frontier-model errors.

A preliminary audit comes first: counting conflicts per task under permissive generation. If the Bio-ML tasks yield only a handful, the effect cannot show at the F-score level. In that case we will report culprit accuracy on the pooled conflicts as the primary measure, and say so up front.