Ruling on Conventions, Not Tokens: Editorial Escalation for Merging Divergent Linguistic Corpora

When two annotated corpora of the same language disagree, the disagreement is usually not an error. More often, the two projects followed different conventions. One Latin treebank files participles under their verb, while another gives them their own adjective lemma. One glossing tradition writes PST, another PRET, and a third splits the morpheme differently. Today, systems that merge such corpora do one of two things. They force everything into one label set and discard what does not fit, as GlossLM did by mapping only the 200 most common labels to UniMorph. Or they store every analysis side by side and leave reconciliation to the user, as the LiLa and Ligt linked-data models do. Neither approach answers the practical question: which disagreements reflect a convention, which reflect genuine ambiguity, and which are simply mistakes?

We propose to borrow one specific idea from how journal editorial boards work. Board members do not reread every manuscript. Reviewers handle individual cases, associate editors spot recurring patterns, and the editor-in-chief rules on policy. The core claim is that, when corpora are merged, scarce human expertise should be spent on convention rulings rather than on individual tokens. A ruling is a short, testable rule such as "in corpus A, participles with comparative morphology are lemmatized as adjectives." The system proposes these rulings from clusters of disagreement, a human expert approves or rejects them, and approved rulings are applied automatically to every matching token, with a record of who decided and under which tradition. We expect that, for the same expert budget, this resolves far more of the divergence at equal accuracy than routing individual hard tokens to humans. That token-by-token routing is what current multi-agent annotation managers do. We also expect rulings to leave genuine errors visible rather than absorbing them into a rule.

What exists and what is missing

The fragmentation literature supplies the representations but not the decision process. A Common Framework for Syntactic Annotation introduced pivot formats and stand-off layers, in which annotations live in separate files pointing back to the shared base text, so that conflicting analyses can coexist. It also notes that mapping site-specific categories into a shared registry requires heavy manual curation. That cost is what we aim to reduce. LiLa (Harmonizing Different Lemmatization Strategies for Building a Knowledge Base of Linguistic Resources for Latin) links Latin corpora through a central bank of lemmas. On the PROIEL treebank, it matched 84.64% of tokens by string matching plus part-of-speech disambiguation, leaving 10,555 tokens unresolved. Its participle study (Harmonizing Divergent Lemmatization and Part-of-Speech Tagging Practices for Latin Participles through the LiLa Knowledge Base) found 2,202 word types, covering 41,173 tokens, whose part-of-speech assignment varies across corpora. The average entropy of these assignments is 0.76. To handle this, it introduced "hypolemmas": citation forms inside a word's regular paradigm that carry a different part of speech from the main lemma. Addressing Variability in Interlinear Glossed Texts with Linguistic Linked Data models label variants, differences in granularity, and parallel analyses in interlinear glossed text, meaning texts annotated morpheme by morpheme with grammatical labels. Its authors list missing provenance as an open limitation. All three papers motivate this proposal: they show that the divergence is real and structured, and they provide the storage format our rulings will write into. None of them decides anything.

On the governance side, CrowdAgent: Multi-Agent Managed Multi-Source Annotation System is the closest existing system. It models a crowdsourcing company as LLM agents for quality assurance, financing, and scheduling. It sends easy items to cheap models and escalates low-confidence items to humans, and it reports cost reductions of up to 82% on classification benchmarks. Its unit of escalation is the individual instance, and it assumes one shared label space. It was not tested on structured linguistic annotation, and its authors note difficulty with the hardest 1–2% of items. Our proposal differs on the unit of decision: escalation happens at the level of a rule that covers many tokens, and the label spaces are allowed to differ. Embodied LLM Agents Learn to Cooperate in Organized Teams shows that a designated leader reduces coordination failures among LLM agents. This supports giving the system a clear hierarchy, though its evidence comes from household simulation, not annotation. Principal-agent theory, as applied in Dilemmas and Governance Strategies in the Implementation of the Modern Apprenticeship System in Chinese Vocational Colleges, offers a lens rather than a method. Each source corpus is an agent whose working conventions are only partly documented, and the merging project is a principal that faces this information gap. Rulings are a way to make the hidden conventions explicit.

Core Claim

Divergence between corpora is concentrated in a small number of systematic conventions. If those conventions can be proposed automatically as candidate rules and adjudicated by an expert once each, then a fixed expert budget spent on rulings will resolve more divergent tokens than the same budget spent on individual tokens, at no loss of precision. The tokens that remain unexplained after rulings are applied should then be enriched for real annotation errors and genuine gray-zone cases. This last property is what makes the residue worth a human's attention.

Research Program

The system has three tiers.

At the first tier, "reviewers" are taggers, lemmatizers, and string-matching passes that align tokens across sources. This is roughly the LiLa linking step. Their output is a set of disagreements.

At the second tier, an "associate editor" works for each pair of sources. It clusters disagreements by features such as source, lemma class, morphological features, and syntactic relation. It then proposes candidate rulings, each consisting of a condition, a mapping, and a scope estimate. We will compare two ways of generating candidates: an LLM that reads disagreement clusters together with each source's annotation guidelines, and a purely statistical inducer that searches for feature conjunctions with low conditional entropy over the label mapping. Keeping the induction method as a variable keeps the claim about escalation level separate from any claim about LLMs.

At the third tier, the "editor-in-chief" is a human expert who accepts, rejects, or refines candidate rulings in order of expected coverage. Accepted rulings fall into one of three categories. A convention ruling becomes a deterministic mapping. A genuine-ambiguity ruling becomes a hypolemma or parallel analysis in LiLa or Ligt. A rejected ruling sends its tokens to a residue queue for token-level review. Every ruling is stored with provenance: the expert, the theoretical tradition invoked, and the date. This directly supplies the provenance layer that the Ligt paper says is missing.

One safeguard is central to the design. Before an expert sees a ruling, it is applied to a held-out slice of its scope, and any exceptions are shown alongside it. This makes it hard for a rule to silently absorb real errors.

Baselines

The naive reference point is the coverage LiLa already achieves on PROIEL with string matching plus part-of-speech disambiguation, as reported in Harmonizing Different Lemmatization Strategies for Building a Knowledge Base of Linguistic Resources for Latin. That is 84.64% of tokens covered, with 10,555 left unresolved and no human input. For glossed text, the matching reference is GlossLM's static mapping of the top 200 labels to UniMorph, which covers about 80% of the data (GlossLM). Beating these baselines means resolving the remainder at audited precision, not merely raising coverage.

The strongest competitor is instance-level escalation in the style of CrowdAgent. The same LLM annotators and the same confidence estimates are used, but the expert budget is spent on the lowest-confidence individual tokens, and the aggregated labels inform the rest. It is the strongest competitor because it is the published approach to managing mixed human and model annotation under a budget. We will reimplement its routing logic and give it the same models, the same budget of expert decisions (200 and 500), and the same tuning time. CrowdAgent's reported numbers are classification accuracies (for example, 88.45% on V-SNLI at 75–82% cost reduction) and do not transfer to this task. We found no reported result on Lacuna for instance-level routing on cross-corpus lemma or gloss harmonization. Our system beats it if, at matched expert budget, it resolves more divergent tokens at equal or higher audited precision, and does not lose on error-detection recall.

The second competitor is a statistical source-confusion model. It learns, for each source, a matrix of how that source's labels relate to a reference label, optionally conditioned on lemma class. It uses no LLM and no expert. It is the cheapest serious threat, because the 5:1 VERB/ADJ ratio reported for participles is exactly the kind of regularity such a model captures.

The key ablation keeps the three tiers but removes expert adjudication, applying the top-k candidate rulings automatically. This isolates whether human rulings are the ingredient that matters. A second ablation swaps the LLM rule proposer for the statistical one.

Why not just…?

Why not just give one LLM both corpora's guidelines and ask it to map each token? The guidelines are often incomplete. The undocumented rules embedded in legacy corpora are the reason mapping is expensive in the first place (A Common Framework for Syntactic Annotation). A token-by-token mapper also produces no reusable, auditable decision. We include it as a condition in the instance-level baseline at zero expert budget. If it already reaches high audited precision on the participle set, then the expert tier is unnecessary for Latin, and we would report that.

Why not just fit per-source confusion matrices and be done? This is a real threat for conventions that depend only on the tag. It cannot express conditional conventions, such as those that depend on degree, syntactic function, or lemma class. It also cannot separate genuine ambiguity from convention. The 0.76 entropy across 2,202 participle types suggests that much of the variation is conditional or genuinely gray. The confusion-model baseline and the statistical rule-proposer ablation settle the question directly. If the conditional statistical model matches our system without an expert, then the contribution reduces to the escalation level alone, and we would say so.

Why not just force everything into UniMorph or Universal Dependencies and drop what does not map? GlossLM's mapping leaves about 20% of labels stranded. The participle study shows that a binary VERB/ADJ choice misrepresents a category with real internal variation. Our downstream experiment tests whether recovering the stranded labels through rulings actually helps a model, rather than assuming that it does.

Proposed Experiments

The first testbed is Latin. It uses LiLa-linked corpora: PROIEL, the Index Thomisticus Treebank, UDante, LASLA's Opera Latina, and CLaSSES. It focuses on the 41,173 inconsistently tagged participle tokens and the 10,555 unresolved PROIEL tokens. Two Latinists will label a stratified audit sample of about 1,500 divergent tokens as convention, genuine ambiguity, or error, and we will report their agreement. To measure error detection with a known ground truth, we will also inject synthetic lemma and tag corruptions into tokens that all sources agree on.

The second testbed is interlinear glossed text from languages present in more than one GlossLM source, such as overlaps between ODIN and IMTVault. There, rulings target the unmapped 20% of labels.

The metrics are as follows: tokens correctly resolved per expert decision; audited precision of resolved tokens; recall of injected errors in the residue queue; and the share of divergence encoded losslessly as hypolemmas or parallel analyses.

The decisive checks are three. First, at 200 and 500 expert decisions, ruling-level escalation must dominate instance-level escalation on the coverage–precision curve. Second, recall of injected errors must not fall below that of instance-level routing, which shows that rulings are not absorbing mistakes. Third, a downstream test. We will retrain a ByT5 glossing model under GlossLM's protocol with and without ruling-recovered labels, and evaluate on the SIGMORPHON 2023 test languages. The reference points are GlossLM's reported results: up to 82.1% morpheme accuracy on Arapaho, a 15.1-point gain on Lezgi from multilingual pretraining, and better results than prior systems on 5 of 7 languages (GlossLM).

A null downstream result alongside a positive harmonization result is still informative. It would mean that rulings mainly improve resources for human use rather than model training. We would report it that way.