Dialogue summarizers often write fluent summaries that credit statements, decisions, or feelings to the wrong person. These are speaker attribution errors. Counting them reliably takes careful human annotation, and the published fixes were each tested against a different baseline, with a different error taxonomy and a different annotation protocol. Nobody knows how contrastive fine-tuning, speaker embeddings, post-correction, and plain LLM prompting rank against each other on this one failure. Clinical medicine had the same problem with drugs that were never compared head-to-head. It developed Bayesian network meta-analysis to handle it: a statistical model that treats each trial as an edge between treatments, borrows strength across shared comparators such as placebo, and reports uncertainty about the full ranking. It also includes explicit tests for when indirect comparisons disagree with direct ones.
This note proposes treating dialogue summarizers as treatments and individual dialogues as trials. Human annotators label speaker errors for only a few systems per dialogue, and a Bayesian network meta-analysis reconstructs the full comparison. The claim to test is specific. Under a fixed annotation budget, an incomplete design analyzed this way should recover the true ranking of speaker-attribution error rates more accurately than a fully crossed design on fewer dialogues. It should also give calibrated intervals and flag which comparisons are unsafe to transfer across domains such as short chats and long meetings. We validate against a separately annotated, fully crossed reference set, so the claim can fail cleanly.
Why this gap is real
The dialogue literature supplies both the motivation and the constraints. Each fix reports gains against its own reference point. CONFIT trains BART and T5 with a contrastive loss on hard negatives and a speaker-identity auxiliary loss. It reports that "Wrong Reference" errors fall from 37% to 17% for BART, but it also notes a gap between human and automatic judgments on AMI meetings. Speaker-embedding injection into PEGASUS adds speaker vectors to the quieter, low-frequency part of the position encoding and reports ROUGE-1 of 51.39 versus 50.82 on SAMSum. ROUGE is a word-overlap score, and word overlap is known not to track speaker faithfulness. Speaker-focused post-correction classifies and rewrites speaker errors in a draft and reports ROUGE-2 gains and human preference. Coref-BART, as analyzed by Wang et al., shows error rates up to 50%, higher than plain BART. The same paper finds that 17% of SAMSum gold references are themselves wrong. Across these papers the base models differ, the outcomes differ (error rates, ROUGE, preference), and the error taxonomies differ. No two of the fixes share a directly measured speaker-error comparison.
Automatic scoring cannot replace annotation yet, which constrains the design. On DIASUMFACT, fine-grained error detectors reach a macro-F1 near 0.25. On LLM summaries, NLI and QA metrics sit at 40–50% balanced accuracy, with a ChatGPT mixture-of-experts detector at 70.27%. Human labels remain the ground truth, and they are expensive, especially on long meetings. QMSum reports that 74% of generated meeting summaries contain inconsistencies such as misattribution.
The clinical papers supply the method, not a finding. The glioblastoma network meta-analysis describes how the Bayesian formulation handles multi-arm trials and closed loops without solving systems of linear equations. The HFpEF synthesis of 64 trials and the male osteoporosis synthesis show MCMC estimation in GeMTC and SUCRA rankings. SUCRA (surface under the cumulative ranking curve) is a 0–100% summary of how likely a treatment is to rank near the top. Both of those syntheses also show a known weakness: rankings flip across outcomes, such as NYHA class versus 6-minute walk distance, or spine versus hip bone density. We expect the same flip between chat and meeting domains and design for it.
We deliberately do not pursue the research file's more literal pairing, which applies transitivity algebra to infer who spoke from shared interlocutors. Speaker identity is categorical and is not a difference of effect sizes. Local context models such as the quote-attribution QA model in Who Said What already reach 92.9% on PDNC. The analogy between the two directions is sound at the level of evidence about systems, not at the level of individual tokens.
Core Claim
We model the number of speaker-attribution errors that annotators find in system $k$'s summary of dialogue $d$ as binomial over the summary's $n_{dk}$ sentences, with probability $p_{dk}$:
$$\mathrm{logit}\, p_{dk} = \mu_d + \delta_k + \beta_{k}\, x_d + u_{dk}, \qquad u_{dk} \sim \mathcal{N}(0, \tau^2).$$
Here $\mu_d$ is the dialogue's baseline difficulty, a nuisance term exactly like a trial's control-arm rate. $\delta_k$ is system $k$'s effect relative to a reference system (plain BART). $x_d$ is a dialogue-level covariate, meeting versus chat, which is the main suspected transitivity violation. $\tau$ is between-dialogue heterogeneity. Each dialogue is a multi-arm trial containing only the few systems annotated on it.
We claim three things. First, at equal annotation cost, the posterior ranking from an incomplete design (many dialogues, three systems each) recovers the reference ranking better than a fully crossed design on fewer dialogues. Second, its 90% intervals on pairwise differences cover the reference differences close to 90% of the time. Third, node-splitting will flag pairs whose direct and indirect estimates disagree, and those flags will predict where rankings fail to transfer from SAMSum to meetings. Node-splitting is the standard meta-analytic check that estimates a comparison once from its direct edge and once through the rest of the network.
Research Program
We assemble eight systems as treatments: BART-large, BART with CONFIT, PEGASUS, PEGASUS with fourth-quarter speaker embeddings, BART followed by speaker post-correction, Coref-BART, one frontier LLM prompted zero-shot, and one open instruction-tuned LLM. Where a paper's code is not available we reimplement from its description and report that fact. We generate summaries on the SAMSum test set and on AMI/ICSI meetings from QMSum.
Annotators label every summary sentence with an error category. The primary outcome is speaker attribution, which covers wrong reference and subject-object swaps. Other factual errors are recorded as a secondary outcome, using a taxonomy aligned with CONFIT's and with DIAC-FactQA's SubObjE category from the dialogue comprehension study. Annotators judge against the dialogue, not the gold reference, given the known 17% reference noise.
Annotation has two layers. The reference layer is fully crossed: 250 SAMSum dialogues and 60 meetings, each annotated for all eight systems by two annotators. It is never used for fitting. The experimental layer uses the same budget of roughly 1,200 summary annotations per domain in several ways. One is a balanced incomplete block design with three systems per dialogue. Another is an unbalanced "star" design in which most dialogues pair a new system with BART only, which mirrors how papers actually evaluate. We fit the model with MCMC and report posterior SUCRA, pairwise odds ratios, $\tau$, and node-split inconsistency p-values.
A second, smaller study turns the lens on the published literature. We build a network from the reported speaker-error comparisons, such as CONFIT's 37%→17% for BART and 46%→13% for T5 and Coref-BART against BART. We then ask whether its indirect conclusions match our controlled reference measurements. A mismatch would give a quantitative estimate of how far annotation-protocol heterogeneity biases cross-paper comparisons in this subfield.
Baselines
The naive reference point is what the field does now. Each system is annotated on its own random subset of dialogues, and raw error rates are compared, with no pairing and no model. At the same budget, this sets the floor for ranking accuracy and interval coverage.
The strongest human-annotation competitor is a fully crossed design on fewer dialogues, about 150 per domain, with all eight systems. It is analyzed by per-pair paired bootstrap on sentence-level error rates. This is the design a careful evaluator would choose by default. It removes dialogue difficulty by construction, so an incomplete design has to beat it on ranking accuracy against the reference layer, measured by Kendall's tau and top-1 identification. It also has to beat it on the width of pairwise intervals while matching coverage. Both designs use the same annotators, the same guidelines, and the same total annotation count.
The strongest zero-human-cost competitor is automatic detection. We run a ChatGPT-based mixture-of-experts detector like the one in the circumstantial-hallucination study, which reports 70.27% balanced accuracy on LLM summaries. We also run FacEval-style likelihood probing from Wang et al. on all summaries in the reference set and rank systems by detected speaker errors. No reported result was found on Lacuna for either detector's accuracy at ranking systems by speaker errors specifically. Beating it means a higher rank correlation with the reference layer.
Two ablations isolate the proposal's ingredients. The first is a frequentist mixed-effects logistic model with dialogue random intercepts, fit to the same incomplete data. It is the same likelihood without priors, so it isolates what the Bayesian treatment adds: small-sample calibration and full ranking posteriors. The second is our model with $\beta_k$ and node-splitting removed, which isolates whether modeling transitivity violations matters for cross-domain transfer.
Why not just…?
Why not just annotate everything fully crossed? On a fixed budget, fully crossed means fewer dialogues. Speaker errors are rare per sentence and vary strongly by dialogue, so the resulting estimates may be noisy. That is the empirical question the main comparison answers. If the fully crossed design matches or beats the incomplete design on ranking accuracy and coverage, the design claim fails. The literature-level and inconsistency findings would still stand, and we would report the result as such.
Why not just use an LLM judge or an automatic detector? Reported detector accuracy is low for fine-grained dialogue errors: roughly 0.25 macro-F1 on DIASUMFACT and at best around 70% balanced accuracy for the strongest LLM-based detector. Errors that correlate with particular systems could still reorder rankings even when average accuracy looks acceptable. The detector baseline tests this directly. Automatic labels could also enter the model as a cheap, noisy outcome, and we will report whether combining them helps.
Isn't this just a mixed-effects model with a new name? Largely yes. The likelihood is a generalized linear mixed model, and the frequentist ablation is designed to show how much, if anything, the Bayesian layer adds. What the meta-analysis tradition contributes beyond the likelihood is its design and diagnostics: multi-arm incomplete trials, SUCRA-style ranking uncertainty, and node-split inconsistency tests. If the ablation performs identically on every metric, we will say that the value lies in those diagnostics and not in the Bayesian estimation.
Decisive checks
The project succeeds if three things hold at equal annotation budget. The incomplete design analyzed with the network model achieves higher Kendall's tau against the reference layer than the fully crossed baseline. Its 90% pairwise intervals cover the reference differences between 85% and 95% of the time. Pairs flagged by node-splitting or by nonzero $\beta_k$ are disproportionately the ones whose ranking flips between SAMSum and meetings. The literature-level network adds a separate check: if its indirect ranking disagrees with the controlled reference beyond posterior uncertainty, that is direct evidence that cross-paper comparisons of speaker-attribution fixes are unreliable as published. That would be a useful negative result for the subfield.
References
- CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning
- Evaluating the Effects of Embedding with Speaker Identity Information in Dialogue Summarization
- Capturing Speaker Incorrectness: Speaker-Focused Post-Correction for Abstractive Dialogue Summarization
- Analyzing and Evaluating Faithfulness in Dialogue Summarization
- Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization
- Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends
- QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization
- Comparative efficacy of antiangiogenic treatment for newly diagnosed glioblastoma
- Efficacy of Chinese traditional patent medicines for heart failure with preserved ejection fraction: a Bayesian network meta-analysis of 64 randomized controlled trials
- Efficacy and safety of abaloparatide, denosumab, teriparatide, oral bisphosphonates, and intravenous bisphosphonates in the treatment of male osteoporosis: a systematic review and Bayesian network meta-analysis
- Who Said What: Formalization and Benchmarks for the Task of Quote Attribution
- Exploring the Factual Consistency in Dialogue Comprehension of Large Language Models