Counting Factual Islands: Topological Fragmentation Scores for Question-Answering Summary Evaluation

A good summary does more than contain the right facts. It keeps them connected: the cause stays attached to its effect, and the person stays attached to the decision they made. Question-answering (QA) based evaluation checks whether a reader can recover a source document's key facts from a summary alone. Current QA metrics score each question independently, so a summary that keeps ten facts but cuts every link between them gets the same credit as one that keeps the ten facts in a connected account. Medical and road-network segmentation met the same problem years ago. There, a predicted vessel can overlap the true vessel almost perfectly by volume while being broken into disconnected pieces. That field responded by building metrics and losses that count connected pieces directly, instead of hoping overlap would catch the breaks.

We propose to carry that lesson into summary evaluation. From the source we build a small graph of facts. Nodes are atomic QA pairs, as in existing metrics. Edges are "bridge" questions whose answer requires linking two facts. We then ask which nodes and which bridges survive in the summary, and score the summary by how many disconnected islands its retained facts break into, compared with how connected those same facts were in the source. The payoff is a metric that separates two things current metrics merge. One is acceptable compression, where a peripheral branch of the story is dropped. The other is fragmentation, where a summary keeps both ends of a causal chain but drops the link between them. Fragmentation is the typical failure of summarizers that copy extractively and of long-document summarizers that drop content from the middle of the input.

Core Claim

The claim is that a fragmentation term, defined as the number of extra connected components in the summary's retained fact graph relative to the source graph restricted to the same facts, carries information about human judgments of summary quality that atomic QA recall does not carry. The claim extends to multi-hop questions scored as an unordered bag. The last part is the load-bearing contrast. If generating relational questions and averaging their accuracy works as well as scoring graph structure, the topological part of the idea is unnecessary, and the experiments are designed to show this if it is true.

Let $G_s$ be the source fact graph, and let $V_r$ be the set of fact nodes whose questions the summary answers correctly. Let $G_s[V_r]$ be the source graph restricted to those nodes, and let $G_r$ be the subgraph of $G_s[V_r]$ keeping only the edges whose bridge questions the summary also answers. Writing $\beta_0(\cdot)$ for the number of connected components, we define

$$\mathrm{Frag} = \frac{\beta_0(G_r) - \beta_0(G_s[V_r])}{\max(|V_r| - 1,\, 1)},$$

which is zero when the summary keeps every link the source had among the retained facts, and approaches one when every retained fact is an isolated island. The final score combines standard QA recall $R$ with connectivity, $\mathrm{TQA} = R \cdot (1 - \alpha\, \mathrm{Frag})$. The multiplicative form is borrowed from the T-measure, which multiplies pixel similarity by a topological-integrity term. Here $\alpha$ is fit on a held-out split. Dropping a leaf fact lowers $R$ but leaves $\mathrm{Frag}$ unchanged. Keeping both endpoints of a trunk edge while losing the bridge raises $\mathrm{Frag}$. That asymmetry is the whole point.

Relation to Prior Work

The QA side supplies the machinery and the motivating evidence. Reference-based QAEval (Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary) asks questions generated from a reference summary. Reference-free QuestEval (QuestEval: Summarization Asks for Fact-based Evaluation) asks questions from the source and from the summary, and adds a learned weighter for saliency. We reuse QuestEval's recall direction and its weighter for node questions. Neither metric models dependencies between questions, and that is the gap we fill.

Deutsch and Roth motivate the gap directly. In Understanding the Extent to which Content Quality Metrics Measure the Information Quality of Summaries, they show on TAC 2008 that noun matches explain about 70% of ROUGE while dependency relations explain 2.2%. QAEval does better, with 42% of its score explained by human content units, but it can still be won by entity matching when the questions do not probe relations. In Benchmarking Answer Verification Methods for Question Answering-Based Summarization Evaluation Metrics, they find that a learned answer verifier helps little because summarizers copy spans verbatim. This is a constraint on our design: better answer matching will not expose fragmentation, so the questions themselves must target the links between copied spans.

SummQ (Learning to Summarize by Learning to Quiz) is the closest QA-based system. It quizzes an examinee agent on long documents and feeds the unanswered questions back to the summarizer. It treats its quiz as a bag of items and costs about $14.45 per document. We contribute a structured score rather than a generation system, and our score could later replace SummQ's bag-of-questions feedback signal.

On the topology side, Beyond Words: A Topological Exploration of Coherence in Text Documents computes persistent homology, a summary of how clusters and loops appear as a graph is thresholded, on RoBERTa attention graphs to classify coherence. It reaches 61.1% average accuracy on the Grammarly Corpus of Discourse Coherence but only 1.7% on the medium class. It does not compare a text to a source. We read its weak medium-class result as a warning: topology computed on raw attention, with no anchoring to meaning, is too blunt. Our nodes are anchored to specific verifiable facts. This mirrors the lesson of TopoMatch (Topologically Faithful Image Segmentation via Induced Matching of Persistence Barcodes), which showed that unanchored topological matching pairs features at the wrong locations up to 99.4% of the time.

The component-counting idea comes from the T-measure (T-measure: A Topology-Consistent Metric for Binary Segmentation), which penalizes a true object split into several pieces. It also draws on clDice (clDice - A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation), which separates skeleton coverage from overlap. The stress-test methodology comes from Citraro et al. (Towards Reliable Evaluation of Road Network Reconstructions), who showed that path metrics like APLS stay flat under injected breaks while stricter one-to-one metrics fall monotonically. We deliberately do not propose reachability QA for segmentation masks. That direction is essentially what Citraro et al.'s path metrics and DeepRoadMapper's pathfinding already test.

Research Program

We will build the graph construction in three stages, all using an open instruction-tuned language model held fixed across experiments.

The first stage produces node questions: salient atomic QA pairs from the source, generated QuestEval-style and filtered by its weighter. The second stage proposes edges between pairs of facts that share an entity, lie in an explicit causal or temporal relation, or stand in a coreference chain. For each proposed edge, the model writes a bridge question whose gold answer is derivable from the source only by combining the two facts, for example "What led to the board's decision to X?" where the answer is the content of the other node. The third stage validates each bridge question with two tests: it must be answerable from the source, and it must not be answerable from either endpoint sentence alone. The second test removes bridges that are really single facts in disguise.

Answering uses a QA model restricted to reading the summary. Answers are verified with token F1, with LERC as a secondary option, following Deutsch and Roth's finding that the two rarely differ.

We will also analyze a trunk-weighted variant. In it, each lost bridge is weighted by how many salient node pairs it disconnects, the text analogue of losing a main airway rather than a tenth-generation branch, which the tubular-segmentation review (Human Treelike Tubular Structure Segmentation) singles out as the regime where overlap metrics hide failure. Cost scales with edges, not with node pairs, because edges are proposed only through shared entities or relations. We will report seconds per document next to QuestEval's roughly 2.5 seconds on an RTX 2080.

Baselines

The naive reference points are ROUGE-1 and BERTScore. QuestEval reports ROUGE-1 at a Pearson correlation of 11.8 with human judgments on SummEval with one reference, and on QAGS-XSUM reports ROUGE-1 at 13.2 and BERTScore at 2.5 (QuestEval). Any structural metric that fails to clear these is not worth its cost.

The strongest existing QA metrics are QuestEval and QAEval. QuestEval is reported at a Pearson correlation of 33.5 on SummEval. QAEval with the LERC verifier is reported at a system-level Kendall's $\tau$ of 0.80 on SummEval, against 0.77 with BERTScore verification (Benchmarking Answer Verification Methods). We will rerun both from their released configurations on identical summaries, rather than rely only on reported numbers. For fairness, QuestEval will also be given the same question-generation model and the same total question budget as our method, node plus bridge questions combined. That way any gain cannot come from simply asking more questions. Beating these baselines means a statistically significant improvement in summary-level correlation on SummEval's coherence and relevance dimensions under a bootstrap over documents, with no loss on consistency.

The decisive ablation is a bag-of-bridges baseline. It uses exactly our node and bridge questions and answers, but scores them as a flat, weighted accuracy with no graph. It isolates whether the component-counting structure adds anything beyond relational questions. We tune its question weights with the same held-out budget we use for $\alpha$.

A second ablation replaces bridge questions with edges inferred only from co-occurrence of answered nodes in the same summary sentence. It tests whether explicit bridge questions matter.

We also include a direct language-model judge of coherence, given the source, the summary, and the same model and compute budget. No reported result found on Lacuna for an LLM judge on SummEval coherence, so we will report it as run ourselves.

For the long-document setting, the reference is SummQ's reported 88% human preference over GPT-4o on its long-document benchmarks (SummQ). We treat this as a target the metric should reproduce in ranking, not as a number to beat.

Why not just…?

Why not just ask multi-hop questions and average the accuracy? This is the real threat, and the bag-of-bridges ablation exists to settle it. Our reason to expect a difference is structural. A flat average treats losing one peripheral bridge and losing the single bridge that joins two halves of the story as equal. The fragmentation term does not. This is the same reason clDice and the T-measure track connectivity separately from coverage: in the T-measure's stress test, 40 random cuts lowered an overlap-style measure by only 3.1% while its topological measure fell 84.3% (T-measure). If on SummEval and on our perturbation suite the bag-of-bridges score matches TQA within noise, we will report that the contribution is the bridge questions, not the topology, and drop the topological framing.

Why not just ask a language model to rate coherence? A judge can see fragmentation, but it gives no account of which link broke. It also inherits the length and fluency preferences that make coherence the dimension where SummEval found automatic metrics weakest (SummEval). We run the judge at matched compute. If it matches TQA on correlation, TQA's remaining value is diagnostic, since it names the severed edge, and on the perturbation suite we test whether the judge responds monotonically to controlled cuts.

Why not just use topological features of the text itself, as in Beyond Words? That approach measures coherence of a single text, without reference to what the source said, and its 1.7% medium-class accuracy (Beyond Words) suggests unanchored attention topology misses intermediate cases. A summary can be internally fluent while having cut a source link. We include its features as a baseline on SummEval coherence to test this directly.

Why not just increase QuestEval's question count? More atomic questions still score facts independently. The matched-budget QuestEval run answers this.

Proposed Experiments

The first experiment is a controlled fragmentation stress test modeled on Citraro et al. We start from SummEval reference and high-scoring system summaries and apply four families of edits. Bridge severing deletes connective clauses and causal connectives while keeping both endpoint facts. Extractive splicing replaces a sentence with two source sentences that carry the same entities but not the relation, which imitates the copying shortcut. Leaf omission drops a peripheral fact. Meaning-preserving paraphrase serves as a control. The decisive pattern is that TQA falls monotonically with the number of severed bridges, falls little under leaf omission, and is flat under paraphrase. ROUGE, BERTScore, QuestEval, and QAEval should respond mainly to omission and barely to severing. If TQA does not separate severing from omission, the core claim fails.

The second experiment is meta-evaluation on SummEval: summary-level and system-level Pearson and Kendall correlations with expert annotations for coherence, relevance, and consistency. We report against QuestEval's 33.5 and QAEval-LERC's 0.80 system-level $\tau$, with bootstrap confidence intervals. We also check that TQA adds significant variance explained over QuestEval in a regression on the human scores. Expert agreement on SummEval is Krippendorff's alpha 0.71, which bounds how much any metric can gain.

The third experiment uses QAGS-XSUM, to confirm that the added term does not hurt factual-consistency correlation, against QuestEval's reported 30.4.

The fourth experiment is long-document ranking on BookSum and GovReport summaries from one-shot prompting and from a SummQ-style quiz loop. The hypothesis is that long-document failures from dropping mid-input content show up as higher $\mathrm{Frag}$, and that TQA orders systems the same way SummQ's human study did. We will also annotate 100 summary pairs for "which summary keeps the story connected," to test the fragmentation term against a direct human judgment of the property it claims to measure.