Repairing Extractive Summaries by Local Replanning

Extractive summaries copy whole sentences from the source, so they rarely invent facts. The cost is readability. Sentences pulled out of context arrive with pronouns that point at nothing, with "the trial" before any trial has been mentioned, and with no connectives between them. Systems that try to fix this apply a single global trade-off. They add a coherence term to the selection objective and turn up its weight. That weight then forces the selector to pick adjacent, low-value sentences simply because they connect cleanly. We propose a different approach, borrowed from agents that recover from execution errors. First pick the salient sentences, then check the draft for specific, typed coherence breaks. Then repair each break locally with a small set of plan edits. One edit inserts the antecedent sentence that introduces a dangling referent. Another swaps a sentence for a same-content alternative with a cleaner entry point. A third makes a bounded surface edit: fix an article, expand a pronoun, or add a connective drawn from a fixed list. Every output sentence stays either verbatim from the source or within a small, auditable edit distance of it.

The setting that makes this matter is long biomedical and clinical documents. There, the easy fix of taking the opening sentences fails because findings are scattered across sections. Free rewriting by a language model is also risky there, because it can introduce clinical errors. We also use this setting to test a question about the division of labor between models. Models fine-tuned on a domain appear to be better at deciding what matters and worse at writing prose that flows. If so, the role that selects and verifies content and the role that diagnoses and repairs discourse should be played by different models.

Core Claim

Most coherence failures in extractive summaries are local and nameable: an unresolved pronoun, a definite description with no introduction, an abrupt topic jump, or a missing contrast marker. Because they are local, they can be repaired without re-solving the whole selection. Under a fixed length budget, a detect-then-repair loop should raise human-rated coherence without the salience loss that globally weighted coherence objectives pay. It should also stay far more faithful than letting a generalist model rewrite the summary.

The new ingredient is to treat the extraction plan as an editable graph that is repaired when a discourse check fails. Prior extractive systems treat selection as a one-shot optimization or a forward-only sequence.

Relation to Prior Work

Four lines of work bear on this proposal, in different roles.

Work that motivates the gap. Coherence-aware extractors establish the problem and its trade-off. Two papers encode coherence as frequent entity-graph patterns mined from human abstracts and solve selection with integer programming: Generating Coherent Summaries of Scientific Articles Using Coherence Patterns and Integrating Importance, Non-Redundancy and Coherence in Graph-Based Extractive Summarization. Both report that the Lead baseline still reads best on news. RNES, from Learning to Extract Coherent Summary via Deep Reinforcement Learning, adds a learned sentence-pair coherence reward during sequential selection. It must tune a single weight to avoid trading away content, and it never goes back to revise a sentence once chosen. Contrastive Hierarchical Discourse Graph for Scientific Document Summarization builds section-aware graphs for long papers but is a static extractor. Our proposal differs from all four in the same way: we do not bake coherence into one global objective. We detect failures after selection and fix them with targeted edits.

Work that supplies repair techniques. Two papers provide surface repairs that we reuse as primitive operators. Discourse Factors in Multi-Document Summarization shows that extraction corrupts referring expressions and that rewriting them helps. Post-Editing Extractive Summaries by Definiteness Prediction edits only articles, and expert judges preferred its output 3 to 1. Neither paper can add a missing antecedent sentence or insert a connective. Neither decides when a surface fix is insufficient and the plan itself must change.

Work that supplies the control structure. The loop comes from dynamic plan adaptation. R³DAO repairs a task graph locally with Refine, Insert, and Skip edits when code execution fails. ATLAS replans when observed feedback diverges from what it expected. Dynamic Planning for LLM-based GUI Automation shows that a compact evolving plan beats a growing interaction history. These systems get unambiguous failure signals such as tracebacks and page-element mismatches. Our setting lacks such signals, and that is the main technical risk: the discourse checker may misdiagnose, a failure mode these papers already flag for noisy feedback.

Work that supplies the domain constraint. The division-of-labor question comes from Large Language Models Are Poor Clinical Decision-Makers. It found that clinical instruction tuning improves safety but hurts open-ended summarization, and that experts preferred generalist prose. Specializing Smaller Language Models towards Multi-Step Reasoning shows a similar tax: math specialization collapses general reasoning. Multi-agent systems such as MALADE pair each worker agent with a critic, but they run every role on the same frontier model in a fixed linear pipeline. We keep the worker-and-critic split but make the pipeline repairable and the model assignment a variable we test.

Research Program

The system has three roles.

The selector. It proposes a salience-ranked extraction under a word budget and keeps a ranked pool of runner-up sentences for each chosen one.

The diagnostician. It reads the draft in order and emits typed break reports. Each report names the sentence, the break type (unresolved pronoun, unintroduced definite description, entity discontinuity, or missing relation marker) and the source span believed to resolve it. Break types are chosen so that most can be cross-checked cheaply. An off-the-shelf coreference resolver run over the source can confirm whether an antecedent exists outside the summary. That is our main defense against misdiagnosis.

The repairer. It chooses one of four operators. Insert adds the antecedent sentence and pays for it by dropping the lowest-salience chosen sentence. Swap replaces a sentence with a runner-up carrying the same content. Surface-edit makes a bounded edit: an article change, a pronoun expansion copied from the source antecedent, or a connective from a closed list. Skip accepts the break when every fix costs too much content. Repairs repeat until the diagnostician reports no fixable breaks or a call budget runs out.

All operators leave a trace, so for every token we can say whether it is verbatim, a copied source span, or one of the closed-list edits.

The second strand is the allocation study. We cross two model types with two role groups. The model types are domain-tuned (Meditron-70B or PMC-LLaMA, as evaluated in ClinicBench) and a generalist of similar cost. One role group is selection plus fact checking; the other is diagnosis plus repair. The hypothesis, read off the ClinicBench findings, is that the best team uses the specialist to choose content and the generalist to repair discourse. The hypothesis would be refuted if either uniform team is at least as good on both content and coherence.

Baselines

Naive references. Lead-k and the unrepaired output of our own selector set the naive reference points. Lead-k matters because it wins on human flow for news. We expect it to lose on content for long biomedical papers, where SumPubMed documents that findings are spread across sections. We will verify that rather than assume it. The unrepaired selector is the cleanest reference, because the repair loop is the only difference.

Strongest existing extractor. On PubMed this is CHANGES, which reports ROUGE-L of 41.58 (Contrastive Hierarchical Discourse Graph). We will also apply our repair loop on top of CHANGES outputs, so the comparison separates better selection from better repair.

Coherence-weighted global optimization. This is the competitor on our central claim. On PLOS Medicine we use the coherence-pattern integer-programming system, which reports ROUGE-2 of 0.103 (Coherence Patterns). On CNN/DailyMail we use RNES with coherence reward, which reports ROUGE-1 of 40.95 against 41.25 without it (RNES). We will sweep its coherence weight to trace its content-versus-coherence curve. Beating it means sitting above that curve: higher human coherence at equal ROUGE, or equal coherence at higher ROUGE.

Surface post-editing alone. Our pipeline with only Surface-edit enabled approximates the definiteness post-editor of Post-Editing Extractive Summaries by Definiteness Prediction extended to pronouns. It isolates whether plan-level edits (Insert and Swap) add anything.

Generalist rewrite. A frontier generalist rewrites the unrepaired extract freely. It has the same word budget and a call budget matched to our loop's average.

Untargeted resampling. This ablation draws N alternative extractions and keeps the one the diagnostician scores best. N is set so that total model calls match our loop. It tests whether diagnosis-directed repair matters, or whether spending the same compute on search does as well.

Fairness conditions. All learned components use the same base models, and every system gets the same length budget and a matched number of prompt-tuning iterations on a shared development split. For ClinicBench hospitalization summarization, no reported result found on Lacuna for a summarization score. We will run all baselines ourselves.

Why not just…?

Why not just have GPT-4o or a similar model read the whole paper and write a coherent summary? It probably will read smoothly. ClinicBench reports quality drops above 2,000 words, and our documents run to 3,000 to 4,700. Free generation also gives up the verbatim guarantee that is the reason for extracting in clinical text. This is a real threat on fluency, so the generalist-rewrite baseline is run directly. We score it on coherence and on the fraction of content claims unsupported by the source. The proposal survives only if it matches coherence closely while being clearly more faithful. If the rewrite is equally faithful, the idea loses its purpose.

Why not just raise the coherence weight in an integer program or reinforcement-learning objective? The RNES paper itself reports that higher coherence weight pulls in low-salience connecting sentences. The coherence-pattern papers report that Lead still wins on flow. Our weight sweep will show whether local repair escapes that curve. If the swept baseline's best point matches ours, the claim is false.

Why not just post-edit articles and pronouns? Definiteness editing is cheap and effective. However, it cannot supply a referent that the summary never introduces, and it cannot add connectives. The surface-only ablation measures how many breaks remain once only surface fixes are allowed. If that number is small, Insert and Swap are unnecessary and we will say so.

Why not just sample more candidate extractions and keep the most coherent? Search may substitute for diagnosis. The compute-matched resampling ablation settles this directly.

Proposed Experiments

Datasets and metrics. The primary evaluations use PubMed, PLOS Medicine, and ClinicBench hospitalization summarization. CNN/DailyMail serves as a check where Lead is strong. Because SummEval and SumPubMed show that ROUGE barely tracks coherence, the primary outcome is expert pairwise coherence judgments. SummEval reports Krippendorff's alpha of 0.71 for experts against 0.49 for crowd workers, which is why we use experts. We complement this with automatic counts of unresolved references and a faithfulness audit that uses the per-token provenance trace. ROUGE measures content retention only.

Checker accuracy first. We annotate breaks on about 300 summaries and measure the diagnostician's precision and recall per break type, with and without the coreference cross-check. Everything downstream depends on this step.

Decisive checks. First, at a fixed budget, repair must raise expert coherence preference over the unrepaired selector while keeping ROUGE within the noise of the unrepaired output. Second, it must beat compute-matched resampling. Third, it must match generalist rewriting on coherence with fewer unsupported claims. Fourth, in the allocation study, the specialist-selects, generalist-repairs team must beat both uniform teams on the joint content-and-coherence criterion.

We also report model calls and wall-clock time per summary, since replanning cost is a known weakness of closed-loop agents.