Story generators and summarizers increasingly depend on learned coherence judges. These are models that read a passage and flag where a story stops making sense: a character appears without introduction, a scene jumps abruptly, or one event contradicts another. The judges pick among candidate continuations and score finished systems. A separate literature shows that neural text classifiers can be flipped by swapping a few words for synonyms. Nobody has asked whether coherence judges share this weakness. If they do, a coherence score partly measures wording rather than story structure, and any generator tuned against that score learns to please the wording.
The question also runs in the opposite direction. The tools that keep word-swap attacks "meaning-preserving" check that a sentence embedding barely moves. Coherence, though, often hinges on exactly the small words those checks treat as interchangeable. Replacing "before" with "after," or "because" with "despite," can leave a sentence looking nearly identical to an embedding model while reversing the plot's logic. A good coherence judge has to satisfy two requirements together. It should give the same verdict when synonyms change, and it should change its verdict when a single connective or name changes the story. This proposal builds a benchmark of human-validated minimal edit pairs that tests both requirements, trains a judge with an objective that enforces both, and checks whether the result helps when the judge is used to rerank long-story generation.
Core Claim
Our hypothesis is that current coherence detectors fail in both directions. They change verdicts under synonym swaps that do not change the story, and they miss relation-reversing swaps of discourse connectives and character mentions that do. We further expect that the standard robustness fixes trade one failure for the other. Synonym augmentation and embedding-alignment defenses should buy invariance by making the detector blind to small words, much as certified word-substitution defenses pay a clean-accuracy "robustness tax." A two-sided training objective should avoid that trade: it pulls representations of coherence-preserving edits together and pushes coherence-breaking edits apart. The test that matters is whether such a judge keeps or improves its recall on real model-generated narrative errors, rather than only on our edits.
What Prior Work Establishes and Leaves Open
The coherence side supplies both the evaluation target and a warning. SNaC: Coherence Error Detection for Narrative Summarization annotates span-level coherence errors in GPT-3 and BART book and movie summaries. Its categories are character introduction errors, unreferenced mentions, abrupt scene transitions and inconsistencies. It trains a T5 detector that reaches 0.90 recall on character errors and 0.82 on scene transitions. It also shows that detectors trained on synthetic corruptions such as sentence shuffling do not transfer to real errors. That negative result constrains our design. Our breaking edits are synthetic too, so evaluation on real SNaC errors has to be the deciding test, not a side check.
Re3: Generating Longer Stories With Recursive Reprompting and Revision uses a trained reranker to choose among candidate passages and a rule-based editor for character-attribute contradictions. Both components take generated text at face value, and their sensitivity to wording has never been tested.
DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer learns discrete plan codes tied to discourse relations such as temporal, causal and conjunction. It shows that baseline models get connectives like "before" wrong much of the time: accuracy rises from 30% for BART to 67% for DiscoDVT. This motivates our choice of connectives as the place where meaning-changing edits hide. We use DiscoDVT's relation categories to organize our breaking edits.
The word-substitution side supplies the attack machinery and the defenses we will compete with. Is BERT Really Robust? introduces TextFooler. It ranks words by how much deleting them changes the model's confidence, then substitutes counter-fitted synonyms that pass a part-of-speech check and a Universal Sentence Encoder similarity check. With it, BERT on SNLI drops from 89.4% to 4.0% accuracy. We reuse this pipeline to attack detectors, and we test whether its embedding-similarity filter waves through relation-reversing connective swaps. Are Synonym Substitution Attacks Really Synonym Substitution Attacks? and How Far Are We from Real Synonym Substitution Attacks? question whether automated swaps actually preserve meaning. That is why every pair in our benchmark is human-validated rather than trusted by construction.
On defenses, Certified Robustness to Adversarial Word Substitutions certifies classifiers by tracking the range of activations every synonym combination could produce. It pays roughly eight points of clean accuracy on IMDB, falling from 89% to 81%. SAFER certifies classifiers by majority vote over random synonym substitutions and reaches 87.35% certified accuracy with BERT on IMDB, at a cost of 5,000 forward passes per input. Natural language adversarial defense through synonym encoding maps each synonym cluster to a single representative word before the model sees it. Self-Supervised Contrastive Learning with Adversarial Perturbations pulls representations of perturbed and clean sentences together. Word Level Robustness Enhancement observes that models withstand random swaps but not targeted ones, which tells us to attack detectors with search, not random noise. Every one of these defenses targets invariance alone, and every one was evaluated on short classification tasks. None of them asks whether invariance destroys sensitivity to the words that carry discourse structure. That is the gap we fill.
Research Program
The first component is a two-sided edit benchmark. We draw 2–4-sentence segments with preceding context, the same localized format SNaC used to raise annotator agreement from Krippendorff's α of 0.19 to 0.69. Segments come from SNaC summaries and from WritingPrompts stories. For each segment we create edits of two kinds.
Coherence-preserving edits are of three types. The first are TextFooler-style counter-fitted synonym swaps. The second are LLM paraphrases that keep every connective's relation. The third are swaps of connectives within the same relation, such as "afterwards" for "later."
Coherence-breaking edits are minimal, usually a single token. We swap a connective across relations, such as before/after, because/despite, or meanwhile/afterwards. We swap a character mention for another character already in context, which induces SNaC-style character or reference errors. And we swap a time or place word in ways that contradict the earlier context.
Annotators follow the SNaC protocol and label whether each edited segment contains a coherence error and of which type. Pairs whose labels disagree with their intended category are dropped. For every pair we record the Universal Sentence Encoder similarity. This tells us directly how many breaking edits pass the filter used by standard attacks, which tests the claim that sentence-level similarity is blind to discourse logic.
The second component is a diagnosis of existing judges. We run four judges on the benchmark: the SNaC-style T5 detector, a Re3-style coherence reranker, and two instruction-tuned LLMs used as zero-shot judges. We measure two things. Invariance is the rate at which a judge's verdict flips on preserving pairs, under both random and TextFooler-targeted search. Sensitivity is the rate at which a judge detects breaking pairs.
The third component is a two-sided judge. We fine-tune the detector with its usual error-detection loss plus a pairwise term. Writing $h(x)$ for the detector's segment representation, the term is
$$ L_{\text{pair}} = \sum_{(x,x^+)} \lVert h(x)-h(x^+) \rVert^2 + \sum_{(x,x^-)} \max\big(0,\ m-\lVert h(x)-h(x^-)\rVert\big)^2 , $$
where $x^+$ is a coherence-preserving edit of segment $x$, $x^-$ is a coherence-breaking edit, and $m$ is a margin. The first sum enforces invariance and the second keeps breaking edits at least distance $m$ away. To prevent this from becoming a "spot the synthetic edit" game, the training edits use different connective pairs and different source stories from the test edits.
The fourth component applies the judge to generation. We use each judge as the reranker in a Re3-style best-of-N story pipeline on WritingPrompts premises and on Re3's 100 machine-generated premises. We then ask whether the two-sided judge produces stories with fewer human-annotated coherence errors, and whether its scores become harder to inflate by rewording.
Baselines
The naive reference point is the SNaC-style T5 detector trained only on SNaC annotations, with no robustness treatment. Its reported recall on real errors, 0.90 for character errors and 0.82 for scene transitions, defines the clean performance any robust variant must preserve. We found no reported result on Lacuna for the robustness of any coherence detector to word substitution, so its flip rates and connective sensitivity are measurements this project produces.
The strongest certified competitor is SAFER applied to the same detector. This is a fair comparison because a detector is a short-segment classifier, the setting where SAFER reports 87.35% certified accuracy with BERT on IMDB. We run it with its reported settings of K=100 and n=5000 samples, using the same synonym dictionary our preserving edits come from. It wins on certified invariance by design, so we compare on sensitivity to breaking edits and on recall on real SNaC errors. Our method beats it if it matches SAFER's empirical flip rate under targeted attack within a few points while detecting substantially more breaking edits and losing no real-error recall. It also has to do this with a single forward pass instead of 5,000.
We add three empirical invariance defenses on the same backbone, with the same training data and the same hyperparameter-search budget. The first is synonym-swap data augmentation. The second is the contrastive alignment of Self-Supervised Contrastive Learning with Adversarial Perturbations. The third is the cluster-representative input encoding of Natural language adversarial defense through synonym encoding. Zero-shot LLM judges serve as the scale baseline.
The key ablations remove one half of the pairwise loss at a time. An invariance-only variant tests whether the sensitivity term is what preserves connective detection. A sensitivity-only variant tests whether the invariance term is needed at all. We also train a variant on breaking edits alone, without real SNaC data, to measure how far synthetic edits transfer by themselves.
Why not just…?
Why not just augment training with random synonym swaps? This is the most likely cheap fix. Our concern is that it will not help in either direction. Word Level Robustness Enhancement reports that models already survive random swaps and fail under targeted ones, so augmentation with random swaps may add little against search-based attacks. Augmentation that includes connective-adjacent words may also teach the model to ignore them. The augmentation baseline, run on equal data and budget and measured on both targeted-attack flip rate and breaking-edit detection, settles this directly. If augmentation matches the two-sided judge on both measures, the pairwise term is unnecessary and we will report that.
Why not just canonicalize the text, mapping every synonym to one representative before scoring? For a fixed synonym dictionary this makes invariance exact at no cost, so it is a real threat on the invariance axis. Its weak points are elsewhere. It handles only the swaps in its dictionary, not LLM paraphrases. It will also either leave connectives untouched, so they stay fragile, or merge connectives, which merges relation-reversing ones like "after" and "afterwards" in some embedding tables. The breaking-edit split shows which of these happens. Canonicalization is also compatible with our objective, and we test the combination.
Why not just use a large LLM as the judge? LLM judges may already be invariant to synonyms. But the instruction-robustness studies Enhancing LLM Robustness to Perturbed Instructions and Towards Self-Robust LLMs via CoIPO show that LLMs shift under lexical variation, and DiscoDVT's results show that strong generators mishandle temporal connectives. We run LLM judges on both splits. If they pass both, the contribution becomes the benchmark itself together with evidence that small trained detectors need the fix. That is still useful, because rerankers inside generation loops are usually small and cheap.
Proposed Experiments
The first decisive check is the premise. On the human-validated benchmark, we test whether the baseline detector flips verdicts on preserving pairs under targeted search, and whether it misses a substantial fraction of breaking pairs that the Universal Sentence Encoder scores as near-identical. If the detector is already two-sided robust, the project shrinks to a negative result about the problem's existence, which is still worth reporting.
The second check is the trade-off. We plot targeted flip rate against breaking-edit detection for all baselines and ablations. The prediction is that invariance-only defenses move along a trade-off curve, while the two-sided judge sits above it.
The third check is transfer, the one SNaC's negative result makes mandatory. We measure per-category recall and precision on held-out real SNaC errors. A judge that improves on edits but loses recall on real character or scene errors counts as a failure.
The fourth check is downstream. In Re3-style best-of-N reranking on WritingPrompts, we compare stories chosen by each judge. Annotators apply the SNaC protocol blind to condition and report errors per thousand words. We also measure how much each judge's score can be raised by rewording a candidate without changing its plot, using targeted synonym search against the judge. The two-sided judge succeeds if it yields fewer annotated errors and smaller gains from rewording than the baseline reranker at equal N.
References
- SNaC: Coherence Error Detection for Narrative Summarization
- Re3: Generating Longer Stories With Recursive Reprompting and Revision
- DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer
- Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment
- Are Synonym Substitution Attacks Really Synonym Substitution Attacks?
- How Far Are We from Real Synonym Substitution Attacks?
- Certified Robustness to Adversarial Word Substitutions.
- SAFER: A Structure-free Approach for Certified Robustness to Adversarial Word Substitutions
- Natural language adversarial defense through synonym encoding
- Self-Supervised Contrastive Learning with Adversarial Perturbations for Defending Word Substitution-based Attacks
- Word Level Robustness Enhancement: Fight Perturbation with Perturbation
- Enhancing LLM Robustness to Perturbed Instructions: An Empirical Study
- Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPO