Symmetry-Protected Detection: Measuring What Survives Paraphrase

Detectors for machine-written text fail under paraphrasing. Rewriting a watermarked passage with an off-the-shelf model drops watermark detection from 99.3% to 9.7% true positives at 1% false positives, and cuts DetectGPT's AUROC (area under the ROC curve; 0.5 means chance) from 96.5% to 59.8% (Can AI-Generated Text be Reliably Detected?). The standard explanation is that the human distribution $P$ and the machine distribution $Q$ are converging, and that total variation distance, the largest possible difference in probability the two assign to any event, caps what any detector can achieve. That explanation mixes two questions. The first is how far apart $P$ and $Q$ are. The second is how much of that gap survives the attacker's rewriting. This proposal is about the second. We ask which properties of a text a paraphraser cannot change, and whether human and machine text still differ on those properties.

Condensed-matter physics has a working method for this kind of question, and we borrow it. In crystals, an energy degeneracy is called "symmetry-protected" when it survives every perturbation that respects a given symmetry. Physicists find out which degeneracies are protected by breaking one symmetry generator at a time and watching which gaps open. In strained SrIrO$_3$, breaking a single glide plane (a reflection combined with a half-step shift of the lattice) while keeping the mirror opens a gap of about 5 meV that scales with the strain (Strain-induced nonsymmorphic symmetry breaking...). In KHgSb, a structural transition breaks inversion but keeps the glide, and the topological states survive (Robustness of topological states...). "Nonsymmorphic" symmetries are the ones that pair a point operation with a fractional translation. Their lesson is that the protected quantities are set by the composite operation, and would not be predicted from either piece alone. We apply this lens, and not a literal Hamiltonian, to paraphrase attacks.

Core Claim

Model the attack as a random rewriting channel $K$ applied to text. By the data-processing inequality, any statistic $f$ that the attack leaves unchanged ($f(Kx)=f(x)$) keeps its human-versus-machine distance exactly. This gives a floor on post-attack detectability:

$$\mathrm{TV}(P_f, Q_f) \;\le\; \mathrm{TV}(PK, QK) \;\le\; \mathrm{TV}(P, Q),$$

where $P_f$ and $Q_f$ are the distributions of $f(x)$ for human and machine text. The right-hand inequality is the ceiling from Can AI-Generated Text be Reliably Detected?. The left-hand quantity, which we call the protected divergence, is what a robust detector can hope to use.

Real paraphrasers are built from simpler moves: word substitution (a "point" edit), reordering of clauses and sentences (a "translation"), and character or subword changes that shift token boundaries by less than one token (a "fractional translation" relative to the tokenizer's grid). The nonsymmorphic analogy makes a prediction we can test. Statistics protected against each move separately will not in general be protected against their composition. Letter-frequency signatures ignore word order but not substitution. Syntactic sequences tolerate substitution but not reordering. The protected divergence under full paraphrase should therefore be carried by a smaller, specific family of statistics. Those statistics can be learned directly as invariants of the composite move set, and their residual divergence can be measured.

The contribution has two parts: a symmetry-resolved robustness map of existing detection statistics, and an estimator of the protected divergence under composite attacks. If the protected divergence is near zero, that is a strong result too. It would turn a worst-case impossibility argument into a measured statement about specific attack families.

Relation to Prior Work

Two results motivate the work, and they appear to contradict each other. Can AI-Generated Text be Reliably Detected? and Limits of Detecting Text Generated by Large-Scale Language Models argue that detection collapses as $\mathrm{TV}(P,Q)\to 0$. In contrast, Beyond Perplexity: Character Distribution Signatures and the MDTA Benchmark collapses the vocabulary into 26 letter frequencies and reaches AUROC 0.960 on MDTA. The contradiction is only apparent. By data processing, a projection can never increase total variation. The letter-frequency result therefore shows that full-space TV is not yet small for these models, and that a low-dimensional statistic is easier to estimate from finite samples. Our framing keeps both results and adds the missing variable, the attack channel. The MDTA paper itself reports that its signal weakens on open-domain writing and under adaptive paraphrase. We test both conditions.

Two watermarking papers constrain the design. Undetectable Watermarks for Language Models and MirrorMark keep the output distribution unchanged by applying keyed transformations to the sampling randomness. MirrorMark's version is a measure-preserving reflection of the random number used to pick each token. Both methods detect by re-deriving that randomness from token-aligned context, so any edit that breaks alignment destroys the signal. We therefore do not propose a symmetry-themed watermark. The failure point is the alignment requirement, and a group action on the sampling randomness would inherit it. Post-hoc detectors supply the method. DetectGPT scores text by how its likelihood changes under masked rewrites. MAGE provides out-of-distribution evaluation, boundary adjustment, and a paraphrase split. We reuse both as statistics to be placed on the robustness map. Finally, Observation of a Charge-2 Photonic Weyl Point shows that the logic of nonsymmorphic protection carries over to classical waves. We only claim that it carries over as a method: break one generator at a time and read off what survives.

Research Program

Attack generators. We first build a library of attack generators, each targeting one "symmetry" of the text:

  • lexical substitution by masked-language-model word replacement at controlled rates, following DetectGPT's replacement protocol;
  • sentence and clause permutation that preserves meaning;
  • sub-token character edits and lipograms, the adversarial strategies already present in MDTA;
  • the full composite, which is recursive paraphrasing with the DIPPER and LLaMA-2 setup of Can AI-Generated Text be Reliably Detected?.

Robustness map. For every existing statistic (perplexity, DetectGPT curvature, the SurpMark divergence from Black-Box Detection... Generalized Jensen Shannon Divergence, LD-Score, and a fine-tuned Longformer), we measure the human-versus-machine divergence before and after each generator. Each generator is applied at graded strength. This gives a statistic-by-generator matrix together with dose-response curves. It is the analog of mapping which gaps open under which strain.

Learned invariant statistics. Next we learn statistics invariant to the composite move set. A text encoder is trained contrastively so that a text and its attacked versions map to nearby representations. The attacked versions are orbits sampled from compositions of the generators. The encoder is trained only on text whose origin label is discarded. Separately, a small classifier on the frozen representation estimates the protected divergence as a held-out AUROC with bootstrap confidence intervals. We also convert it to a total-variation lower bound using the classifier's accuracy. Keeping invariance learning and detection separate matters: it ensures the encoder was not tuned to separate the classes.

Baselines

The naive reference points are a chance-level detector (AUROC 0.5) and raw perplexity thresholding. MAGE reports that raw perplexity thresholding misclassifies over 60% of human texts as machine-written out of domain. Any statistic we report must beat perplexity under the composite attack. Otherwise the robustness map says nothing.

DetectGPT is the strongest zero-shot competitor on this question. Its reported numbers under recursive paraphrasing come from Can AI-Generated Text be Reliably Detected?: AUROC falls from 96.5% to 59.8%. We rerun it on the same generations with the same scoring model and number of perturbations. Beating it means a post-paraphrase AUROC clearly above 59.8% with non-overlapping bootstrap intervals.

The strongest supervised competitor is the MAGE Longformer with boundary adjustment. MAGE reports in-distribution accuracy of 96.6% and AUROC falling from 0.94 to 0.75 under paraphrase. It gets the same in-domain adjustment examples we allow ourselves.

The strongest low-dimensional competitor is LD-Score combined with DNA-DetectLLM. On MDTA it reports average F1 0.921 and AUROC 0.960 (Beyond Perplexity). We evaluate it per adversarial strategy, which the paper does not break out.

The baseline that matters most is a paraphrase-augmented supervised detector. This is the Longformer trained on the same attack orbits we use for invariance learning, with an equal compute and hyperparameter budget. No reported result was found on Lacuna for paraphrase-augmented detectors on these benchmarks.

Two ablations isolate the key ingredient:

  • Single-generator invariance. The encoder is trained to be invariant to one generator only. This tests the claim that invariance to the composition carries information that single invariances do not.
  • No invariance objective. The same encoder is used without the invariance loss. This tests whether invariance, not architecture, drives the result.

Why not just…?

Why not just recalibrate the threshold on a few in-domain examples? MAGE shows that boundary adjustment with about 0.1% of data raises GPT-4 recall above 80%. The same paper shows that paraphrase still drops AUROC from 0.94 to 0.75. Recalibration shifts the decision threshold but cannot restore a signal the attack has erased. We apply boundary adjustment to every statistic, including ours, so it becomes a shared add-on rather than a competitor.

Why not just use letter-frequency signatures, which already reach 0.960 AUROC? Letter frequencies ignore order, so reordering attacks cannot touch them. The robustness map will show whether they are also protected against substitution and sub-token edits. The authors themselves concede that adaptive paraphrase and persona prompting erode the signal, and that it is weakest on WritingPrompts. If LD-Score survives the composite attack as well as learned invariants do, we will report that the protected sector is essentially letter statistics. That would be a clean and useful finding, not a failure.

Why not just train a detector on paraphrased examples? This is the real threat. Learned invariance and augmentation overlap heavily. The difference is in what each produces. Augmentation gives one classifier's accuracy. Our separate estimator gives an attack-specific lower bound, a breakdown of which moves destroy which signals, and a test of the composite-versus-single prediction. If the augmented detector matches our invariant detector's AUROC, we will say so and present the map and the bound as the contribution. If it exceeds our protected divergence, the invariants we learned are incomplete, and that is a measurable diagnostic in its own right.

Why not just use a distortion-free watermark? Watermarks answer a different question: they need cooperation from the model provider. Under paraphrase they fail more severely than post-hoc detectors, with true positives falling from 99.3% to 9.7%. They are also unavailable for deterministic, low-entropy outputs (Undetectable Watermarks).

Proposed Experiments and Decisive Checks

Data. We generate machine text on XSum, SQuAD, and WritingPrompts, matching the DetectGPT and Sadasivan et al. setups. We add MDTA across its four models, three temperatures, and adversarial strategies, and use MAGE's out-of-distribution and paraphrase splits for transfer. All numbers are post-attack AUROC and true positive rate at 1% false positive rate, with bootstrap intervals over prompts.

Composite prediction. The first decisive check tests the composite-versus-single prediction. Take statistics that are robust to each generator applied alone. Under the composite attack they should lose most of their divergence. Statistics learned to be invariant to the composite should keep significantly more. If the composite does no more damage than the worst single generator, the nonsymmorphic analogy adds nothing, and we will report the map without it.

Graded breaking. The second check applies each generator at graded strength. Protected statistics should show flat dose-response curves, and unprotected ones should decline monotonically. This is the textual analog of a gap that opens in proportion to strain.

Content-origin sanity check. The third check addresses a confound. A statistic invariant to paraphrase must label LLM-paraphrased human text as human. If it labels that text as machine-written, it is detecting the rewriting model rather than the origin of the content, and the protected-divergence estimate is invalid. We report this false positive rate for every method.

Scaling. Finally, we plot protected divergence against generator size within the MDTA model families. The question is whether the protected sector shrinks with scale, as the convergence argument predicts, or stays roughly constant. If it stays constant, the robust part of detection is not governed by ambient convergence.