Is the Syntax Detector Reading the Text or the Parser?

Detectors of machine-generated text reach 99% accuracy when training and test text come from the same language model and topic. They collapse when either changes. One proposed fix is to classify sentence structure, the grammatical tree of which word depends on which, instead of word choice, on the theory that structure survives synonym swaps. The best example, PRDetect, keeps 98.6% accuracy on HC3 with 30% of words replaced, while a fine-tuned RoBERTa falls to about 50%. The tree, however, is not observed. An off-the-shelf parser, trained on human news text, produces it. Machine translation research learned long ago that parser mistakes pass silently into everything downstream: roughly a quarter of errors in syntax-based reordering systems trace back to wrong parses (Dependency-based Pre-ordering for Chinese-English Machine Translation).

This note asks a question no detection paper has asked. When a syntax-based detector separates human from machine text, how much of that separation comes from real structural differences, and how much from the parser making different mistakes on the two kinds of text? If the parser's mistakes carry the signal, the detector will generalize exactly as badly as the parser does. The parser's mistakes shift with the generator and the domain. We propose to measure this directly, with a small hand-parsed audit set and parser-swap tests. We then test whether replacing the single "best" tree with the parser's full probability over possible heads, trained to agree across parsers, closes the cross-generator gap.

Core Claim

Syntax-based detectors trained on one best parse from one parser partly learn that parser's error profile. A simple test exposes this: train with one parser and test with another. A second, more direct test compares detector decisions on predicted trees against hand-corrected trees. We expect the damage to be worst under generator shift. HC3 and PRDetect report that ChatGPT text produces deeper trees with more nouns, verbs and conjunctions, which is the regime where a parser trained on news is least reliable (The Change that Matters in Discourse Parsing, Error-repair Dependency Parsing for Ungrammatical Texts).

The secondary claim is constructive. A detector that reads a soft graph can discount parser noise, provided it is also trained to be indifferent to which parser supplied the graph. In a soft graph, the edge weight from word $i$ to word $j$ is the parser's probability that $i$ is the head of $j$.

What would be new is the following. PRDetect treats its parser as an oracle and never measures parser accuracy on machine text. The forest-based translation models (Forest-Based Neural Machine Translation, FOREST2SEQ) show that weighting many parse hypotheses by parser confidence beats committing to one tree. They do this for translation, and they worry only about parser errors hurting quality, not about errors becoming a spurious label-correlated feature. That second failure is specific to classification. It is the thing we target.

Research Program

Audit set. The first component is a small gold treebank for detection. We sample about 600 sentences each from human and machine text across HC3 and three M4 generators: ChatGPT, Dolly-v2, and BLOOMz, which is M4's hardest outlier. Annotators hand-correct them in Universal Dependencies. We then report labeled attachment accuracy (the fraction of words assigned the correct head and relation) for two parsers, split by human vs. machine and by generator. These are a fast statistical parser (spaCy's small model, which PRDetect used) and a transformer biaffine parser. The first result is simply whether the error-rate gap between human and machine text is large and whether it varies by generator. Nobody has reported it.

Attribution. The second component asks where the detector's signal comes from. We retrain PRDetect on HC3 and run it three ways on the gold subset: on predicted trees, on gold trees, and on trees from the other parser. The share of correct decisions that flip when predicted trees are replaced by gold ones estimates how much the detector depends on parser error. A large train-parser/test-parser drop is the operational symptom. We also fit a probe that predicts the detector's output from per-sentence parser error counts alone. If that probe does well, errors are a usable shortcut.

Detector. The third component is the constructive detector. We keep PRDetect's design, RoBERTa word embeddings passed through a two-layer graph convolutional network (a network that updates each word's vector from its neighbors in a graph). We change what the graph is. The input becomes the soft adjacency

$$A_{ij} = p_\theta(\text{head}(j) = i \mid x),$$

the head probabilities of parser $\theta$ on sentence $x$. These are computed exactly for dependency parsers via the matrix-tree marginals, so no forest linearization or sequence blow-up is needed. We add a consistency term

$$\mathcal{L} = \mathcal{L}_{\text{cls}} + \lambda \, \mathrm{KL}\big(f(x, A^{(1)}) \,\|\, f(x, A^{(2)})\big).$$

Here $f$ is the detector's output distribution, $A^{(1)}$ and $A^{(2)}$ come from two different parsers, and $\lambda$ sets the strength. The term pushes the detector toward structure that both parsers agree on and away from either parser's quirks. Dependency marginals rather than phrase-structure forests keep the cost close to PRDetect's. PRDetect reports 2.5 seconds on its benchmark split, against 1,298 seconds for DetectGPT.

Baselines

The naive reference is fine-tuned RoBERTa on raw text. Under a 30% synonym perturbation on HC3, PRDetect reports it at about 50% accuracy, with CoCo at about 53%. We will also run ELECTRA, which M4 found generalizes better across domains.

The strongest direct competitor is PRDetect itself. It reports 98.6% accuracy at 30% perturbation on HC3 and about 87% when trained on HC3 and tested on GPT3.5-Mixed. We found no reported result on Lacuna for PRDetect on M4 cross-generator splits or on MAGE, so we reproduce it on those splits. For the strongest general detector in the wild we use MAGE's Longformer (MAGE). MAGE reports 96.6% in-distribution accuracy, over 60% of human texts misclassified as machine-written in unfamiliar domains, and AUROC falling from 0.94 to 0.75 under paraphrase. As the zero-shot reference we use DetectGPT (DetectGPT), which reports 0.95 AUROC on GPT-NeoX news.

Fairness rules apply to every comparison. All systems share training data, the same RoBERTa-base backbone where applicable, and an equal hyperparameter search budget of 20 trials. MAGE-style boundary adjustment with the same few target examples is applied either to every system or to none.

Four ablations isolate the ingredients. The first is PRDetect with a stronger one-best parser. The second is soft adjacency without the consistency term. The third is the consistency term over two one-best trees. The fourth, a control we consider essential, is the same graph network run on random trees and on a left-to-right chain with matched edge counts. If the chain matches PRDetect, then its robustness comes from the node embeddings and graph smoothing, not from syntax.

Beating PRDetect means three things on M4 leave-one-generator-out, especially held-out BLOOMz: higher mean AUROC, a smaller train-parser/test-parser drop, and a lower false-positive rate on human text at a fixed 5% machine miss rate.

Why not just…?

Why not just use a better parser? This is the real threat. If a transformer parser's error rate on machine text matches its rate on human text, one-best trees from it may be enough. The audit set measures this directly, and the "stronger one-best parser" ablation tests it. If that ablation closes the gap, we report that the fix is parser choice plus a mandatory parser-swap check, and drop the soft-graph method. The finding about parser error in PRDetect-style detectors stands either way.

Why not just combine the top few parses or vote across parsers? A top-k union is a cheap approximation of the soft graph. Forest-Based NMT found that multiple trees without confidence scores did significantly worse than confidence-weighted ones. Their setting was translation, however, so we include the unweighted-union variant rather than assume the result transfers.

Why not skip syntax and recalibrate a strong text detector? MAGE shows a few target examples fix much of the out-of-distribution miss rate. That shifts a threshold without changing the representation, and paraphrase still costs 0.19 AUROC. We give the text baselines the same boundary adjustment so that any remaining gap is attributable to representation.

Why not flat part-of-speech counts? UMUTeam's fusion of part-of-speech statistics with RoBERTa (UMUTeam at SemEval-2024 Task 8) reached only 75.35% accuracy against a 74.61% baseline. It also mislabeled 27.73% of human texts as Dolly, consistent with tagger artifacts biasing the classifier. We include it on the SemEval Subtask B split.

Proposed Experiments

Training runs on HC3 English and on M4 English with one generator held out at a time. We test across generators, across domains (arXiv to Reddit, which M4 identifies as the domain that transfers best), and on MAGE's unseen-model and unseen-domain splits. Metrics are AUROC, accuracy, false-positive rate on human text, and the train/test parser-swap drop. For perturbation we reuse PRDetect's WordNet synonym protocol and add sentence-level paraphrase, to resolve the tension between PRDetect's robustness and MGTBench's collapse from 0.99 to 0.04 F1 (MGTBench).

Three checks decide the outcome. First, does PRDetect's accuracy on the gold subset change when predicted trees are swapped for gold trees, and by more on machine text? Second, does the chain-graph control match PRDetect? Third, does the parser-invariant soft-graph detector beat both PRDetect and the stronger-parser ablation on held-out BLOOMz?

A null on the first check, combined with a positive on the second, would be the most consequential outcome. It would mean syntax-tree detectors were never really using syntax.