Detectors of machine-generated text look excellent on aggregate metrics and then fail where they are actually used. When the threshold is set so that almost no human writer is wrongly flagged, say one in a thousand, their detection rate falls to nearly nothing. A second, quieter problem makes this worse: even when a detector does work at that threshold, nobody can show it does for a particular population, such as non-native test-takers or expert peer reviewers. Measuring a one-in-a-thousand error rate directly takes many thousands of labeled human texts from that exact population. Rare-event physics has lived with both problems for decades. A neutrinoless double-beta decay search must reject background that outnumbers the signal by many orders of magnitude, and it cannot collect a large clean sample of the signal region. Physicists do not rely on one very strict classifier. They stack several independent cuts, and they estimate the background left after all cuts by multiplying leakage rates measured in "control regions", where only some of the cuts are passed.
We propose to bring that practice to text forensics. The detector would require two or more scores to exceed moderate thresholds. The second score would be trained so that it is statistically independent of the first on human text. The joint false-positive rate would then be estimated per human subgroup as the product of the separately measured single-cut leakages, with a correction factor fitted where direct measurement is still possible. The goal is twofold: more sensitivity at a fixed false-positive rate of $10^{-3}$ or lower, and a credible false-positive estimate for a subgroup from roughly the square root of the human data a direct measurement would need.
Core Claim
Let $s_1, s_2$ be two detector scores and $H$ the human-text distribution of one subgroup, for example non-native essays. The joint false-positive rate of the "both cuts pass" rule is
$$\mathrm{FPR}(t_1,t_2) = P_H(s_1>t_1,\ s_2>t_2) = P_H(s_1>t_1)\,P_H(s_2>t_2)\,\kappa(t_1,t_2),$$
where $t_1, t_2$ are thresholds and $\kappa$ measures how far the two cuts are from independent under human text ($\kappa=1$ means exact independence). If $\kappa$ is close to 1 and stable, reaching a target $\alpha$ needs each marginal leakage near $\sqrt{\alpha}$ only, for example 3% for $\alpha=10^{-3}$. Each marginal can be measured with about $10/\sqrt{\alpha}$ human samples rather than $10/\alpha$, because a rate $p$ needs on the order of $1/p$ samples to see enough exceedances.
Signal acceptance is the product of each detector's true-positive rate at a loose 3% cut, not one detector's rate at a strict 0.1% cut. The collapse documented by Tufts et al. suggests these differ enormously. We make three claims:
- training the second detector to be independent of the first on human text makes $\kappa$ near 1 and stable across subgroups;
- the factorized estimate of $\kappa$, fitted at moderate thresholds and extrapolated, passes held-out "closure tests" in which the true joint rate is measured directly;
- the resulting rule has higher sensitivity at matched, audited false-positive rate than a single conformally thresholded detector.
This estimate is not distribution-free. We trade the finite-sample guarantee of conformal calibration for data efficiency, and we make that trade testable rather than hidden.
Relation to Prior Work
The motivating evidence comes from benchmark audits. Tufts et al. show detectors catching 99% of synthetic text at a 20% false-positive rate and "virtually zero" at 0.1%. RAID shows fixed thresholds failing across generators, domains and attacks. Niu et al. and Yu et al. show that the false positives concentrate on specific human groups: non-native test-takers and expert reviewers. Together these say the problem is subgroup-level tail behavior, not average separation.
The closest methods supply our baselines rather than our mechanism:
- Multiscaled Conformal Prediction calibrates a threshold on human reference text so the false-positive rate is bounded without parametric assumptions. It certifies one score marginally over the reference pool. It does not raise sensitivity at the bound, and it cannot certify $10^{-4}$ for a subgroup with a few thousand samples.
- The deep kernel relative test handles drift in the human distribution by testing against reference anchors. We reuse it as one candidate score.
- Style representations provide a detector family that plausibly errs on different human texts than likelihood-based scores. That makes it a natural second veto.
- End-to-end low-FPR ranking optimizes sensitivity directly in the low-FPR region. It addresses sensitivity but not certification, and we compare against it.
The mechanism comes from physics:
- The MAJORANA program reaches its background goal through layered, separately characterized rejection steps, such as delayed charge recovery for surface alphas. DEAP-3600 does the same with pulse-shape discrimination, where the waveform's shape separates signal from background.
- Gambhir et al. estimate background under a signal peak from sidebands, which is the same logic as our control regions.
- PandaX-4T freezes its analysis before unblinding. We adopt that discipline for test data.
A constraining lesson from León et al.: their LEGEND waveform-cleaning pipeline, which groups waveforms with a clustering method called affinity propagation and filters them with support vector machines, provides no bound on signal loss. That warns against a stacked pipeline whose leakage is never measured. Every cut here carries a measured leakage.
Research Program
We would build three things.
A decorrelated second detector. Starting from a strong first score $s_1$, such as a perplexity-style zero-shot score, we train $s_2$ on a style or kernel representation to separate human from machine text. The training adds a distance-correlation penalty between $s_1$ and $s_2$ computed on human examples only. We do not decorrelate on machine text, where agreement between the two detectors is desirable. This is the physics idea of decorrelating a discriminant from the background-estimation variable, applied to human text as the background.
A per-subgroup background budget. For each human subgroup (general web text, non-native essays, peer reviews, paraphrased human text) we measure the two marginal leakages. We fit $\kappa$ on a grid of moderate thresholds where the joint rate is directly observable, and extrapolate it to the target region. The estimate comes with bootstrap intervals and a conservative inflation factor.
A blinding protocol. Thresholds, $\kappa$ fits and inflation factors are frozen on calibration data before any test-split human text is scored.
The analysis side asks when factorization fails. Our working hypothesis, drawn from the edge-effect analogy in the research file, is that failure concentrates in texts that are simultaneously short, formulaic and low-perplexity. For such texts we would declare the input out of scope, the analogue of a fiducial cut that discards an ill-defined detector region. We would report the fraction of texts excluded, not hide it.
Baselines
The naive reference is the best single detector with its threshold set on pooled validation human text to hit the target rate. This is the setting in which Tufts et al. report 99% true-positive rate at 20% false-positive rate collapsing to "virtually zero" at 0.1%. That is the only reported number in this regime; no reported result was found on Lacuna for per-detector true-positive rate at 0.1% on RAID, the peer-review benchmark, or the language-test corpus.
The strongest certification competitor is Multiscaled Conformal Prediction. We run it both pooled and group-conditional, with each subgroup calibrated separately, on the same calibration human texts. Group-conditional calibration is the sharpest version a reviewer would propose.
The strongest sensitivity competitor is a learned combiner over $s_1$ and $s_2$ with a conformal threshold. We will implement two versions: logistic stacking, and the low-FPR ranking objective. Both get the same features, training data and hyperparameter budget as our rule. We also include the deep kernel relative test as a strong drift-robust single score.
Two ablations isolate the key ingredient:
- the "both cuts pass" rule with an ordinary, non-decorrelated second detector, which tests whether decorrelation matters;
- the same rule thresholded by direct conformal calibration instead of factorized estimation, which tests whether the budget estimate matters.
We beat the field if two things hold at an audited false-positive rate of $10^{-3}$ in each subgroup. First, the true-positive rate exceeds group-conditional conformal on the best single score and matches or exceeds the learned combiner. Second, with only $10/\sqrt{\alpha}$ calibration texts per subgroup, our estimated rate lies within its interval of the directly measured rate, while the direct methods cannot certify at all.
Why not just…?
Why not just use group-conditional conformal calibration on the best detector? It gives a real finite-sample guarantee, and when a subgroup has more than about $10/\alpha$ human samples it is the correct tool. Our comparison runs it at full strength. Its limits are twofold. It does nothing about the collapse in sensitivity that Tufts et al. document, since it only places the threshold honestly on the same flat curve. And for small subgroups like the language-test population of Niu et al., it cannot reach $10^{-3}$ at all. If our sensitivity gain vanishes in the large-subgroup setting, the contribution shrinks to data-efficient certification only, and we would report that.
Why not just train a learned combiner on both scores? This is the most serious threat. In principle the optimal combination dominates a rectangular cut. The problem is that the combiner's decision boundary is shaped by the bulk of the data. Its extreme human tail must still be certified directly, needing $10/\alpha$ samples, and it cannot factorize. We will run the combiner and the ranking objective with an equal budget. If they win on sensitivity at matched direct certification, we would adopt the combiner as $s_1$. The factorized budget still applies whenever a decorrelated veto can be added on top.
Why not just adopt watermarking and skip passive detection? Watermarks such as the distribution-adaptive and multibit schemes do give principled false-attribution control. However, they only cover cooperating generators, and the survey of Yang et al. notes they are absent or removable for open or adversarial models. This proposal targets that uncovered case.
Proposed Experiments
The data come from four sources. Human and machine text across generators and attacks from RAID form the general subgroup and the paraphrase-attack setting. The language-proficiency corpus supplies the non-native subgroup. The peer-review benchmark supplies the expert subgroup. The practical evaluation suite lets us reproduce the single-detector collapse. Metrics are:
- true-positive rate at audited false-positive rates of $10^{-2}$, $10^{-3}$ and, where the human pool allows direct checking, $10^{-4}$;
- the ratio of estimated to measured false-positive rate per subgroup;
- the fraction of inputs excluded by the out-of-scope cut.
Closure test. On the largest human pool, fit $\kappa$ using thresholds whose joint rate is at least $10^{-2}$. Then predict the joint rate at $10^{-3}$ and compare it with direct measurement on held-out human text never used in fitting. The method survives only if the prediction lies within its bootstrap interval in every subgroup, both with and without the decorrelation penalty. We expect the non-decorrelated version to fail closure, with $\kappa$ drifting well above 1.
Transfer test. Fit $\kappa$ on general web text and check closure on non-native essays and peer reviews without refitting. This asks whether independence under human text is a property of the detector pair or of the population.
Sensitivity test. At matched, directly audited $10^{-3}$, compare the true-positive rate of the decorrelated two-cut rule against every baseline above, all under frozen, blinded thresholds.
If closure fails outside the out-of-scope region, the factorized budget is not trustworthy, and the proposal reduces to an empirical study of detector decorrelation.
References
- A Practical Examination of AI-Generated Text Detectors for Large Language Models
- RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
- Detecting LLM-Assisted Cheating on Open-Ended Writing Tasks on Language Proficiency Tests
- Is Your Paper Being Reviewed by an LLM? Benchmarking AI Text Detection in Peer Review
- Reliably Bounding False Positives: A Zero-Shot Machine-Generated Text Detection Framework via Multiscaled Conformal Prediction
- Deep Kernel Relative Test for Machine-generated Text Detection
- Few-Shot Detection of Machine-Generated Text using Style Representations
- When False Positive is Intolerant: End-to-End Optimization with Low FPR for Multipartite Ranking
- The MAJORANA Project
- $$\alpha $$-event characterization and rejection in point-contact HPGe detectors
- First Results from the DEAP-3600 Dark Matter Search with Argon at SNOLAB
- Isolating Unisolated Upsilons with Anomaly Detection in CMS Open Data
- Dark Matter Search Results from <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline"><mml:mrow><mml:mn>1.54</mml:mn><mml:mtext> </mml:mtext><mml:mtext> </mml:mtext><mml:mi>Tonne</mml:mi><mml:mo>·</mml:mo><mml:mi>Year</mml:mi></mml:mrow></mml:math> Exposure of PandaX-4T
- Machine learning-powered data cleaning for LEGEND: a semi-supervised approach using affinity propagation and support vector machines
- Theoretically Grounded Framework for LLM Watermarking: A Distribution-Adaptive Approach
- Every Bit, Everywhere, All at Once: A Binomial Multibit LLM Watermark
- A Survey on Detection of LLMs-Generated Content