Abstract
Machine-Generated Text (MGT) detection aims to identify a piece of text as machine or human written. Prior work has primarily formulated MGT detection as a binary classification task over an entire document, with limited work exploring cases where only part of a document is machine generated. This paper provides the first in-depth study of MGT that localizes the portions of a document that were machine generated. Thus, if a bad actor were to change a key portion of a news article to spread misinformation, whole document MGT detection may fail since the vast majority is human written, but our approach can succeed due to its granular approach. A key challenge in our MGT localization task is that short spans of text, e.g., a single sentence, provides little information indicating if it is machine generated due to its short length. To address this, we leverage contextual information, where we predict whether multiple sentences are machine or human written at once. This enables our approach to identify changes in style or content to boost performance. A gain of 4-13% mean Average Precision (mAP) over prior work demonstrates the effectiveness of approach on five diverse datasets: GoodNews, VisualNews, WikiText, Essay, and WP.
The Challenge of Granular Detection
The rapid advancement of Large Language Models (LLMs) has made it trivial to generate convincing text. While "whole-document" Machine-Generated Text (MGT) detection is a well-studied field, it has a major blind spot: localization. In real-world scenarios, misinformation often isn't a 100% AI-generated essay; it’s a legitimate news article where a few key sentences have been surgically modified by an LLM to change the narrative.
Current detectors struggle with localization for two main reasons. First, they aren't designed to provide sentence-level labels. Second, reliable detection usually requires a significant amount of text (at least 50 tokens); a single sentence often lacks the statistical "fingerprint" needed for confident classification.
AdaLoc: Context-Aware Localization
To address these challenges, we propose AdaLoc (Adaptor for localization), a method that shifts the focus from individual sentences to contextual chunks. Instead of training a massive new model, AdaLoc uses a frozen, pre-trained MGT detector (such as RoBERTa) as a feature extractor. A lightweight "Localization Adaptor"—consisting of two fully connected layers—is added on top.
The core innovation is Contextual Prediction. Rather than evaluating one sentence in isolation, AdaLoc processes a "chunk" of $m$ sentences (typically 3) and predicts a binary vector of labels for all of them simultaneously. This allows the model to identify stylistic "seams" or factual inconsistencies between neighboring sentences. To further boost accuracy, a sliding window approach is used: since a sentence might appear in multiple overlapping chunks, the final label is determined by a majority vote of all predictions for that specific sentence.
Experimental Results
We evaluated AdaLoc across five diverse domains: news (GoodNews & VisualNews), Wikipedia (WikiText), student essays (Essay), and creative writing (WP). Since no standard benchmark existed for mixed human-AI articles, we created a data pipeline to substitute sections of existing datasets with LLM-generated text from models like GPT-2, GPT-Neo, and OPT.
As shown in the results, AdaLoc achieved a 4–13% mAP improvement over baseline adaptations of state-of-the-art detectors. On the GoodNews dataset, the Roberta-L + AdaLoc configuration reached 78.49 mAP, significantly outperforming the base RoBERTa-Large model (50.67 mAP) and commercial tools like GPT-Zero, which often suffer from low recall on sentence-level analysis in mixed documents.
Discussion and Future Work
Our findings highlight that while a single AI-generated sentence might appear human, its relationship to surrounding human text often reveals its origin. Interestingly, AdaLoc demonstrated strong zero-shot generalization; when trained on news articles, it performed remarkably well on Wikipedia and other domains without further tuning, suggesting that the "signatures" of AI-human transitions are somewhat universal in formal writing.
However, challenges remain. Performance drops when facing highly advanced models like GPT-4, where stylistic seams are harder to detect. Additionally, the model currently focuses on direct generation; identifying paraphrased content or defending against adversarial actors who intentionally "smooth" transitions remains an area for future research.
Summary updated