TL;DR
As Large Language Models (LLMs) become capable of generating sophisticated scientific text, traditional plagiarism detection is failing. This paper introduces two new datasets of "generative plagiarism" designed to mimic malicious actors and evaluates how well modern retrieval systems can trace AI-synthesized papers back to their original human-authored sources.
What Problem Does This Paper Address?
We have entered a "post-LLM" academic landscape. Modern AI can now polish, expand, or even generate entire scientific papers that occasionally pass peer review. While legitimate use for editing is common, these tools also allow "malicious actors" to synthesize new papers from existing ones with minimal effort, effectively laundering uncredited ideas through heavy paraphrasing.
Traditional detection systems rely on lexical overlap—finding identical strings of words. However, LLMs are masters of paraphrasing. The challenge addressed here is Semantic Document Retrieval: given a suspicious, AI-generated document, can we find the specific original papers used to create it, even if not a single sentence is identical?
Methods
The authors developed a "malicious actor" pipeline to create a benchmark for the community. They focused on two generation strategies:
- Sequential Generation (The Chain): To maintain coherence and manage context limits, the system merges papers one by one. It merges abstracts first, compresses them, and then generates full sections (Introduction, Methodology, etc.). This "left-fold" design ensures the final product is a unified narrative rather than a disjointed survey.
- One-Shot Generation: The model is fed multiple source papers in a single prompt and asked to synthesize them immediately. This is more realistic but prone to "hallucinations," where the AI invents facts not found in the sources.
To ensure the dataset was high-quality, the authors used a "committee of LLMs" (including Gemini 3 Flash and Claude 4.6) to score the generated papers on criteria like "Multi-Source Visibility" and "Coherence." Only the most "realistic yet detectable" samples were kept.
Datasets
The task utilized a source corpus of 100,000 scientific articles from ar5iv (an HTML5 mirror of arXiv). From this, they built:
- The Sequential Dataset: 200 plagiarized documents (the primary evaluation set).
- The One-Shot Dataset: 543 plagiarized documents (used for internal testing and code evaluation).
- Source Tuples: Each plagiarized paper was derived from a 4-tuple of sources selected based on semantic similarity (60th to 95th percentile) to ensure they were related but not near-duplicates.
Key Insights and Ideas
- The Resilience of BM25: Surprisingly, the classic BM25 (a keyword-based algorithm) outperformed several modern embedding-based baselines. Why? Scientific terminology (e.g., "contextual vision transformers") is highly specific. Even when an LLM paraphrases a sentence, these "hard" technical terms often remain, acting as a lexical fingerprint.
- Granularity Matters: The winning team (JLP) found that breaking a document down into single-sentence spans and querying them individually was more effective than querying with the whole document. This helps isolate specific "clues" that might be buried in a sea of AI-generated filler.
- Numbers are Hard Facts: LLMs tend to treat numerical values as immutable. A "Numerical Matching" baseline showed that simply looking for shared constants between papers can provide a weak but useful signal for detection.
- The Hallucination Hurdle: In the One-Shot dataset, models hallucinated significantly more. This raises a philosophical question for the field: if an AI "hallucinates" a new idea inspired by a source, is it still plagiarism, or has it become a new (albeit fake) work?
Key Results
The competition saw 26 submissions from six teams.
- Top Performer: Team JLP took the lead using a hybrid approach: BM25 + GTE RRF (Reciprocal Rank Fusion). They achieved an impressive nDCG@10 of 0.799 and a Mean Reciprocal Rank (MRR) of 0.976 on the sequential dataset.
- Baselines: The BM25 baseline (0.553 nDCG@10) proved to be a "formidable floor," outperforming the LinQ embedding baseline (0.391).
- Model Consistency: GPT-OSS and Qwen 3.5 were generally easier to "trace" back to sources than Llama 3.1 or DeepSeek-R1, suggesting that different generator models leave different levels of "source evidence" in their wake.
Limitations and Future Work
The authors acknowledge that the One-Shot dataset is a double-edged sword. While it represents a more realistic "lazy" plagiarism attempt, the high rate of hallucinations makes it difficult to guarantee that all sources are actually traceable.
Future work will likely focus on:
- Refining "Traceability": Developing better metrics for when a source's influence is too diluted to be considered plagiarism.
- Handling Hallucinations: Improving retrieval systems to ignore AI-invented content and focus strictly on the "anchors" of real source material.
- Efficiency: As the search corpus grows from 100k to millions of papers, the computational cost of sentence-level semantic search will become a major bottleneck.
Summary updated