Abstract
Automatic plagiarism detection has long been hindered by inconsistent evaluation metrics and small, proprietary datasets. This paper introduces a comprehensive evaluation framework featuring tailored performance metrics, including a novel "granularity" measure, and the PAN-PC-10 corpus—a massive dataset of over 68,000 artificial and crowdsourced plagiarism cases. This standardized testbed enables objective, large-scale comparison of plagiarism detection algorithms.
The Need for Standardization
Historically, evaluating plagiarism detection algorithms lacked rigor. A survey of over 200 papers revealed that 80% of text plagiarism researchers tested their methods on "homemade" datasets rather than established benchmarks. Furthermore, many of these datasets were small, with 68% containing fewer than 1,000 documents.
Standard metrics like precision and recall were often inconsistently applied to the specific task of identifying overlapping text chunks. This fragmentation made it impossible to objectively track the state-of-the-art or compare different systems.
A New Evaluation Framework
The authors address these challenges through two primary contributions: refined metrics and scalable data generation.
Tailored Performance Metrics
The framework formalizes character-level micro- and macro-averaged precision and recall. Crucially, it introduces detection granularity, a metric that penalizes algorithms for reporting a single contiguous plagiarized passage as multiple fragmented detections. These metrics are combined into a single overall score using an $F_1$-measure adjusted by the logarithm of the granularity.
Scalable Data Generation
To overcome the legal and ethical difficulties of publishing real plagiarism cases, the authors developed two methods for generating high-quality test data:
- Simulated Plagiarism: Amazon Mechanical Turk (AMT) workers were tasked with rewriting source passages to preserve meaning while changing wording, mimicking the behavior of a student copying homework.
- Artificial Plagiarism: Automated obfuscation strategies, such as word shuffling, synonym swapping, and POS-preserving transformations, were used to generate cases at scale.
The PAN-PC-10 Corpus
The resulting PAN-PC-10 corpus contains 27,073 documents and 68,558 plagiarism cases, including both external and intrinsic plagiarism scenarios. To validate the realism of this data, the authors compared it against existing reference corpora: Clough09 (simulated plagiarism) and METER (journalistic text reuse).
Validation tests using n-gram Vector Space Models (VSM) showed that the statistical distributions of the automated "artificial" plagiarism were remarkably similar to human-written "simulated" plagiarism and real-world text reuse. This confirms that algorithmic obfuscation is a valid, cost-effective substitute for human rewriting in building large-scale training corpora.
Impact and Future Directions
The framework has already proven its utility as the foundation for the PAN'09 and PAN'10 benchmarking workshops, which evaluated 31 different plagiarism detectors. By providing a massive, standardized dataset and meaningful metrics like granularity, the field can finally move toward objective performance tracking.
Future work involves refining automated paraphrasing methods to improve readability and exploring the true statistical distribution of parameters in real-world plagiarism to further enhance the realism of artificial corpora.
Summary updated