Introduction
Persistent Identifiers (PIDs), most notably Digital Object Identifiers (DOIs), serve as the foundational infrastructure for digital scholarship. By providing a permanent link to a resource, they ensure that citations remain functional even as publisher websites change. However, the efficacy of PIDs depends on their consistent use. Researchers at the University of Edinburgh recently investigated the "leakage" problem—where users bypass the PID (the "original" or "actionable" form) in favor of the direct, unstable "locating" URL—and evaluated whether the massive Common Crawl dataset is a viable tool for monitoring this phenomenon.
Methodology
The researchers conducted two primary studies using Common Crawl data from April 2014 through April 2017.
- Study 1 (Broad Sweep): A comprehensive analysis of all links within the HTML bodies of pages from the 2014 and 2017 crawls. This involved processing approximately 36TB of compressed WAT files (metadata records).
- Study 2 (Longitudinal Pilot): A more granular look at the 2014–2017 period, specifically targeting
doi:schemes within HTML<head>elements, such as<link>and<meta>tags.
To handle the scale of the data, the team transitioned from an ad-hoc cluster of student lab machines to a robust Microsoft Azure environment, achieving an eight-fold increase in processing speed. Bloom filters were employed to efficiently compare PIDs discovered in 2014 against links found in the 2017 dataset.
Dataset Quality and Common Crawl Evolution
A significant portion of the research focused on the reliability of Common Crawl (CC) as a longitudinal research tool. The authors noted that while CC is the only large-scale public web archive available, its internal consistency has varied significantly over time.
As shown in the table above, the quality of the crawl improved dramatically between 2014 and 2017. In 2014, the dataset suffered from a massive duplication problem, with 34.9% of URIs being duplicates. By 2017, this figure dropped to just 1.2%. Despite these improvements, a major hurdle remains: temporal overlap. Only about 7% of the URIs crawled in 2014 were revisited in 2017, making it difficult to track the evolution of specific web pages over long periods.
Key Findings
The Growth of DOIs
The study revealed a robust expansion of the PID ecosystem. The number of distinct DOIs found in page bodies grew from 5.3 million in 2014 to 12.6 million in 2017—a 2.5x increase. Furthermore, the use of DOIs within page metadata (e.g., Dublin Core or EPrints standards) showed a steady upward trend.
The Leakage Problem
The researchers confirmed that "locating form" leakage is a measurable issue. Approximately 8% of the actionable PIDs identified in 2014 were found in the 2017 crawl only as their unstable "locating" URLs. This suggests that users frequently copy-paste URLs from their browser's address bar after a DOI has resolved, inadvertently stripping away the persistence the PID was designed to provide.
Format and Encoding Hurdles
The study highlighted that PIDs are most prevalent in scholarly PDFs, a format that Common Crawl largely ignores in favor of HTML. Consequently, CC-based studies likely represent a significant undercount of total PID usage. Additionally, the researchers encountered frequent character encoding errors and formatting inconsistencies in the WAT files, necessitating extensive manual data cleaning.
Conclusion and Future Directions
While Common Crawl provides an unprecedented window into the web's evolution, it is not a "plug-and-play" resource for longitudinal PID studies. The authors suggest that future research should account for the "PDF gap" and the low temporal overlap between crawls. They recommend that researchers process every monthly crawl over a year-long period to mitigate the variability of individual snapshots. Ultimately, while the PID ecosystem is growing, browser behavior and user habits continue to pose a threat to the long-term stability of digital citations.
Summary updated