Abstract
The proliferation of online digital libraries offers users a great opportunity to search their desired literatures on the Web. Cross-library search applications can help users search more literature information from multiple digital libraries. Duplicate literature detection is always a necessary step when merging the search results from multiple digital libraries due to heterogeneity and autonomy. To this end, this paper proposes a holistic solution which includes achieving automatic training set generation, holistic attribute mapping, and weight of attribute training. Experiments on real digital libraries show that the proposed solution is highly effective.
Introduction
When searching for research papers across multiple digital libraries—such as IEEE Xplore, ACM Digital Library, or institutional repositories—users often encounter redundant results. However, these duplicates are rarely identical: one library might list a full author list, while another uses "et al."; one might include a DOI, while another omits it. Identifying these "fuzzy" duplicates is critical for providing a clean, unified search experience.
Traditional duplicate detection solutions face three major hurdles:
- Scalability: Manually creating training sets for thousands of libraries is impossible.
- Complexity: Traditional methods attempt to map every library to every other library ($O(n^2)$ implementations), which does not scale as the number of sources grows.
- Heterogeneity: Different libraries use different schemas (attribute names and formats), making direct comparisons difficult.
Methodology
The authors propose a three-stage "holistic" pipeline designed to automate the detection process and handle library heterogeneity efficiently.
1. Automatic Training Set Extraction
Instead of manual labeling, the system uses a "naïve" similarity check (tf-idf) on all record pairs. By ranking these pairs and applying least-squares curve fitting and curvature computation, the system identifies a "head" segment of high-confidence matches. This creates a functional training set automatically, even if the initial data is "noisy."
2. Holistic Attribute Mapping
To avoid the $O(n^2)$ mapping problem, the framework uses a "bridge strategy." Libraries are ranked by their number of attributes and mapped sequentially. A central, attribute-rich library acts as a bridge to map other libraries, drastically reducing the number of complex mappings required to integrate new sources.
3. IBITA: Weight Learning
The core algorithm, Inequalities Based Iterative Training Approach (IBITA), learns attribute weights (e.g., determining that "Title" is more significant than "Publisher"). It treats duplicate detection as an optimization problem based on inequalities, iteratively adjusting weights to ensure that the similarity of a "matched" pair is consistently higher than any "non-matched" pair.
Experimental Evaluation
The system was evaluated using 10 popular university digital libraries, specifically targeting "paper" and "book" literature domains. The authors submitted six independent queries to each library pair to build their experimental data set.
Key Results
- High Precision: The total precision consistently hovered between 82% and 97% across different library pairs.
- Low Manual Effort: The "Uncertainty" metric—representing cases where the system required human intervention—was remarkably low, generally under 3%.
- Efficiency: The iterative IBITA algorithm converged quickly, typically requiring fewer than two iterations on average to find stable weights.
- Weight Findings: In the paper domain, "Title" (avg weight 0.92) and "ISSN" (1.0) were the strongest predictors of a match, while "Publisher" (0.31) was the weakest.
Insights and Conclusion
A key insight of this research is that attribute importance is domain-dependent rather than source-dependent. In the "paper" domain, "Title" is universally more important than "Author" for identification, regardless of which library the data comes from. By leveraging this and the "bridge" mapping strategy, the framework can integrate new, sparser libraries without needing to compare them to every existing source.
While the method is robust, it does rely on some degree of overlap between digital libraries to extract a meaningful automatic training set. Future work will focus on optimizing the end-to-end pipeline for real-time web-scale applications and expanding the framework to other types of data entities.
Summary updated