WikiRef: Wikilinks as a route to recommending appropriate references for scientific Wikipedia pages

Abhik Jana, Pranjal Kanojiya, Pawan Goyal, Animesh Mukherjee

Introduction

Wikipedia serves as a primary gateway to scientific knowledge for millions. However, maintaining the integrity of its "References" section remains a significant challenge. While the text of articles evolves rapidly, the citations often lag behind, leading to pages that are informative but lack robust, up-to-date sourcing. This paper introduces WikiRef, an automated system designed to bolster Wikipedia's reliability by recommending high-quality scientific references through the analysis of internal wikilinks.

The Reference Gap

The motivation for WikiRef stems from a fundamental imbalance in how Wikipedia is edited. While users frequently update article text and add internal wikilinks to navigate between topics, they rarely add or update citations. Research indicates that while approximately 32% of edits involve wikilinks, a mere 1% focus on references. This "reference gap" suggests that adding citations is perceived as a significantly more difficult task for the community.

Edit statistics for sample Wikipedia pages showing the low frequency of reference edits compared to wikilink edits.

To bridge this gap, the authors propose leveraging the existing network of wikilinks. Since wikilinks are updated more frequently and reflect topical connections, they can serve as a "fresher" signal for identifying relevant source material.

The WikiRef Framework

WikiRef operates through a two-step pipeline:

Not every link in an article points to a page that can provide useful scientific references. For instance, a page on "Machine Learning" might link to "Linear Algebra" (a rich source of foundational papers) but also to "Silicon Valley" (unlikely to contain relevant academic citations). WikiRef employs a Random Forest classifier to distinguish between these "potential sources" and irrelevant links. Features used for this classification include Tf-idf similarity of article summaries, link overlap (comparing inlinks and outlinks), and semantic vector representations generated via bi-directional LSTMs.

2. Reference Recommendation and Ranking

Once a relevant source page is identified, the system must determine which specific references from that page should be "inherited" by the target article. This is framed as a learning-to-rank problem using SVMRank. The model prioritizes references based on:

  • Citation Context: The similarity between the sentence containing the reference in the source article and the sentence containing the wikilink in the target article.
  • Title Similarity: Matching the reference title against the context of the target page.

Evaluation and Results

The researchers tested WikiRef on extensive datasets from Computer Science (CS) and Physics (PH), using existing Wikipedia references as the ground truth.

Performance metrics (Precision, Recall, F-Measure) for WikiRef at different values of k.

WikiRef consistently outperformed traditional baselines. In the Computer Science domain, the system achieved a Precision@1 of 0.44, marking a 10-15% improvement over previous methods. Human evaluation further validated these results; in a study with 10 experts, WikiRef's suggestions received a relevance score of 0.63, meaning that roughly 60% of the "new" references suggested by the system were deemed highly appropriate by subject matter experts.

Conclusion

WikiRef demonstrates that Wikipedia's internal link structure is a powerful, underutilized resource for automated curation. By treating wikilinks as proxies for topical relevance, the system can effectively "crowdsource" references from well-sourced articles to those in need. Future work aims to implement WikiRef as a real-time tool for Wikipedia editors and expand its application to broader fields like medicine and biology.

Summary updated

BibTeX
@inproceedings{jana2018wikiref,
  author = {Abhik Jana and Pranjal Kanojiya and Pawan Goyal and Animesh Mukherjee},
  title = {{WikiRef: Wikilinks as a route to recommending appropriate references for scientific Wikipedia pages}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=ecFAmoB9Jc1},
}
Figure