On augmenting the references section with a citation network visualization

Putra Manggala, Tigran Atoyan, Gracia Samosir, Jan Varsava, Johannes RufRethinking ML Papers @ ICLR 2021 · OpenReview

TL;DR

This paper tackles the challenge of navigating the ever-growing volume of scientific literature by proposing an enhanced reference section. They introduce an embedded citation network visualization, enriched with author-provided comments, to help researchers quickly understand the relationships between papers and discover relevant work more efficiently.

What Problem Does This Paper Address?

In fields like Machine Learning, the sheer volume of new publications makes it incredibly difficult for researchers to keep up, discover relevant prior work, and understand the landscape of a research area. Traditional reference sections, which are essentially flat lists, don't adequately support these needs. Researchers often struggle with questions like: "What are the foundational papers in this field?", "Which papers are most accessible for understanding a concept?", "What are the key papers that influenced this work?", and "What are other researchers doing in this area?". Existing tools for navigating citation networks are often external and cumbersome, leading to "tool fatigue" and a lack of integration into the reading workflow.

Methods

The core of the proposed solution is an embedded citation network visualization. This visualization, powered by tools like D3.js, represents articles as nodes and citations as edges. Crucially, these edges are labeled with citation types, categorizing the nature of the citation (e.g., "Background," "Technical Basis," "Comparison"). The authors draw upon existing research for these citation type schemas, referencing work by Moravcsik & Murugesan (1975), Spiegel-Rosing (1977), and Dong & Schäfer (2011).

Beyond just visualization, they propose augmenting citation types with author comments. This allows authors to provide free-form, context-specific notes for each reference, adding a layer of expressiveness that generic citation types might miss, especially when many citations fall into the same category. The idea is to embed this visualization directly within or link it from the article's reference section to combat "tool fatigue."

Datasets

Not discussed in the paper. The paper focuses on a conceptual proposal and a demonstration using the paper itself as the central node in a citation network.

Key Insights and Ideas

  • Citation types are crucial for understanding relationships: Simply knowing that a paper is cited isn't enough; understanding why it's cited (e.g., for background, as a method, for comparison) significantly aids comprehension.
  • "Tool Fatigue" is a real barrier: Researchers are less likely to use external tools for citation navigation if they are not seamlessly integrated into their reading workflow. Embedding or linking visualizations directly within articles is key.
  • Author comments add invaluable nuance: While citation types provide structure, author comments offer a way to convey specific insights that generic labels can't capture, especially for complex or nuanced relationships.
  • Accessibility is paramount: The visualization tool is designed with accessibility in mind, adhering to WCAG 2.1 Level AA guidelines, and offering features like adjustable contrast and zoom.

Key Results

The paper presents a prototype implementation of the citation network visualization for the article itself. They demonstrate how this visualization, combined with citation types, can help answer common research questions (Q1-Q5) identified through their user interviews. While quantitative results are not presented, the qualitative findings from their user interviews highlight the problems this solution aims to address and the potential benefits of their proposed approach in improving research efficiency and inclusivity.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Choice of Citation Schema: The paper discusses the trade-offs between different citation type schemas and suggests that a standardized, expressive schema (like T2 from Teufel et al.) would be beneficial.
  • Who Classifies Citations: The paper explores whether authors, third-party annotators, or automated methods should classify citations, suggesting that authors including these visualizations themselves is a promising starting point.
  • Visualization Provision: They question the best way to present the visualization (e.g., URL, QR code, embedded) and how to make it easy for authors to include.
  • Higher-Order Neighborhoods: While the current focus is on the immediate neighborhood, they recognize that answering some research questions might require exploring further out in the citation network.
  • Evaluating Effectiveness: They propose using web measurement techniques to evaluate the effectiveness of different visualization aspects.
  • Measuring Comment Efficacy: Metrics are needed to understand how often authors provide comments and how often users engage with them.
  • Increasing Author Comment Input: Making the comment workflow simple and providing incentives are key to encouraging authors to contribute.
  • Website Accessibility: The current website is a prototype with minimal features, and future accessibility improvements will depend on user feedback and uptake.

Summary updated

Abstract

Researchers often find relevant articles by looking at the references section. We conducted user interviews with researchers about their workflow and their needs when carrying out literature research. Based on this study, we identify a set of problems encountered by researchers. We then propose to embed classified citation networks into articles as a solution, and to complement this graph with optional comments about references by authors. We demonstrate this idea by implementing it for this article. We argue that our solution helps increase inclusivity and improves the efficiency of reading scientific articles.
BibTeX
@inproceedings{manggala2021augmenting,
  author = {Putra Manggala and Tigran Atoyan and Gracia Samosir and Jan Varsava and Johannes Ruf},
  title = {{On augmenting the references section with a citation network visualization}},
  booktitle = {Rethinking ML Papers @ ICLR},
  year = {2021},
  url = {https://openreview.net/forum?id=u3c4pW2r5Gr},
}
Figure