TOBY: A Tool for Exploring Data in Academic Survey Papers

Tathagata Chakraborti, Jungkoo Kang, Christian Muise, Sarath Sreedharan, Michael E. Walker, Daniel Szafir, Tom WilliamsarXiv 2023 · arXiv · OpenReview

Abstract

This paper describes TOBY, a visualization tool that helps a user explore the contents of an academic survey paper. The visualization consists of four components: a hierarchical view of taxonomic data in the survey, a document similarity view in the space of taxonomic classes, a network view of citations, and a new paper recommendation tool. In this paper, we will discuss these features in the context of three separate deployments of the tool.

Introduction

Literature surveys are fundamental to academic research, providing essential roadmaps for both novices and experts. However, these surveys are typically published as static PDFs, effectively trapping rich structured data—such as taxonomies, citation networks, and feature comparisons—within immutable tables. Furthermore, traditional visualization tools are retrospective, focusing on what has been published rather than identifying future research opportunities.

TOBY (Tool for Exploring Data in Academic Survey Papers) addresses these limitations by transforming static surveys into interactive, searchable databases. Beyond simple data exploration, TOBY introduces a generative approach to research discovery by identifying "gaps" in the literature through logical SAT-solving, suggesting specific, unexplored research projects.

Methods: The TOBY Pipeline

TOBY is designed as a modular system comprising a Python-based backend and a React-based frontend. The core of the system is the "TOBY Compiler," which orchestrates the transformation of raw survey data into a deployable web portal.

The TOBY system architecture, illustrating the workflow from survey author configuration to the interactive reader interface.

The workflow is divided into two primary steps:

  1. Configuration: The survey author provides structured data, including a spreadsheet (XLSX/CSV) of papers tagged with taxonomic features, a YAML or JSON configuration file, and the paper PDFs.
  2. Deployment: The compiler processes these inputs through the backend—utilizing tools like Flask, DSHARP, and pdfplumber—to generate build files for the frontend, which is built using React and the Carbon Design System.

Data Ingestion and Configuration

To enable the interactive features, authors must map their survey's taxonomy to the tool's logic. This is achieved through a configuration file that defines how the spreadsheet columns correspond to paper metadata (title, abstract, authors, etc.) and taxonomic categories.

An example YAML configuration file used by the TOBY compiler to map spreadsheet data to the visualization components.

Core Visualization Components

TOBY provides four primary views to the end-user:

  • Hierarchy View: A nested visualization of the field’s taxonomy, allowing users to drill down into specific sub-domains.
  • Affinity View: This view maps papers in a conceptual "feature space" using SentenceTransformer embeddings, highlighting papers with similar thematic content.
  • Network View: An automatically generated citation graph. TOBY uses pdfplumber to extract citations directly from paper PDFs, significantly reducing the manual effort required to map academic influence.
  • Insights View (The "Gap Finder"): The most novel contribution of TOBY is its ability to identify research gaps. By converting the taxonomy into a logical theory (SAT-encoding), the tool treats every possible combination of taxonomic features as a potential paper. By applying "semantic constraints" defined by the author (e.g., mutually exclusive features), TOBY identifies valid but currently non-existent "feature profiles" in the literature.

Evaluation and Key Insights

The authors demonstrated TOBY across three AI sub-fields: VAM-HRI (175 papers), MACQ (43 papers), and XAIP (75 papers).

Key Results

  • Efficiency: The automated citation extraction processed over 1,400 pages of text in approximately five minutes.
  • Discovery: In the VAM-HRI dataset, the tool identified over 412 billion theoretically possible (but unwritten) paper configurations. Generating specific, constrained recommendations for new research takes only seconds (1.33s to 5.65s).
  • Surveys as Software: The authors advocate for surveys as living artifacts. TOBY includes a "contribute" button to allow the community to keep the dataset current, moving away from the "static PDF" model.
  • Semantic Validation: The process of defining logical constraints for the taxonomy often helped survey authors identify inconsistencies or "bugs" in their own manual tagging process.

Conclusion and Future Work

While TOBY significantly automates survey exploration, the initial manual effort of tagging papers remains a bottleneck. Future work will focus on more sophisticated hierarchical embeddings and formal user studies to evaluate the real-world impact of the "unwritten paper" recommendations on research productivity. TOBY is open-source and represents a step toward more dynamic, generative academic communication.

Summary updated

Abstract

This paper describes TOBY, a visualization tool that helps a user explore the contents of an academic survey paper. The visualization consists of four components: a hierarchical view of taxonomic data in the survey, a document similarity view in the space of taxonomic classes, a network view of citations, and a new paper recommendation tool. In this paper, we will discuss these features in the context of three separate deployments of the tool.
BibTeX
@misc{chakraborti2023toby,
  author = {Tathagata Chakraborti and Jungkoo Kang and Christian Muise and Sarath Sreedharan and Michael E. Walker and Daniel Szafir and Tom Williams},
  title = {{TOBY: A Tool for Exploring Data in Academic Survey Papers}},
  year = {2023},
  eprint = {2306.10051},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.10051},
}
Figure