ATLAS: A System for PDF-centric Human Interaction Data Collection

Alexa F. Siu, Zichao Wang, Joshua Hoeflich, Naman Kapasi, Ani Nenkova, Tong SunNAACL 2024 · OpenReview

Abstract

The Portable Document Format (PDF) is a popular format for distributing digital documents. Datasets on PDF reading behaviors and interactions remain limited due to the challenges of instrumenting PDF readers for these data collection tasks. We present ATLAS, a data collection tool designed to better support researchers in collecting rich PDF-centric datasets from users. ATLAS supports researchers in programmatically creating a user interface for data collection that is ready to share with annotators. It includes a toolkit and an extensible schema to easily customize the data collection tasks for a variety of purposes, allowing collection of PDF annotations (e.g., highlights, drawings) as well as reading behavior analytics (e.g., page scroll, text selections). We open-source ATLAS to support future research efforts and review use cases of ATLAS that showcase our system’s broad applicability.

Introduction: Bridging the "PDF Gap"

Despite the ubiquity of PDFs—with an estimated 2.5 trillion in existence—there is a significant scarcity of datasets capturing how humans actually interact with them. Most existing AI training data for documents is "content-centric," focusing on labeling static elements like tables or text blocks. However, building AI that truly collaborates with humans requires "human-centric" data: understanding how people scroll, where they linger, and which sections they highlight when synthesizing information.

The technical difficulty of instrumenting PDF readers to capture fine-grained behaviors (zoom levels, search queries, mouse patterns) often forces researchers to convert PDFs to HTML. This process destroys the original layout and visual context, which are critical to the PDF's utility. ATLAS addresses this "PDF Gap" by providing a full-stack solution for collecting rich interaction data.

The ATLAS System Architecture

ATLAS provides a programmatic way to generate data collection interfaces, enabling the creation of high-quality datasets for training and evaluating document-grounded AI models. The system follows a streamlined three-step workflow from project definition to data collection.

The ATLAS workflow: (A) Programmatic project creation via JSON, (B) Management of documents and studies, and (C) Deployment for response collection.

Programmatic Toolkit and Native Interface

Researchers define their data collection tasks using a Python toolkit that generates a JSON schema. This allows for "zero-code" UI generation—researchers do not need to write web code to create complex annotation tasks. By using the Adobe PDF Embed API, ATLAS provides a familiar, high-fidelity reading experience. This ensures that the data collected reflects real-world behavior rather than artifacts of a clunky interface. The system supports a wide variety of tasks, including bounding box annotations, free-form drawings, and standard survey elements like radio buttons and text entry.

User Interaction and Data Logging

The core of ATLAS is its extensible annotation interface. It is designed to be intuitive for users while capturing granular data for researchers. Beyond saving final answers, ATLAS logs 14 types of reading behavior events with timestamps, including scrolls, text selections, and mouse-over patterns.

The ATLAS user interface, featuring integrated PDF viewing, annotation tools, and customizable task panels.

The interface is divided into several key components:

  • Page Instructions: Clear guidance for the annotator.
  • PDF Document Viewer: A high-fidelity viewer that preserves document layout.
  • Document Annotation: Tools for highlighting and marking the source text directly.
  • Page Tasks: A sidebar for structured questions, ratings, and data entry.

Use Cases and Impact

ATLAS has already powered several large-scale research efforts:

  • Document-Grounded QA: A dataset of over 20,000 question-answer pairs based on 3,000+ unique PDFs, including human-authored attributions (highlights) within the files.
  • QA Evaluation: Annotators used ATLAS to rate the quality and "hallucination" levels of AI-generated answers by comparing them directly against source PDFs.
  • Digital Reading Behavior: An ongoing study collecting implicit reading patterns from professionals across various industries.

Conclusion

By democratizing data collection through a programmatic, JSON-based UI generation system, ATLAS lowers the barrier for HCI and NLP researchers to run large-scale studies. The system emphasizes that the process of annotation—how long it took, what parts were read first—is as important as the final label. Open-sourcing ATLAS provides a standardized way for the community to share and replicate document-interaction studies, ultimately leading to more human-aligned AI models.

Summary updated

Abstract

The Portable Document Format (PDF) is a popular format for distributing digital documents. Datasets on PDF reading behaviors and interactions remain limited due to the challenges of instrumenting PDF readers for these data collection tasks. We present ATLAS, a data collection tool designed to better support researchers in collecting rich PDF-centric datasets from users. ATLAS supports researchers in programmatically creating a user interface for data collection that is ready to share with annotators. It includes a toolkit and an extensible schema to easily customize the data collection tasks for a variety of purposes, allowing collection of PDF annotations (e.g., highlights, drawings) as well as reading behavior analytics (e.g., page scroll, text selections). We open-source ATLAS1 to support future research efforts and review use cases of ATLAS that showcase our system’s broad applicability.
BibTeX
@inproceedings{siu2024atlas,
  author = {Alexa F. Siu and Zichao Wang and Joshua Hoeflich and Naman Kapasi and Ani Nenkova and Tong Sun},
  title = {{ATLAS: A System for PDF-centric Human Interaction Data Collection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=12uTJgRMgL},
}

Loading related content...

Figure