PAWLS: PDF Annotation With Labels and Structure

Mark Neumann, Zejiang Shen, Sam Skjonsberg

TL;DR

Annotating information within PDF documents is notoriously difficult due to their visual-first nature. This paper introduces PAWLS, a novel web-based annotation tool specifically designed for PDFs, enabling mixed-mode annotation of both text and visual elements. PAWLS aims to streamline the creation of rich, structured datasets for multimodal machine learning tasks.

What Problem Does This Paper Address?

Natural Language Processing (NLP) research heavily relies on high-quality annotated datasets for training and evaluating models. While formats like XML and HTML are machine-readable, the Portable Document Format (PDF) remains a dominant format for distributing documents, especially in scientific publishing, law, and government. PDFs are designed for visual fidelity, not machine readability, meaning they lack inherent structural markup that would allow NLP practitioners to easily extract and annotate information. Extracting text from PDFs often results in a jumbled stream without crucial context like paragraph boundaries, figure placements, or headers/footers. Existing annotation tools, like BRAT, primarily focus on plain text, which fails to capture the rich visual context present in PDFs. This gap hinders the development of NLP models that can understand and process information embedded within the visual structure of documents.

Methods

PAWLS tackles the PDF annotation challenge with several key design choices:

  • PDF-Native Annotation: The tool renders the actual PDF document directly in the browser, allowing annotators to work within their familiar visual context. Annotations are stored relative to the original PDF's absolute dimensions but are automatically resized to fit the rendered canvas, ensuring consistency.
  • Visual Token Selection and Box Snapping: PAWLS pre-processes PDFs to extract token-level bounding box information. When annotators draw bounding boxes, the tool highlights included tokens and "snaps" the box to a normalized boundary around these tokens. This feature ensures consistent and uniform annotations without requiring extra effort from annotators.
  • Mixed-Mode Annotation: PAWLS supports both span-based textual annotations and freeform, non-textual bounding boxes. This allows for annotating not only text but also visual elements like figures, tables, and specific regions within them.
  • N-ary Relational Annotations: Beyond simple bounding boxes, PAWLS supports defining relationships between annotated regions. This is crucial for capturing complex structures, such as linking figure captions to their corresponding figures or relating text descriptions to specific table columns.
  • Command Line Interface (CLI): A CLI is provided for project administration, including task assignment, progress monitoring, quality control (inter-annotator agreement), and exporting annotations in various formats.
  • Annotation Pre-population: The CLI supports pre-populating annotations by feeding model predictions, enabling a "model-in-the-loop" workflow where annotators correct model outputs.

Datasets

The paper mentions that PAWLS is currently in use for a "PDF Layout Parsing project for academic papers." This project has resulted in an initial set of gold-standard annotations comprising 80 PDF pages with 2558 densely annotated bounding boxes across 20 categories, collected from 3 annotators. The paper does not detail the creation or release of a new, large-scale benchmark dataset beyond this initial study.

Key Insights and Ideas

  • Bridging the Visual-Textual Divide: The core insight is that to effectively annotate PDFs, the annotation tool must embrace the visual nature of the document rather than abstracting it away into plain text. PAWLS's PDF-native rendering and box-snapping mechanism are key to this.
  • Annotation Consistency through Automation: The "box snapping" feature is a clever way to enforce annotation consistency without burdening annotators, directly addressing a common challenge in annotation projects.
  • Enabling Multimodal Datasets: By supporting both text and freeform bounding boxes, PAWLS is designed to facilitate the creation of datasets that are inherently multimodal, which is increasingly important for advanced AI models.
  • Context is King: The emphasis on annotating directly on the PDF canvas highlights the importance of preserving the original document context, which is often lost in text-only annotation tools.

Key Results

The paper reports on an initial annotation effort for a PDF Layout Parsing project. Inter-annotator agreement scores were measured:

  • Textual Labels: Achieved high agreement, with token accuracy scores around 94.43% and 86.58% for textual labels.
  • Non-Textual Labels: Agreement for non-textual labels (like figures and tables), measured using Average Precision (AP), was also reported, with scores around 88.69% and 84.79%.

These results suggest that PAWLS, with its design choices, can lead to reliable and consistent annotations even for complex, mixed-mode tasks.

Limitations and Future Work

  • Limitations:
    • The paper focuses on the tool's design and initial evaluation, not on a comprehensive benchmark dataset.
  • While PAWLS supports various pre-processors (GROBID, PdfPlumber, Tesseract), the quality of token extraction and OCR can still impact annotation accuracy.
  • The current implementation relies on browser-based rendering, which might have performance limitations for extremely large or complex PDFs.
  • Future Work:
    • PAWLS plans to incorporate higher-level PDF structure elements (document titles, paragraphs, section headings) to enable more sophisticated annotation modes.
  • The tool aims to support active learning-based annotation suggestions through models running as a service, further enhancing efficiency.
  • Extensions for more advanced annotation types and configurations are envisioned.

Summary updated

BibTeX
@inproceedings{neumann2021pawls,
  author = {Mark Neumann and Zejiang Shen and Sam Skjonsberg},
  title = {{PAWLS: PDF Annotation With Labels and Structure}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=PFpqU8FltcL},
}

Loading related content...

Figure