Overview of the ICDAR 2013 Competition on Book Structure Extraction

Antoine Doucet, Gabriella Kazai, Sebastian Colutto, Günter Mühlberger2013

TL;DR

This paper summarizes the 2013 ICDAR competition on automatically extracting book structure, specifically focusing on generating hyperlinked tables of contents (ToCs) from digitized books. The competition aimed to advance techniques for making large digital libraries more navigable and searchable, with a key innovation being the use of an external provider for ground truth creation, leading to a larger and more consistent dataset.

What Problem Does This Paper Address?

Digitizing vast collections of books, like those undertaken by the Million Book Project or Google Books, results in full-text content but often lacks rich structural information beyond basic paragraphs. This makes it difficult for users to navigate and search within these digital libraries effectively. While OCR can identify text, it typically doesn't recognize higher-level structures like chapters or sections. The challenge is to automatically derive this structural information, such as creating hyperlinked tables of contents, to enhance user experience and search capabilities in digitized book collections.

Methods

The competition's core task involved participants developing systems to generate hyperlinked tables of contents (ToCs) for a set of 1,000 digitized books. Systems were provided with OCR text in XML format and the corresponding PDF/DjVu files. Participants could use any techniques, leveraging either text or page images. The output was an XML file conforming to a specific Document Type Definition (DTD). Evaluation was performed using two complementary metrics: a title-based measure (comparing ToC entry titles and their hierarchical depth and page number accuracy) and a link-based measure (focusing on equivalent page links).

Datasets

The competition utilized a test set of 1,000 digitized books. This collection was derived from the larger INEX book corpus of over 50,000 books. Notably, 200 of these 1,000 books were specifically selected to not contain a printed ToC, forcing participants to rely on deriving structure from the full book content. The ground truth for this dataset was created using a specialized tool called FEP (Functional Extension Parser) developed by the University of Innsbruck. This process involved an initial automatic structure analysis followed by manual correction. A significant improvement over previous years was the use of an external, third-party company (DIGI-TEXX) for manual correction, ensuring consistency and unbiased evaluation. The ground truth data for 967 books (after removing some non-ToC books and duplicates) was made freely available for research purposes.

Key Insights and Ideas

  • The value of structured data for navigation: The fundamental premise is that structured information, like a ToC, is crucial for making large digital book collections usable beyond simple keyword searches.
  • Ground truth quality is paramount: The paper highlights how the shift to an external, professional ground truth creation process significantly improved the reliability and consistency of the evaluation, a critical factor in any benchmark.
  • Two distinct approaches to ToC extraction: Participants primarily fell into two camps: those focusing on detecting and exploiting printed ToC pages, and those attempting to derive structure from the entire book content. A third "hybrid" approach also emerged.
  • Link-based evaluation is more robust: The link-based evaluation metric showed improved performance compared to title-based metrics, suggesting that it's more forgiving of minor variations in ToC entry titles and better captures the essential functionality of linking to the correct page.
  • The challenge of unsupervised methods: While unsupervised methods for structure extraction were presented, they generally lagged behind supervised or semi-supervised approaches, indicating the difficulty of robustly identifying complex structures without prior knowledge.

Key Results

The 2013 competition saw 6 participating research teams submit runs. The University of Innsbruck achieved the best performance in the link-based evaluation, slightly outperforming the long-standing leader, MDCS. MDCS, however, maintained its lead in the title-based evaluation. The paper provides detailed score sheets (Tables II and IV) summarizing the precision, recall, and F-measure for each participating team across both evaluation metrics. For instance, the link-based F-measure for the top-performing University of Innsbruck was 67.2%, while MDCS achieved 66.6%. The paper also notes that the number of participants increased compared to previous years, potentially due to the availability of training data and the simplified ground truth process.

Limitations and Future Work

The authors acknowledge that while the ground truth creation was more robust, the overall task of book structure extraction remains challenging. A limitation noted is the lack of successful attempts to combine the analysis of ToC areas and full book content simultaneously within a single document, suggesting this as a promising avenue for future research. Future work planned by the authors includes investigating the usability of extracted ToCs for both readers (navigation) and indexing systems (search). They also aim to explore qualitative evaluation measures beyond precision and recall to better understand what makes a ToC truly useful. The paper also suggests that crowdsourcing methods could be a viable approach for building even larger evaluation datasets in the future.

Summary updated

Abstract

This paper summarizes the 3rd Book Structure Extraction competition that was run at the ICDAR 2013. Its goal is to evaluate and compare automatic techniques for deriving structure information from digitized books, which could then be used to aid navigation inside the books. More specifically, the task that participants are faced with is to construct hyperlinked tables of contents for a collection of 1,000 digitized books. This paper reviews the setup of the competition, the book collection used in the task, and the measures used for the evaluation. The main novelty of the 2013 competition is that we were able to rely on an external provider for the ground truthing phase, hence granting the consistency of the evaluation. In addition, this allowed us to nearly double the number of annotated books from the 1,040 books annotated in 2009 and 2011 to over 2,000 books. The paper further presents the resulting performance of the 6 participating research teams, and briefly summarizes their approaches. I.
BibTeX
@misc{doucet2013overview,
  author = {Antoine Doucet and Gabriella Kazai and Sebastian Colutto and Günter Mühlberger},
  title = {{Overview of the ICDAR 2013 Competition on Book Structure Extraction}},
  year = {2013},
}

Loading related content...

Figure