TL;DR
This paper summarizes the 2013 ICDAR competition on automatically extracting book structure, specifically focusing on generating hyperlinked tables of contents (ToCs) from digitized books. The competition aimed to advance techniques for making large digital libraries more navigable and searchable, with a key innovation being the use of an external provider for ground truth creation, leading to a larger and more consistent dataset.
What Problem Does This Paper Address?
Digitizing vast collections of books, like those undertaken by the Million Book Project or Google Books, results in full-text content but often lacks rich structural information beyond basic paragraphs. This makes it difficult for users to navigate and search within these digital libraries effectively. While OCR can identify text, it typically doesn't recognize higher-level structures like chapters or sections. The challenge is to automatically derive this structural information, such as creating hyperlinked tables of contents, to enhance user experience and search capabilities in digitized book collections.
Methods
The competition's core task involved participants developing systems to generate hyperlinked tables of contents (ToCs) for a set of 1,000 digitized books. Systems were provided with OCR text in XML format and the corresponding PDF/DjVu files. Participants could use any techniques, leveraging either text or page images. The output was an XML file conforming to a specific Document Type Definition (DTD). Evaluation was performed using two complementary metrics: a title-based measure (comparing ToC entry titles and their hierarchical depth and page number accuracy) and a link-based measure (focusing on equivalent page links).
Datasets
The competition utilized a test set of 1,000 digitized books. This collection was derived from the larger INEX book corpus of over 50,000 books. Notably, 200 of these 1,000 books were specifically selected to not contain a printed ToC, forcing participants to rely on deriving structure from the full book content. The ground truth for this dataset was created using a specialized tool called FEP (Functional Extension Parser) developed by the University of Innsbruck. This process involved an initial automatic structure analysis followed by manual correction. A significant improvement over previous years was the use of an external, third-party company (DIGI-TEXX) for manual correction, ensuring consistency and unbiased evaluation. The ground truth data for 967 books (after removing some non-ToC books and duplicates) was made freely available for research purposes.
Key Insights and Ideas
- The value of structured data for navigation: The fundamental premise is that structured information, like a ToC, is crucial for making large digital book collections usable beyond simple keyword searches.
- Ground truth quality is paramount: The paper highlights how the shift to an external, professional ground truth creation process significantly improved the reliability and consistency of the evaluation, a critical factor in any benchmark.
- Two distinct approaches to ToC extraction: Participants primarily fell into two camps: those focusing on detecting and exploiting printed ToC pages, and those attempting to derive structure from the entire book content. A third "hybrid" approach also emerged.
- Link-based evaluation is more robust: The link-based evaluation metric showed improved performance compared to title-based metrics, suggesting that it's more forgiving of minor variations in ToC entry titles and better captures the essential functionality of linking to the correct page.
- The challenge of unsupervised methods: While unsupervised methods for structure extraction were presented, they generally lagged behind supervised or semi-supervised approaches, indicating the difficulty of robustly identifying complex structures without prior knowledge.
Key Results
The 2013 competition saw 6 participating research teams submit runs. The University of Innsbruck achieved the best performance in the link-based evaluation, slightly outperforming the long-standing leader, MDCS. MDCS, however, maintained its lead in the title-based evaluation. The paper provides detailed score sheets (Tables II and IV) summarizing the precision, recall, and F-measure for each participating team across both evaluation metrics. For instance, the link-based F-measure for the top-performing University of Innsbruck was 67.2%, while MDCS achieved 66.6%. The paper also notes that the number of participants increased compared to previous years, potentially due to the availability of training data and the simplified ground truth process.
Limitations and Future Work
The authors acknowledge that while the ground truth creation was more robust, the overall task of book structure extraction remains challenging. A limitation noted is the lack of successful attempts to combine the analysis of ToC areas and full book content simultaneously within a single document, suggesting this as a promising avenue for future research. Future work planned by the authors includes investigating the usability of extracted ToCs for both readers (navigation) and indexing systems (search). They also aim to explore qualitative evaluation measures beyond precision and recall to better understand what makes a ToC truly useful. The paper also suggests that crowdsourcing methods could be a viable approach for building even larger evaluation datasets in the future.
Summary updated