TL;DR
This paper introduces the setup and results of the ICDAR 2009 Book Structure Extraction competition, which aimed to evaluate automatic methods for generating hyperlinked tables of contents (ToCs) from OCR'd books. The competition established a novel framework, dataset, and evaluation metrics to foster research in this area, crucial for improving navigation and search within digital libraries.
What Problem Does This Paper Address?
Mass digitization projects are creating vast collections of digital books, but current OCR technologies often provide only raw text with minimal structural information. This lack of structure, such as chapters, sections, and logical page numbering, hinders effective navigation and search within these digital libraries. The paper addresses the challenge of automatically extracting this structural information, specifically focusing on generating hyperlinked tables of contents (ToCs) that can significantly enhance user experience.
Methods
The core of the paper lies in the setup of a competition to drive research in this area. Key methodological aspects include:
- Competition Framework: A structured competition was designed to evaluate and compare different automatic techniques for structure extraction.
- Task Definition: Participants were tasked with generating hyperlinked ToCs for a collection of 1,000 digitized books.
- Dataset Creation: A specific corpus of 1,000 digitized books was curated, comprising 800 books with printed ToCs and 200 without, to allow for separate evaluation of techniques.
- Ground Truth Construction: A collaborative effort involving participating institutions was undertaken to manually build a ground truth ToC for a subset of the books, ensuring quality and consistency.
- Evaluation Metrics: Novel evaluation measures were defined to assess the accuracy of extracted ToC titles, links, and depth levels, as well as complete ToC entries. A modified Levenshtein distance was used for title matching to account for variations. An alternative link-based measure was also proposed.
Datasets
- Primary Dataset: A collection of 1,000 digitized books was used for the competition. This dataset was further divided into:
- 800 books with detected printed ToCs.
- 200 books without detected printed ToCs.
- Ground Truth Data: A manually constructed ground truth ToC was created for a subset of the books. Initially, 649 annotated books were collected, which were then reviewed and cleaned, resulting in 527 annotated books for the final ground truth.
- Freely Available Ground Truth: The ground truth set from the previous year's (INEX 2008) Book Structure Extraction task, consisting of 100 ToCs, was made available for future research.
Key Insights and Ideas
- The Importance of Structure: The paper strongly emphasizes that beyond raw text, structural information is critical for making digital books truly usable and navigable.
- Challenges in ToC Extraction: Extracting ToCs is non-trivial due to variations in title formatting, the potential difference between printed and actual book content titles, and the need for accurate page linking.
- Value of a Competition Framework: Establishing a competition with a defined task, dataset, and evaluation metrics is a powerful way to accelerate research in a specific domain.
- Ground Truth is Labor-Intensive: Creating high-quality ground truth data for structure extraction is a significant undertaking, requiring collaborative efforts and careful validation.
- Different Techniques for Different Scenarios: The evaluation highlighted that techniques relying on printed ToCs perform differently than those attempting to infer structure solely from the book's content, suggesting the need for specialized approaches.
Key Results
- Performance Variation: The competition revealed a wide range of performance among participating systems. The best F-measure for complete entries on the entire dataset was 41.51% (MDCS).
- Impact of Printed ToCs: Systems that exploited printed ToCs generally performed better. For books with detected ToCs, the best F-measure was 50.84% (MDCS), while for books without detected ToCs, the best F-measure was significantly lower at 16.09% (University of Caen for titles only) and 7.81% (XRCE for complete entries).
- Annotator Agreement: The study on annotator agreement showed an F-measure of over 70% for complete entries, indicating a reasonable level of consistency in the manual ground truth. However, title matching was identified as a source of disagreement, suggesting potential for improvement in the matching criteria.
- Alternative Metrics: The XRCE link-based measure showed improved scores compared to the title-based measure, suggesting that link accuracy is a crucial aspect of ToC quality.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the evaluation metrics, particularly title matching, might be too strict or require further refinement.
- The performance on books without printed ToCs was significantly lower, indicating a major area for improvement.
- The manual ground truth creation process, while collaborative, could still be subject to annotator bias.
- Future Work:
- Continue running the competition to foster further research and improvement.
- Investigate the usability of extracted ToCs using qualitative evaluation measures beyond precision and recall.
- Explore what properties make a ToC useful and important to users for reading and searching.
- Develop a toolbox of metrics to provide researchers with more comprehensive evaluation capabilities.
- Further research into techniques for books lacking printed ToCs.
Summary updated