TL;DR
This paper introduces a novel approach for automatically extracting both the content and hierarchical structure of Terms and Conditions (T&Cs) from German and English online shops. Their method significantly outperforms existing general-purpose content extraction tools, addressing the unique challenges posed by the structured nature of legal documents.
What Problem Does This Paper Address?
Analyzing legal documents like online shop Terms and Conditions (T&Cs) is crucial for consumer protection, but it's a challenging task. While general content extraction from web pages is a well-studied problem, existing tools struggle with T&Cs due to their specific, often strictly hierarchical structure. These documents are not like typical news articles or blog posts; they have distinct sections, subclauses, and formatting that general extractors often miss or misinterpret. This leads to incomplete or inaccurate data, hindering further automated analysis. The authors aim to bridge this gap by developing a domain-specific solution that can reliably extract both the core text and its organizational structure.
Methods
The core of the proposed solution lies in a two-pronged approach: Content Extraction and Hierarchy Extraction.
For Content Extraction, they developed the LowestCommonAncestorExtractor. This method leverages the observation that T&Cs typically form a large, contiguous block of content with a common visual style. It first identifies the "Most Common Style" (MCS) across the page's DOM tree. Then, it finds the largest node that covers a significant portion (around 85%) of this MCS, assuming this node contains the main T&Cs. A fallback mechanism is in place for cases where a clear container isn't present. This approach is designed to be robust against malformed HTML.
For Hierarchy Extraction, they build upon the idea that headings and visual styles signal structure. The algorithm identifies headings based on their distinct visual styles and short text length. It then recursively extracts subclauses, using visual cues and numerical patterns (like bullet points or numbered lists) to define the hierarchy. The system also accounts for enumerations that might interrupt sections and handles different numbering schemes (Arabic, Roman, alphabetic).
The overall architecture involves several components: a downloader to fetch web pages, a DOM parser to create a structured representation, and libraries like Selenium for style extraction and SoMaJo for sentence segmentation.
Datasets
The authors used a dataset of T&Cs pages from 30 German and 20 English online shops. This dataset was sampled from a previously collected corpus (Braun and Matthes, 2020). They also created a separate test sample for evaluation to avoid overfitting.
Key Insights and Ideas
- T&Cs are structurally different: Unlike news articles, T&Cs are typically uninterrupted and highly structured, making general-purpose extractors ill-suited.
- Visual style is a strong signal: The common visual style of T&C content is a key indicator for content extraction, and visual differences are crucial for identifying headings and hierarchy.
- Hierarchy is not always reflected in HTML tags: While HTML tags can provide some clues, they are often misused for SEO or not used consistently, necessitating a more robust approach that combines visual and textual cues.
- Enumerations are key to structure: The frequent use of numbered or bulleted lists in T&Cs is a vital signal for understanding the document's hierarchy.
- Robustness is essential: T&C pages can have malformed HTML, requiring an extraction method that can handle such imperfections gracefully.
Key Results
The developed LowestCommonAncestorExtractor demonstrated superior performance compared to existing tools like Boilerpipe (ArticleExtractor, CanolaExtractor), JusText, and Trafilatura.
- Content Extraction: The paper reports that their approach achieved significantly higher "correct" detection rates for both the start and end of T&Cs. For instance, in detecting the start, their method achieved 45 correct extractions out of 50, with only 1 processing error, outperforming the next best (Trafilatura) which had 37 correct. Similar strong results were observed for end detection.
- Hierarchy Extraction: While hierarchy extraction is inherently more complex and prone to minor errors (e.g., wrong parent, wrong title), the algorithm showed good results, with error scores of 0 (perfect extraction) achieved in 8 cases for German and 9 for English in the sample. The authors note that most errors were minor.
Limitations and Future Work
- Domain Specificity: The authors acknowledge that their algorithm's success is based on domain-specific assumptions about T&Cs. Its performance on a generic web corpus is uncertain and warrants further investigation.
- Style Extraction Bottleneck: The current reliance on Selenium for style extraction is identified as a performance bottleneck, as retrieving CSS properties can be time-consuming. Future work could explore more efficient style extraction methods.
- Parallelization: The authors suggest that several content extraction threads could operate in parallel to improve efficiency.
- Hierarchy Irregularities: The visual representation of T&Cs can vary greatly between online shops, making consistent hierarchy extraction challenging. Cases where operators highlight very different elements visually can lead to more errors.
Summary updated