Information Extraction from Semi-Structured Web Data

27 concepts19 papers1 proposal

Most natural language processing tools are built to read the way humans read books: left-to-right, decoding sentences that rely on strict grammatical syntax. Traditional Information Extraction (IE) pipelines look for subjects, verbs, and objects to map relationships between entities.

However, the web does not communicate solely in prose. A massive portion of the internet's factual data is locked away in semi-structured formats: HTML tables, bulleted lists, infoboxes, and product directories. When standard text-based analysis methods encounter these non-sentential formats, their performance degrades sharply. To unlock this data, extraction systems must shift their paradigm from parsing linguistic grammar to exploiting structural layout.

The Prose Bias and Why It Fails

When an information extraction model optimized for narrative text encounters a web table or a bulleted list, it faces a fundamental representation crisis.

First, standard linguistic features disappear. As Lockard et al. observe, Open Information Extraction (OpenIE) techniques designed for natural language struggle with the semi-structured web because these pages present facts through visual layouts and key-value pairs rather than standard verb phrases. Without explicit subjects and verbs, traditional dependency parsers fail.

Second, the horizontal context provided by a sentence is lost. Lu et al. point out that semi-structured web records lack the horizontal context of sentences, making tasks like entity typing exceptionally difficult for traditional models.

Because of this "prose bias," standard pipelines suffer from massive recall gaps. Early web-scale extraction research by Cafarella et al. highlighted that focusing exclusively on narrative text misses high-density sources of factual information. Statistical Named Entity Recognition (NER) models show a similar vulnerability; Wang et al. and Wu et al. demonstrate that models trained on prose routinely fail to extract entities like product names when they are isolated inside HTML lists or tables.

This failure is not unique to the open web. In specialized domains, the same limitation appears. Kratzwald et al. note that neural architectures struggle to interpret tables or charts in technical and business documents. Similarly, Lindes et al. and Foroutan et al. show that the telegraphic, non-standard structures found in historical records and legal prospectuses—such as enumerated or bulleted lists—frequently break traditional NLP pipelines.

The Structural Context Solution

If semi-structured data lacks sentences, what does it offer instead? It offers structural consistency.

In a web table or list, the layout itself acts as a "free" semantic blueprint. Cucerzan et al. leverage this insight for question answering, demonstrating that the pre-organized relational data provided by table authors allows for highly effective fact retrieval with minimal manual annotation. Because items in the same column or list share a structural relationship, the layout explicitly maps connections that are rarely stated out loud in narrative prose. For example, Yang et al. show that mining these non-prose structures from Wikipedia provides rich semantic relatedness patterns that significantly improve coreference resolution.

To exploit this, extraction systems must treat structural elements as primary features. Li formalizes this approach through structure-aware entity extraction, targeting HTML tables and lists directly to identify candidate entities rather than searching for sentence-level linguistic features.

Sentential Extraction (Fails on Tables):
[Product Name] -> (Missing Verb Phrase) -> [Price]

Structure-Aware Extraction (Succeeds):
Row Semantics:    [Entity: Product X] <--- (Aligned) ---> [Value: $99]
Column Semantics: [Header: Product]                      [Header: Price]

By shifting the focus to structural cues, models can achieve a high degree of language independence. Because structural consistency relies on layout rather than grammar, systems can bypass complex, language-specific linguistic parsers entirely. For instance, Cohen et al. developed a system (ASIA) that extracts comprehensive entity sets across English, Chinese, and Japanese using the exact same web-mining logic, relying purely on the fact that related items are grouped together in HTML structures. This structural dependency is further confirmed by a head-to-head evaluation in a study on web-scale semantic class mining, which reveals that pattern-based entity extraction relies heavily on HTML cues; stripping away these structural markers causes a substantial decline in extraction quality.

Modern Architectures for Structure-Aware Extraction

Modern approaches scale this intuition by building architectures that explicitly model the geometry and hierarchy of semi-structured data.

Instead of flattening a table into a single string of text, modern frameworks preserve its multi-dimensional nature. Wu et al. introduce a framework (SEE-ST) that treats table rows and columns as primary semantic units, connecting them to surrounding text within a unified evidence graph to improve fact verification. To handle the generation of concise answers from the web, Kim et al. utilize HTML-aware pretrained language models (HTML-PLMs) to capture crucial information buried inside complex tables and lists that standard text-based models miss.

However, extracting data from the web introduces a unique challenge: noise. As Chakrabarti et al. point out, the presence of structural tags does not always mean the content is genuinely structured; only about 10% of web tables contain actual relational data, while the rest are used purely for visual layout.

When extraction systems successfully separate layout-oriented markup from true information-bearing structures, the data yield is massive. Prakash et al. demonstrate that targeting overlooked semi-structured elements like infoboxes uncovers dense, high-value facts that are entirely absent from narrative text. By treating structure as a first-class citizen, these techniques allow researchers like Koukourikos et al. to continuously enrich global knowledge graphs, transforming the messy, semi-structured web into a clean, queryable database of facts.

Key Foundational Work

Methods for Domain-Independent Information Extraction from the Web: An Experimental Comparison (2004) — contributed 1 concept elements to this direction.
Factoid Question Answering over Unstructured and Structured Web Content (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semi_structured_data_extraction_fc326cfb,
  title = {semi-structured data extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semi-structured-data-extraction/rdr_fc326cfb60a94f23be849df6cbc6927e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semi_structured_data_extraction_fc326cfb_v1,
  title = {semi-structured data extraction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semi-structured-data-extraction/rdr_fc326cfb60a94f23be849df6cbc6927e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake