Learning to Recognize Tables in Free Text

Hwee Tou Ng, Chung Yong Lim, Jessica Li Teng KooACL 1999 · OpenReview

Abstract

This paper presents a machine learning approach to the problem of recognizing tables in plain ASCII text. By decomposing the task into boundary recognition, column recognition, and row recognition, the authors demonstrate that adaptive classifiers like C4.5 and backpropagation neural networks can significantly outperform traditional rule-based systems. The proposed method achieves high accuracy by leveraging surface features of the text geometry, proving to be robust across different document domains.

Introduction: The Challenge of Tabular Data in ASCII

In the late 1990s, a vast amount of digital information was stored in plain ASCII text format. While humans can intuitively identify the structure of a table within a document, automated Information Extraction (IE) systems frequently struggle. If an IE system fails to recognize a table and instead treats its rows as standard sentences, the resulting data extraction is often nonsensical.

The core difficulty lies in the idiosyncratic nature of tables. There is no universal standard for column spacing, header formatting, or row alignment. Traditional approaches relied on deterministic algorithms—essentially long, brittle lists of "if-then" rules based on specific punctuation and spacing patterns. These systems often failed when encountering new layouts or domains. The authors sought to replace these rigid rules with an adaptive, trainable framework.

A sample of raw text where a table is embedded, illustrating the challenge of identifying boundaries and rows.

Methodology: A Sequential Machine Learning Pipeline

The researchers broke the complex 2D problem of table recognition into three manageable 1D classification subproblems:

  1. Boundary Recognition: Identifying which horizontal lines (hlines) in a document belong to a table.
  2. Column Recognition: Determining the vertical boundaries (vlines) within those identified tables to separate data fields.
  3. Row Recognition: Grouping hlines into logical rows, accounting for the fact that a single table row often spans multiple lines of text.

Instead of semantic analysis, the system relies on surface features—statistical measures of the text's geometry. These include the proportion of space characters versus alphanumeric characters, the number of contiguous space segments, and character type transitions (e.g., moving from a space to a special character). By feeding these feature-value vectors into C4.5 decision trees and backpropagation neural networks, the system learns the visual "signature" of a table without needing to understand the actual content.

Key Results: Superiority of Adaptive Classifiers

The machine learning models were trained and tested on 100 Wall Street Journal (WSJ) documents and 50 TIPSTER patent documents. The results were clear: the ML approach significantly outperformed the deterministic baseline across all tasks.

Learning curves for boundary recognition showing the performance of C4.5 and Neural Networks (Bp) compared to a deterministic baseline (Det).

As shown in the performance metrics, while the deterministic algorithm (Det) hovered around 70% accuracy, the ML models (C4.5 and Bp) achieved between 85% and 95% accuracy. Notably, the models reached high performance levels relatively quickly, showing substantial improvement with as few as 20 to 40 training examples. Furthermore, the system demonstrated strong domain independence; a classifier trained on financial news from the WSJ maintained high accuracy when tested on complex technical patent filings, proving that the geometric features are universal indicators of tabular structure.

Conclusion and Future Directions

By prioritizing geometry over semantics, the authors developed a robust system for table recognition that moves beyond the limitations of hand-crafted rules. While the current system excels at structural identification, future work could involve distinguishing between different functional row types (such as headers vs. data rows) and exploring exemplar-based methods like k-nearest neighbors to further refine accuracy in extremely complex or nested layouts. This research marks a significant step toward more flexible and intelligent document processing systems.

Summary updated

BibTeX
@inproceedings{ng1999learning,
  author = {Hwee Tou Ng and Chung Yong Lim and Jessica Li Teng Koo},
  title = {{Learning to Recognize Tables in Free Text}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {1999},
  url = {https://openreview.net/forum?id=0MLyQxVKM-u},
}
Figure