Bootstrapping Information Extraction from Field Books

Sander Canisius, Caroline SporlederEMNLP 2007

TL;DR

This paper tackles the challenge of extracting structured information from semi-structured field books when no labeled training data is available. They propose two methods: one that artificially creates training data from existing databases, and another that uses unsupervised Hidden Markov Models (HMMs) combined with language models. The HMM approach significantly outperforms other methods, demonstrating the feasibility of bootstrapping information extraction systems from databases alone.

What Problem Does This Paper Address?

Many scientific fields, like zoology or archaeology, rely on field books to record observations. These books contain valuable information, but they are often semi-structured, meaning they have some organization but lack explicit markers for different data fields (like "species," "location," or "date"). This makes it difficult to automatically search and retrieve specific information. While databases can store this information in a structured way, manually creating them from field books is time-consuming and expensive, especially for large datasets. The core problem is how to automatically extract structured information from these field books without relying on extensive manual annotation, leveraging existing, albeit imperfect, structured data sources like databases.

Methods

The paper presents two main approaches to "bootstrap" information extraction:

  • Supervised Learning with Artificially Generated Training Data: This method takes an existing database that mirrors the field book data. It artificially creates labeled training data by concatenating database fields. To make this data more realistic, they explore two strategies:
  • Random Concatenation: Simply joining database fields in a random order.
  • Biased Concatenation: Using a small amount of manually annotated data to estimate field ordering probabilities and generate training data that reflects more likely sequences. This artificial data is then used to train a standard supervised machine learning model (specifically, a memory-based learner).

  • Unsupervised Hidden Markov Models (HMMs) with Language Models: This approach is more sophisticated and aims for unsupervised learning. It involves two stages:

  • Segmentation: An n-gram language model, trained on the database content, is used to divide the field book entries into unlabelled segments. The idea is that unusual token sequences are likely to indicate segment boundaries.
  • Labeling: A Hidden Markov Model is then trained. The state-emission distributions (how likely a segment is to be a certain field type) are estimated using language models trained on individual database columns. The initial-state and state-transition distributions (how segments start and transition between each other) are estimated using the Baum-Welch algorithm, treating the database as a source of "almost perfect" supervised data for these specific distributions.

Datasets

The authors evaluated their approaches on two datasets provided by Naturalis, the Dutch National Museum of Natural History:

  • Reptiles and Amphibians (RA): A large dataset with 16,670 entries and 41 columns in the database. Field books were manually annotated for evaluation.
  • Pisces: A smaller, more regular dataset with 1,375 entries and 4 columns, focusing on fish collection stations. Field books were also manually annotated for evaluation.

The paper does not introduce new benchmark datasets but uses existing field book and database pairs.

Key Insights and Ideas

  • Databases as a "Noisy" but Valuable Resource: The paper demonstrates that even though databases derived from field books have systematic differences (e.g., missing punctuation, different formatting, added information), they still contain enough structural clues to bootstrap effective information extraction systems.
  • The Power of Sequential Constraints: The Hidden Markov Model approach significantly outperforms others because it explicitly models sequential constraints between segments. This is crucial for understanding how different pieces of information typically relate to each other in a document.
  • Language Models are Key: Both approaches heavily rely on language models. The supervised method uses them to create features, while the HMM approach uses them for both segmentation and estimating emission probabilities.
  • Unsupervised Learning is Feasible: The success of the HMM approach shows that it's possible to achieve good results without manually labeling large amounts of data, which is a major bottleneck in traditional information extraction.
  • Data Preprocessing Matters: Simple, knowledge-driven rules to align database content with field book text (e.g., adding back RMNH prefixes, converting month formats) significantly improve the performance of the HMM segmentation model.

Key Results

  • HMM Dominance: The Hidden Markov Model approach achieved the best performance across both datasets.
  • On the Reptiles and Amphibians dataset, the HMM achieved a token accuracy of 56.9% and an F-score of 60.3% for segment labeling.
  • On the Pisces dataset, the HMM achieved a token accuracy of 94.4% and an F-score of 86.9%.
  • Baselines Show Improvement with Database Matching: Baseline methods that used database lookups (like ExactB, UniB, TriB) performed significantly better than a simple majority class predictor, indicating the utility of the database.
  • Supervised Learner Benefits from Biased Data: The memory-based learner trained on artificially generated data showed improvement when field ordering probabilities were estimated from a small amount of manual annotation (10 entries), outperforming random concatenation.
  • Segmentation Quality is Crucial: The paper highlights that good segmentation is a prerequisite for good labeling. The HMM's ability to model sequential constraints likely contributed to its superior segmentation performance.

Limitations and Future Work

  • Limitations of Database Data: The paper acknowledges that databases often lose information present in field books (e.g., punctuation as delimiters) and may contain extra information not found in the original texts. This requires careful handling and preprocessing.
  • Need for Some Manual Annotation (for Supervised Approach): While the HMM is largely unsupervised, the supervised approach still benefits from a small amount of manual annotation to estimate field ordering probabilities for more realistic training data generation.
  • Future Work: The authors suggest that the HMM approach could be extended to handle more complex document structures and potentially be applied to other domains. Further exploration of how to best leverage database information and integrate domain-specific knowledge could also be beneficial.

Summary updated

BibTeX
@inproceedings{canisius2007bootstrapping,
  author = {Sander Canisius and Caroline Sporleder},
  title = {{Bootstrapping Information Extraction from Field Books}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure