Information Extraction with Character-level Neural Networks and Noisy Supervision

Philipp Meerkamp, Zhengyi ZhouarXiv 2016 · OpenReview

TL;DR

This paper introduces a novel information extraction (IE) architecture that combines a traditional constraint-based parser with a character-level neural network. By training the neural network on "noisy" supervision derived from database consistency checks, the system significantly boosts precision in financial text IE, outperforming a mature, hand-tuned system.

What Problem Does This Paper Address?

Extracting structured information from unstructured text, especially in specialized domains like finance, is crucial for applications like forecasting and algorithmic trading. However, traditional IE systems often struggle with the sheer volume and complexity of textual data, leading to either low recall or precision issues. Financial text, in particular, is often terse and grammatically loose, making accurate entity and relation extraction challenging. While deep learning has shown promise, it typically requires large amounts of meticulously annotated data, which is expensive and time-consuming to obtain. This paper tackles the challenge of improving IE precision without relying on extensive manual annotation, leveraging existing domain knowledge and readily available, albeit imperfect, data sources.

Methods

The proposed architecture augments a high-recall candidate-generating parser with a deep neural network. The process involves four stages:

  1. Candidate Generation: A parser, potentially constraint-based, identifies potential extractions (entities and relations) from the document.
  2. Consistency Scoring: A "noisy" supervision signal is generated by checking the extracted data's consistency against an existing database. This results in a consistency score ($s$).
  3. Neural Network Encoding & Scoring: Each candidate extraction, along with its surrounding text, is encoded into feature vectors. Crucially, this includes character-level embeddings that incorporate both the character itself and information about parser-identified entities at that position. A character-level LSTM network then processes these features to compute a neural network candidate correctness score ($\tilde{y}$).
  4. Linear Classification: A final linear classifier uses the consistency score ($s$) and the neural network's correctness estimate ($\tilde{y}$) to make a final decision on whether to accept or discard the extraction.

The key innovation lies in the character-level neural network and the use of database consistency as a form of "free" or noisy supervision for training, avoiding the need for large-scale manual annotations.

Datasets

The paper does not explicitly name specific public datasets used for training or evaluation. Instead, it describes using a database of historical time series data for financial text as the source of "noisy supervision" to check the consistency of extracted numerical data. This approach leverages existing domain-specific data rather than relying on newly created benchmarks.

Key Insights and Ideas

  • Leveraging "Free" Supervision: The most significant insight is the effective use of database consistency as a noisy supervision signal. This bypasses the expensive manual annotation bottleneck, making the approach practical for real-world applications where large, clean datasets are scarce.
  • Character-Level Power: Encoding text at the character level, combined with parser output information, allows the neural network to learn fine-grained linguistic features and entity characteristics, which is particularly beneficial for noisy or abbreviated financial text.
  • Hybrid Approach Synergy: Combining a constraint-based parser (for domain knowledge and initial candidate generation) with a deep neural network (for learning complex features from data) creates a powerful synergy. The parser can enforce constraints that are hard for pure neural networks, while the neural network boosts precision.
  • Boosting Existing Systems: The architecture is designed to augment, rather than replace, existing IE systems, offering a path to significant accuracy improvements with minimal disruption.

Key Results

The deployed pipeline in a production financial information extraction setting achieved:

  • > 90% reduction in false positives compared to a mature, highly tuned constraint-based system.
  • A recall drop of less than 1% relative to the production system.
  • The neural network, even with only 256 hidden LSTM cells, significantly outperformed a 2-layer fully connected network using document text and parser annotations.

Limitations and Future Work

  • Noisy Supervision Assumption: The method assumes that the noise in the supervision source is limited in magnitude (e.g., < 5%) and does not exhibit strong patterns that correlate with candidate attributes. If these assumptions are violated, the correctness estimates might be less reliable.
  • Future Work: While not explicitly detailed, the authors suggest the setup can be easily applied to various other information extraction tasks where noisy reference data is available, implying a broad applicability beyond financial text. The paper also mentions that the pipeline could be trained with supervision from human interaction (e.g., clicks on online advertisements), suggesting a direction for incorporating more direct human feedback.

Summary updated

Abstract

We present an architecture for information extraction from text that augments an existing parser with a character-level neural network. The network is trained using a measure of consistency of extracted data with existing databases as a form of noisy supervision. Our architecture combines the ability of constraint-based information extraction systems to easily incorporate domain knowledge and constraints with the ability of deep neural networks to leverage large amounts of data to learn complex features. Boosting the existing parser's precision, the system led to large improvements over a mature and highly tuned constraint-based production information extraction system used at Bloomberg for financial language text.
BibTeX
@misc{meerkamp2016information,
  author = {Philipp Meerkamp and Zhengyi Zhou},
  title = {{Information Extraction with Character-level Neural Networks and Noisy Supervision}},
  year = {2016},
  eprint = {1612.04118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1612.04118},
}
Figure