Regular Expression Guided Entity Mention Mining from Noisy Web Data

Shanshan Zhang, Lihong He, Slobodan Vucetic, Eduard C. DragutEMNLP 2018 · DOI

Abstract

Extracting entity mentions from web data is challenging due to the inherent noise and variability of source text. While Regular Expressions (REs) provide a high-precision starting point, they are notoriously brittle. Conversely, deep learning models are robust but require extensive labeled datasets. This paper introduces a framework that bridges this gap by using REs to weakly label large-scale unlabeled data for pre-training a character-level Bidirectional LSTM (BiLSTM). This pre-trained model is then fine-tuned with a minimal set of human-labeled examples. Our evaluation across four diverse entity extraction tasks shows that this hybrid approach achieves superior performance, particularly in low-resource scenarios, often surpassing the accuracy of the original REs and purely supervised models.

Introduction

The extraction of entities such as dates, email addresses, and course codes from the web is a critical task for information retrieval and data mining. Traditionally, this has been the domain of Regular Expressions (REs), which are effective for entities following predictable patterns. However, REs are highly sensitive to minor textual variations; a single misplaced character or typo can render a complex expression useless.

In contrast, modern Named Entity Recognition (NER) systems based on Deep Learning offer significant robustness to noise. However, their performance is heavily dependent on the availability of thousands of manually labeled training examples. In the diverse landscape of the web, where data formats vary wildly across different domains, obtaining such large-scale human annotations is often prohibitively expensive. This research addresses this "goldilocks" problem by proposing a method to utilize the imperfect but "free" knowledge embedded in existing REs to jumpstart robust neural networks.

Methodology

The proposed framework integrates the strengths of rule-based systems and deep learning through a four-step process:

  1. Weak Labeling: A large corpus of unlabeled web strings is processed using existing, potentially imperfect REs. Strings matching the RE are labeled as positive ("1"), while others are labeled as negative ("0"), creating a massive, noisy dataset ($D_{RE}$).
  2. Pre-training ($M_{RE}$): A character-level BiLSTM model is trained on $D_{RE}$. By operating at the character level, the model learns the underlying morphology and structural patterns of the target entities, rather than just memorizing specific tokens.
  3. Small-scale Human Labeling: A human expert provides high-quality labels for a very small subset of data (ranging from 20 to 50 instances in experimental settings).
  4. Fine-tuning ($M_{RE+human}$): The pre-trained $M_{RE}$ model is fine-tuned using the small human-labeled dataset to refine its boundaries and adapt to real-world noise.
The integrated framework for RE-guided pre-training and human-labeled fine-tuning.

The architecture, as shown in Figure 1, utilizes an embedding layer followed by two BiLSTM layers, a max-pooling layer, and fully connected layers to produce a binary classification. This structure allows the model to capture long-range dependencies within the character sequences.

Experimental Setup

The framework was evaluated on four distinct tasks using a variety of datasets:

  • Date Time: Over 760,000 strings from news articles.
  • Course Number: Approximately 44,000 strings from the CMU World Wide Knowledge Base.
  • Bill Date: 49,000 strings from US Congress bills.
  • Email Address: 29,000 strings from the Enron email dataset.

To establish a ground truth for evaluation, 6,000 random strings from each dataset were manually labeled to serve as a "gold standard."

Results and Discussion

The experimental results highlight several key advantages of the RE-guided approach. Most notably, the framework demonstrates exceptional performance in ultra-low resource settings.

Performance in Low-Resource Scenarios

As illustrated in Figure 2, the hybrid model ($M_{RE+human}$) consistently outperforms the model trained solely on human labels ($M_{human}$), especially when the training size is small. For the Date Time task, with only 20 human labels, the F1 score of the hybrid model exceeds 85%, whereas the human-only model struggles at approximately 40%.

F1 Score performance across four tasks, comparing the proposed hybrid model against baselines.

Generalization and Robustness

A significant finding is that the neural network often outperforms its "teacher"---the original RE. In tasks like Date Time and Course Number, the fine-tuned model achieved higher F1 scores than the REs used for weak labeling. This is primarily due to improved recall; the model generalizes the patterns learned from the REs and becomes less sensitive to the rigid constraints that cause traditional REs to fail on noisy data (e.g., missing colons or extra spaces).

The RE-Labeling Trade-off

The study also explored the relationship between RE quality and human effort. While a high-quality RE reduces the need for human labels, the model is resilient enough that even mediocre REs can be compensated for by a modest increase in human-labeled data. Once the human-labeled dataset exceeds 1,000 examples, the initial quality of the RE becomes less critical to the final model's performance.

Conclusion

This research demonstrates that Regular Expressions, despite their brittleness, are a valuable source of weak supervision for training robust deep learning models. By combining RE-based pre-training with small-scale human fine-tuning, the proposed framework achieves high accuracy in entity mention mining with minimal manual effort. Future work will focus on extending this binary classification approach to full sequence labeling and exploring the use of neural models as diagnostic tools to detect when underlying data formats have shifted significantly.

Summary updated

BibTeX
@inproceedings{zhang2018regular,
  author = {Shanshan Zhang and Lihong He and Slobodan Vucetic and Eduard C. Dragut},
  title = {{Regular Expression Guided Entity Mention Mining from Noisy Web Data}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1224},
}
Figure