Improving Mention Detection Robustness to Noisy Input

Radu Florian, John F. Pitrelli, Salim Roukos, Imed ZitouniEMNLP 2010 · OpenReview

Abstract

Information-extraction (IE) research typically focuses on clean-text inputs. However, an IE engine serving real applications yields many false alarms due to less-well-formed input. For example, IE in a multilingual broadcast processing system has to deal with inaccurate automatic transcription and translation. The resulting presence of non-target-language text in this case, and non-language material interspersed in data from other applications, raise the research problem of making IE robust to such noisy input text. We address one such IE task: entity-mention detection. We describe augmenting a statistical mention-detection system in order to reduce false alarms from spurious passages. The diverse nature of input noise leads us to pursue a multi-faceted approach to robustness. For our English-language system, at various miss rates we eliminate 97% of false alarms on inputs from other Latin-alphabet languages. In another experiment, representing scenarios in which genre-specific training is infeasible, we process real financial-transactions text containing mixed languages and data-set codes. On these data, because we do not train on data like it, we achieve a smaller but significant improvement. These gains come with virtually no loss in accuracy on clean English text.

The Challenge of Noisy Input

Most Information Extraction (IE) research assumes high-quality, monolingual input. In practice, however, systems often encounter "noisy" text—ranging from speech recognition errors and untranslated foreign phrases to database-specific markup. Standard Mention Detection (MD) models, which identify and categorize entities such as people, organizations, and locations, often fail catastrophically on such data. They exhibit "garbage-in, garbage-out" behavior, generating numerous false alarms by misidentifying noise as valid entities.

The diverse set of entity types targeted by the mention detection system.

The goal is to develop a system that degrades gracefully. It should ignore non-language material, extract recognizable entities from foreign contexts (e.g., an English name in a Spanish sentence), and maintain peak performance on clean text, all without prior knowledge of the input genre.

Multi-Stage Robustness Framework

The proposed solution is a cascaded architecture that dynamically routes input sentences to specialized models based on their perceived quality.

  1. Preprocessing: Non-target character sets (e.g., Arabic or Chinese characters for an English system) are stripped, and the text is tokenized.
  2. Perplexity Routing: The system calculates the perplexity ($PP$) of each sentence using an English n-gram language model. This score serves as an unsupervised metric for "English-likeness."
  3. Cascaded Models: Sentences are routed to one of three Maximum Entropy Markov Models (MEMMs) based on their $PP$ score:
  4. Clean Model: A standard MD model for low-perplexity (clean) text using lexical and gazetteer features.
  5. Mixed Model: Trained on mixed-language data, this model includes localized language-identification features to distinguish between English, foreign languages, and data detritus.
  6. Gazetteer Model: For high-perplexity (very noisy) text, the system falls back to simple dictionary look-ups. This avoids the "liability" of context, where surrounding noise might otherwise mislead a statistical model.

Experimental Evaluation

The system was evaluated across three distinct datasets:

  • English: 367 web news documents (~170,000 words) used for standard training and testing.
  • Latin: 23 web news articles (~31,000 words) in 11 non-English Latin-alphabet languages, annotated for entities recognizable to an English speaker.
  • Transactions: ~60,000 words from real financial transaction logs containing abbreviations and markup. This served as a zero-shot test, as the system was never trained on this specific format.

Results and Discussion

The multi-faceted approach demonstrated significant robustness gains across all noisy conditions.

Performance comparison (Precision, Recall, F-measure) across English, Latin, and Transaction datasets.

As shown in the performance metrics, the "Combination" system dramatically outperforms the "Clean" baseline on noisy data. On the non-English Latin dataset, the system eliminated 97% of false alarms at a 0.6 miss rate. On the highly irregular Transactions dataset, the F-measure improved from 24.3 to 43.2—a substantial gain for a zero-shot scenario where the model had never seen the data format during training.

Crucially, these improvements did not come at the cost of standard performance. On clean English text, the F-measure only shifted from 76.1 to 75.6, representing a negligible penalty for the added robustness.

Conclusion

By using perplexity as a triage tool and combining specialized models, we can build Mention Detection systems that are resilient to the unpredictability of real-world data. The results suggest that in high-noise environments, reducing reliance on context in favor of gazetteers and specialized "mixed" models can effectively suppress false alarms while preserving extraction capabilities. Future work will explore extending this framework to other languages and broader IE tasks.

Summary updated

Abstract

Information-extraction (IE) research typically focuses on clean-text inputs. However, an IE engine serving real applications yields many false alarms due to less-well-formed input. For example, IE in a multilingual broadcast processing system has to deal with inaccurate automatic transcription and translation. The resulting presence of non-target-language text in this case, and non-language material interspersed in data from other applications, raise the research problem of making IE robust to such noisy input text. We address one such IE task: entity-mention detection. We describe augmenting a statistical mention-detection system in order to reduce false alarms from spurious passages. The diverse nature of input noise leads us to pursue a multi-faceted approach to robustness. For our English-language system, at various miss rates we eliminate 97% of false alarms on inputs from other Latin-alphabet languages. In another experiment, representing scenarios in which genre-specific training is infeasible, we process real financial-transactions text containing mixed languages and data-set codes. On these data, because we do not train on data like it, we achieve a smaller but significant improvement. These gains come with virtually no loss in accuracy on clean English text.
BibTeX
@inproceedings{florian2010improving,
  author = {Radu Florian and John F. Pitrelli and Salim Roukos and Imed Zitouni},
  title = {{Improving Mention Detection Robustness to Noisy Input}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2010},
  url = {https://openreview.net/forum?id=HkZn2WGOWB},
}
Figure