Enhanced Integrated Scoring for Cleaning Dirty Texts

Wilson Wong, Wei Liu, Mohammed BennamounarXiv 2008 · arXiv

Introduction: The Challenge of "Dirty Text"

As artificial intelligence systems increasingly rely on automated ontology construction from web-based data, they encounter a significant hurdle: the prevalence of "dirty text." Unlike curated academic papers or news articles, the vast majority of digital communication—including emails, blogs, and customer service chat logs—is characterized by linguistic noise. This noise typically manifests as spelling errors, ad-hoc abbreviations, and inconsistent casing.

Traditional Natural Language Processing (NLP) tools often treat these issues as isolated problems. However, in real-world data, these errors are deeply interrelated. For instance, a string like "cta" could be a misspelling of "cat" or a domain-specific abbreviation. Treating these in silos often leads to cascading errors in downstream AI tasks. To address this, researchers at the University of Western Australia developed ISSAC (Integrated Scoring for Spelling error correction, Abbreviation expansion, and Case restoration), a system designed to scrub digital grime through a unified approach.

Examples of linguistic noise in chat logs, including spelling errors, abbreviations, and improper casing.

The ISSAC Methodology

The core philosophy behind ISSAC is the Interdependency Principle: spelling, casing, and abbreviations must be resolved simultaneously to maintain semantic integrity. The system generates candidate corrections from multiple sources—including traditional dictionaries like Aspell, abbreviation lists, and web-scale data—and re-ranks them using a composite Normalized Score ($NS$):

  1. Normalized Edit Distance (NED): Measures the lexical proximity of the suggestion to the original error.
  2. Abbreviation Factor (AF): Checks if the word matches known shorthand patterns.
  3. Domain Significance (DS): Prioritizes words frequent within the specific context (e.g., customer support).
  4. General Significance (GS): Accounts for the word's commonality in general language.
  5. Reuse Factor (RF): A memory component that favors corrections previously validated by high-confidence sources.

Enhancing the Framework

While the basic ISSAC system provided a strong foundation, the researchers introduced three critical upgrades to create "Enhanced ISSAC":

  • Web-Scale Intelligence: By integrating the Google SOAP API, the system gains access to a dynamic, trillion-page corpus. This allows it to recognize emerging terminology (e.g., "iPod," "Xbox") that static dictionaries often miss.
  • Contextual Disambiguation: The enhanced system leverages bigram and trigram analysis. By looking at neighboring words, it can distinguish between similar-sounding corrections based on grammatical context.
  • Refined Reuse Mechanism: To prevent the propagation of errors, the system transitioned from a simple history list to a local "spelling dictionary" populated exclusively by high-confidence Google suggestions or manual overrides.

Empirical Evaluation and Results

The system was benchmarked using a rigorous dataset of 700 real-world chat records from 247Customer.com. These records represent a "worst-case scenario" for text cleaning, featuring heavy use of slang and rapid-fire typing errors.

Comparative performance of Enhanced ISSAC, Basic ISSAC, and Aspell across multiple evaluation sets.

The results demonstrate a clear hierarchy of performance. Standalone tools like Aspell struggled with the informal nature of the text, achieving only 71.03% accuracy. The Basic ISSAC system significantly improved this to 96.53%. However, the Enhanced ISSAC reached a peak average accuracy of 98.01%.

Crucially, the enhancements specifically targeted the system's previous weaknesses. The rate of "missing suggestions" (where the correct word was not in the candidate list) dropped from 2.0% to 0.8%, while errors stemming from inadequate context dropped from 1.0% to 0.7%.

Conclusion

The Enhanced ISSAC system proves that leveraging the "collective intelligence" of the web alongside integrated scoring can nearly eliminate the noise in dirty texts. By solving for spelling, abbreviations, and casing in a single pass, the system provides a robust front-end for automated knowledge engineering. Future work will focus on minimizing the remaining 2% error gap, which often involves complex linguistic anomalies where multiple corrections are equally plausible.

Summary updated

Abstract

An increasing number of approaches for ontology engineering from text are gearing towards the use of online sources such as company intranet and the World Wide Web. Despite such rise, not much work can be found in aspects of preprocessing and cleaning dirty texts from online sources. This paper presents an enhancement of an Integrated Scoring for Spelling error correction, Abbreviation expansion and Case restoration (ISSAC). ISSAC is implemented as part of a text preprocessing phase in an ontology engineering system. New evaluations performed on the enhanced ISSAC using 700 chat records reveal an improved accuracy of 98% as compared to 96.5% and 71% based on the use of only basic ISSAC and of Aspell, respectively.
BibTeX
@misc{wong2008enhanced,
  author = {Wilson Wong and Wei Liu and Mohammed Bennamoun},
  title = {{Enhanced Integrated Scoring for Cleaning Dirty Texts}},
  year = {2008},
  eprint = {0810.0332},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/0810.0332},
}
Figure