On-Line Error Detection of Annotated Corpus Using Modular Neural Networks

Qing Ma, Bao-Liang Lu, Masaki Murata, Michinori Ichikawa, Hitoshi IsaharaICANN 2001 · DOI

TL;DR

This paper introduces a novel "on-line" method for detecting errors in manually annotated corpora by leveraging the inherent properties of min-max modular (M³) neural networks. By identifying modules that fail to converge during training, the system can pinpoint potentially erroneous data points, offering a more efficient and cost-effective way to improve corpus quality and enhance machine learning models.

What Problem Does This Paper Address?

Manually annotating large corpora for natural language processing (NLP) tasks like part-of-speech (POS) tagging is a crucial but incredibly time-consuming and expensive endeavor. While automatic methods have improved, errors inevitably creep into these annotated datasets. These errors, especially subtle ones like inconsistent tagging, can significantly degrade the performance of machine learning models trained on them. Existing error detection methods are often "off-line," requiring a separate, computationally intensive pass over the entire corpus before training, which is impractical for very large datasets. The authors aim to develop an "on-line" error detection mechanism that integrates seamlessly with the training process, making it more efficient and cost-effective.

Methods

The core of the proposed method lies in the use of min-max modular (M³) neural networks. These networks tackle complex problems by decomposing them into many smaller, independent two-class subproblems, each handled by a dedicated module. The key insight is that M³ networks are designed to have guaranteed convergence. Therefore, if a module fails to converge during training, it suggests that the data it's learning from contains inconsistencies.

The authors propose an on-line error detection strategy:

  1. Decomposition: A K-class problem (like POS tagging with 97 classes) is recursively decomposed into numerous two-class problems.
  2. Modular Learning: Each two-class subproblem is learned by an individual module, typically a simple multilayer perceptron.
  3. On-line Detection: During the training of these modules, if a module fails to converge, it's flagged. The data associated with this unconverged module is then examined for conflicting pairs (same input, different desired outputs), indicating potential errors in the annotation.
  4. Integration: The unconverged modules and their associated conflicting data are then used to improve the corpus quality, either through manual intervention or by re-training the modules with corrected data.

Datasets

The experiments were conducted on a subset of the Kyoto University Text Corpus. Specifically, they used 217 Japanese sentences that were known to contain at least one error. This subset comprised 6,816 words, 2,410 distinct words, and 97 kinds of POS tags. This dataset was chosen for preliminary study to test the error detection capabilities.

Key Insights and Ideas

  • Unconvergence as a Signal: The most significant insight is reframing the "problem" of unconvergence in neural networks as a valuable "signal" for error detection. Instead of being a failure, unconvergence in M³ modules points directly to problematic data.
  • Cost-Effectiveness: By detecting errors during training, the method avoids the need for a separate, expensive off-line pass, making it highly cost-effective for large corpora.
  • Improved Corpus Quality: The on-line detection allows for immediate feedback and correction, leading to a continuously improving corpus quality as it's being learned.
  • Discovery of Linguistic Knowledge: Beyond error detection, the analysis of unconverged modules revealed subtle linguistic nuances, such as the complex case of the Japanese word "de," which requires more than just local word and POS information to tag correctly. This suggests the method can uncover deeper linguistic rules.

Key Results

  • The preliminary experiment on 217 sentences identified 82 modules out of 23,231 that did not converge.
  • Of these 82 modules, 81 had exactly 97 pairs of contradictory learning data.
  • Manual checking by an NLP expert revealed that 94 out of these 97 pairs contained true POS errors.
  • The precision rate for error detection was nearly 97%.
  • The recall rate was low due to the small size of the experimental corpus, but the authors expect it to increase dramatically with larger datasets.

Limitations and Future Work

  • Small Corpus Size: The primary limitation acknowledged is the small size of the experimental corpus (217 sentences). This led to a low recall rate, as insufficient data was available to identify all errors.
  • Future Work: The authors plan to conduct a full-scale experiment using the entire Kyoto University Text Corpus. They anticipate a significant increase in the recall rate with this larger dataset, while maintaining high precision. They also suggest that this method could be valuable for improving POS tagging systems and discovering linguistic knowledge.

Summary updated

BibTeX
@inproceedings{ma2001line,
  author = {Qing Ma and Bao-Liang Lu and Masaki Murata and Michinori Ichikawa and Hitoshi Isahara},
  title = {{On-Line Error Detection of Annotated Corpus Using Modular Neural Networks}},
  booktitle = {ICANN},
  year = {2001},
  doi = {10.1007/3-540-44668-0\_165},
}
Figure