Enhanced Word Decomposition by Calibrating the Decision Threshold of Probabilistic Models and Using a Model Ensemble

Sebastian Spiegler, Peter A. FlachACL 2010

Abstract

Morphological analysis is a critical task in natural language processing, particularly for agglutinative languages like Zulu. This paper investigates methods to improve the performance of generative probabilistic models for word decomposition. By moving beyond default decision thresholds and employing model ensembles, we demonstrate significant improvements in F-measure scores. Specifically, threshold calibration and the combination of standard and higher-order models allow for a more nuanced interpretation of boundary probabilities, leading to state-of-the-art results on the Ukwabelana Zulu corpus.

Introduction

Morphological analysis involves decomposing words into their constituent morphemes (e.g., roots, prefixes, and suffixes). In languages like Zulu, where words are formed by complex agglutination, this task is both essential and challenging. Standard probabilistic models often suffer from over-segmentation or rigid decision-making based on a default 0.5 probability threshold. This paper addresses these limitations by exploring higher-order models, threshold calibration via Precision-Recall analysis, and model ensembling.

Methodology

The study evaluates three primary configurations:

  1. PROMODES: A baseline generative model utilizing a zero-order boundary model and a first-order letter transition Markov chain.
  2. PROMODES-H: A higher-order evolution incorporating a first-order boundary model (where decisions depend on previous boundaries) and a second-order letter transition model.
  3. PROMODES-E (Ensemble): A model that averages the boundary probabilities from PROMODES and PROMODES-H.

A key innovation is Threshold Calibration. Rather than using the standard 0.5 cutoff, the researchers identify an optimal threshold ($h^*$) by analyzing Precision-Recall (PR) curves on a validation set.

Results and Analysis

The experimental results on the Zulu Ukwabelana corpus highlight the impact of calibration and ensembling.

The Impact of Calibration

The default 0.5 threshold is often sub-optimal due to the skewed distribution of morpheme boundaries. For the baseline PROMODES model, calibrating the threshold to approximately 0.36 resulted in a dramatic F-measure increase from 0.4378 to 0.6699.

Precision-Recall curves for PROMODES, PROMODES-H, and PROMODES-E, showing the improvement from default (diamonds) to optimal (triangles) thresholds.

Model Ensembling and Spatial Performance

The ensemble model, PROMODES-E, achieved the highest overall performance (F-measure of 0.6927). This success stems from the complementary strengths of the individual models. Spatial analysis revealed that PROMODES-H excels at identifying non-boundaries at the beginning of words, whereas the simpler PROMODES model is more effective toward the end of words.

Absolute true negatives (TN) across relative word positions for calibrated models, illustrating how different models excel at different parts of the word.

Conclusion

This research demonstrates that the interpretation of a model's output is as vital as the model architecture itself. By calibrating decision thresholds and leveraging ensembles to exploit local model strengths, significant gains in morphological decomposition can be achieved, even with limited training data. Future work may explore more sophisticated ensembling techniques, such as weighted averages or meta-learning, to further refine these results.

Summary updated

BibTeX
@inproceedings{spiegler2010enhanced,
  author = {Sebastian Spiegler and Peter A. Flach},
  title = {{Enhanced Word Decomposition by Calibrating the Decision Threshold of Probabilistic Models and Using a Model Ensemble}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
}
Figure