HIIG at GermEval 2022: Best of Both Worlds Ensemble for Automatic Text Complexity Assessment

Hadi Asghari, Freya HewettGermEval@KONVENS 2022 · OpenReview

Introduction

Text complexity assessment is a critical component of digital accessibility, ensuring that information remains readable for diverse audiences, including language learners and individuals with cognitive disabilities. While research in English-language text complexity is well-established, German-language resources remain relatively sparse. To address this gap, the HIIG team participated in the GermEval 2022 Text Complexity DE Challenge, which tasked researchers with predicting the Mean Opinion Score (MOS) of German sentences sourced from Wikipedia.

Methodology

The challenge utilized a dataset of 1,000 training sentences, where complexity was rated on a scale from 1 (very easy) to 7 (very complex) by German learners at the B-proficiency level. The HIIG team developed a hybrid approach combining neural networks with traditional linguistic features.

Neural Approach

The team fine-tuned XLM-RoBERTa (XLM-R), a cross-lingual transformer model, for regression. The standard classification head was replaced with a regressor layer and a dropout layer. To maximize the utility of the small training set, 5-fold cross-validation was employed.

Feature-based Approach

In parallel, a Linear Regression model was trained using 34 linguistic features. These included surface-level metrics (e.g., words per sentence, characters per word) and complex grammatical features (e.g., ratios of nouns, verbs, or subordinate clauses).

Ensemble Strategy

The final "Best of Both Worlds" model was an ensemble that averaged the predictions of the neural and feature-based models. Through experimentation, a weighted split of 30% Linear Regression and 70% XLM-R was identified as the optimal configuration.

Experiments with data augmentation using the Hewett and Stede (2021) Lexica Corpus were also conducted but ultimately proved unsuccessful due to label noise and domain mismatch between the augmentation data and the challenge data.

Results

The ensemble approach yielded a significant performance boost over individual models. The 30:70 ensemble achieved a mapped Root Mean Square Error (RMSE) of 0.446, outperforming both the standalone XLM-R model and the feature-based baseline.

RMSE performance across different model configurations.

Discussion

The success of the ensemble highlights the complementary nature of the two modeling paradigms. The feature-based model provides a robust structural baseline, while the transformer model captures deeper semantic and contextual nuances.

Comparison of predictions between the feature-based and transformer models.

As shown in the scatter plot, while the models are generally correlated, they diverge on specific types of complexity. For instance, the feature-based model often underestimates the difficulty of short sentences containing rare, specialized vocabulary. Conversely, the transformer model can struggle with long sentences that are syntactically simple, such as lists.

This divergence confirms that "classic" linguistic features still capture nuances that deep learning models might overlook when training data is limited. Furthermore, the subjectivity of complexity remains a challenge; for example, the sentence "Das Meerwasser ist leicht basisch" (Seawater is slightly alkaline) was rated as very simple (MOS 1) despite containing the technical term "basisch," suggesting that human raters may prioritize sentence structure over vocabulary in certain instances.

Conclusion

The HIIG team's participation in GermEval 2022 demonstrates that ensembling modern transformers with traditional feature-based models provides a superior solution for text complexity assessment. Future research will explore automated weighting for the ensemble and delve deeper into the subjective factors that influence human complexity ratings.

Summary updated

BibTeX
@inproceedings{asghari2022hiig,
  author = {Hadi Asghari and Freya Hewett},
  title = {{HIIG at GermEval 2022: Best of Both Worlds Ensemble for Automatic Text Complexity Assessment}},
  booktitle = {GermEval@KONVENS},
  year = {2022},
  url = {https://openreview.net/forum?id=YXIhUnysEY7},
}

Loading related content...

Figure