Introduction
Text complexity assessment is a critical component of digital accessibility, ensuring that information remains readable for diverse audiences, including language learners and individuals with cognitive disabilities. While research in English-language text complexity is well-established, German-language resources remain relatively sparse. To address this gap, the HIIG team participated in the GermEval 2022 Text Complexity DE Challenge, which tasked researchers with predicting the Mean Opinion Score (MOS) of German sentences sourced from Wikipedia.
Methodology
The challenge utilized a dataset of 1,000 training sentences, where complexity was rated on a scale from 1 (very easy) to 7 (very complex) by German learners at the B-proficiency level. The HIIG team developed a hybrid approach combining neural networks with traditional linguistic features.
Neural Approach
The team fine-tuned XLM-RoBERTa (XLM-R), a cross-lingual transformer model, for regression. The standard classification head was replaced with a regressor layer and a dropout layer. To maximize the utility of the small training set, 5-fold cross-validation was employed.
Feature-based Approach
In parallel, a Linear Regression model was trained using 34 linguistic features. These included surface-level metrics (e.g., words per sentence, characters per word) and complex grammatical features (e.g., ratios of nouns, verbs, or subordinate clauses).
Ensemble Strategy
The final "Best of Both Worlds" model was an ensemble that averaged the predictions of the neural and feature-based models. Through experimentation, a weighted split of 30% Linear Regression and 70% XLM-R was identified as the optimal configuration.
Experiments with data augmentation using the Hewett and Stede (2021) Lexica Corpus were also conducted but ultimately proved unsuccessful due to label noise and domain mismatch between the augmentation data and the challenge data.
Results
The ensemble approach yielded a significant performance boost over individual models. The 30:70 ensemble achieved a mapped Root Mean Square Error (RMSE) of 0.446, outperforming both the standalone XLM-R model and the feature-based baseline.
Discussion
The success of the ensemble highlights the complementary nature of the two modeling paradigms. The feature-based model provides a robust structural baseline, while the transformer model captures deeper semantic and contextual nuances.
As shown in the scatter plot, while the models are generally correlated, they diverge on specific types of complexity. For instance, the feature-based model often underestimates the difficulty of short sentences containing rare, specialized vocabulary. Conversely, the transformer model can struggle with long sentences that are syntactically simple, such as lists.
This divergence confirms that "classic" linguistic features still capture nuances that deep learning models might overlook when training data is limited. Furthermore, the subjectivity of complexity remains a challenge; for example, the sentence "Das Meerwasser ist leicht basisch" (Seawater is slightly alkaline) was rated as very simple (MOS 1) despite containing the technical term "basisch," suggesting that human raters may prioritize sentence structure over vocabulary in certain instances.
Conclusion
The HIIG team's participation in GermEval 2022 demonstrates that ensembling modern transformers with traditional feature-based models provides a superior solution for text complexity assessment. Future research will explore automated weighting for the ensemble and delve deeper into the subjective factors that influence human complexity ratings.
Summary updated