Text Complexity DE Challenge 2022 Submission Description: Pairwise Regression for Complexity Prediction

Leander GirrbachGermEval@KONVENS 2022

Introduction

Automated readability scoring is a vital component for developing educational tools and accessible content. However, training robust models for this task often requires large amounts of labeled data, which is frequently unavailable for specific languages or domains. This paper describes a submission to the Text Complexity DE Challenge 2022, which provided a dataset of only 1,000 labeled German sentences. To address the challenge of limited data, the authors investigated a pairwise regression approach, hypothesizing that predicting the relative complexity between sentence pairs would effectively augment the training signal and provide reliable uncertainty estimates.

Methodology

The core of the proposed system is a deep learning pipeline that utilizes pre-trained language models as feature extractors. The authors compared three primary embeddings: GottBERT, a German-specific BERT (dbmz), and a multilingual sentence transformer (S-BERT).

Pairwise Regression Architecture

The pairwise model is designed to predict the difference in complexity between two sentences rather than an absolute score for a single sentence. As shown in Figure 2, each sentence in a pair is passed through a Multi-Layer Perceptron (MLP) after embedding. The resulting representations are concatenated and processed by a final MLP to output a relative complexity score.

Architecture of the pairwise regression model.

Inference and Scoring

During inference, an unseen sentence is compared against every sentence in the training set. This process generates a distribution of estimated absolute scores. The final complexity prediction is derived from the mean of this distribution. Theoretically, the variance of these predictions could serve as a proxy for model uncertainty, where a higher standard deviation indicates lower confidence in the prediction.

Results and Discussion

The models were evaluated using 10-fold cross-validation on the training set. Contrary to the initial hypothesis, the pairwise approach did not significantly outperform standard single-sentence regression.

Model Performance

As summarized in Figure 5, the "Combined" standard regression model (using all three embeddings) achieved the lowest RMSE of 0.6068. Among the individual embeddings, GottBERT proved to be the most effective feature extractor for German text complexity. Notably, all deep learning configurations outperformed the traditional Random Forest baseline, which relied on manually extracted linguistic features.

RMSE performance of various models and baselines.

Uncertainty and Error Analysis

A key goal of the pairwise method was to extract uncertainty estimates. However, analysis revealed that the standard deviation of the predicted score distributions did not correlate with actual prediction errors (Pearson correlation of -0.18). This suggests that the model's internal variance is not a reliable indicator of its accuracy on specific samples. Furthermore, attempts to use traditional linguistic features to predict model errors were unsuccessful, yielding an $R^2$ score of only 0.02.

Conclusion

While the pairwise regression approach provided a novel way to frame the text complexity task, it did not yield the expected performance gains or useful uncertainty metrics over standard regression. Nevertheless, the study confirms that deep learning models using pre-trained embeddings like GottBERT significantly outperform traditional feature-based methods, even in low-data regimes. Future work should focus on improving the explainability of these "black-box" models to make them more suitable for educational applications.

Summary updated

BibTeX
@inproceedings{girrbach2022text,
  author = {Leander Girrbach},
  title = {{Text Complexity DE Challenge 2022 Submission Description: Pairwise Regression for Complexity Prediction}},
  booktitle = {GermEval@KONVENS},
  year = {2022},
}
Figure