TL;DR
This paper argues that the Flesch-Kincaid Grade Level (FKGL) metric is a poor choice for evaluating text simplification systems because it's easily manipulated without improving actual text quality. The authors demonstrate this by showing how simple post-processing techniques can drastically lower FKGL scores with minimal impact on other established metrics like BLEU and SARI. They propose using FKGL's component statistics (average sentence length and syllable count) for more insightful post-hoc analysis instead.
What Problem Does This Paper Address?
Evaluating the quality of text simplification systems is crucial for their development and deployment. Text simplification aims to make text easier to understand while preserving its original meaning. Researchers often rely on automated metrics to assess system performance, as human evaluation is costly and time-consuming. While metrics like BLEU and SARI are commonly used, the Flesch-Kincaid Grade Level (FKGL) has also been a popular, albeit older, metric for measuring readability. However, the authors of this paper question FKGL's suitability for text simplification evaluation. They highlight that FKGL, originally designed to assess human-written text, might not be robust enough for evaluating machine-generated simplifications. Specifically, they are concerned that FKGL might be "gameable," meaning systems could artificially inflate their scores without genuinely improving the simplification's quality. This paper aims to investigate this "gameability" and demonstrate why FKGL should be reconsidered as a primary evaluation metric.
Methods
The core of the paper's methodology involves a series of experiments designed to test the "gameability" of the FKGL metric. The authors introduce six simple, explicit post-processing techniques that can be applied to the output of text simplification systems. These techniques are designed to manipulate the statistics that FKGL relies on: average sentence length and average syllables per word. The six methods are:
- random-period: Inserts a period to split sentences, reducing average words per sentence.
- random-the: Inserts the word "the" to reduce average syllables per word.
- replace-longest: Replaces the longest word with "the" to reduce average syllables per word.
- replace-rand-period: Replaces a random word with a period, also removing a word.
- replace-rand-the: Replaces a random word with "the", also removing a word.
- rand-period+repl-longest: Combines the effects of
random-periodandreplace-longest.
These modifications were applied to the output of five different text simplification systems at varying percentages (10% to 100%) of sentences. The impact of these modifications was then measured not only on FKGL but also on BLEU and SARI to see if other metrics were similarly affected. The experiments were repeated 100 times to ensure reliable averages.
Datasets
The experiments were conducted using the Newsela corpus. This corpus consists of news articles manually simplified at various levels of simplicity and is noted for its explicit human simplification and lower noise compared to other corpora. The authors also mention conducting similar analyses on the Wikipedia corpus, which contains automatically aligned sentences between English Wikipedia and Simple English Wikipedia, and found similar results.
Key Insights and Ideas
- FKGL is easily manipulated: The most striking insight is how easily FKGL scores can be drastically reduced with very simple, almost trivial, post-processing steps. This suggests that a low FKGL score might not necessarily indicate a high-quality simplification.
- Minimal impact on other metrics: Crucially, these FKGL-manipulating modifications had a negligible impact on BLEU and SARI scores. This highlights a significant disconnect: FKGL can be "gamed" independently of metrics that are more closely tied to semantic similarity and fluency.
- FKGL's historical context is a limitation: The paper points out that FKGL was developed in the 1940s based on studies of children and Navy personnel, using text assumed to be error-free. These assumptions may not hold for machine-generated text, which can have different characteristics.
- Component statistics are more informative: The authors propose that instead of reporting the overall FKGL score, it's more beneficial to report its constituent parts: average sentence length and average syllables per word. These components provide a clearer picture of how a system is simplifying text and are better correlated with human judgment.
Key Results
The experimental results clearly demonstrate the gameability of FKGL. For instance, applying the rand-period+repl-longest modification to 100% of the sentences in the Dress-Ls output reduced the FKGL score to almost zero. This was achieved with only a minor drop in BLEU (from 0.2374 to 0.1710) and virtually no change in SARI (less than 0.004). Similar trends were observed across all tested systems and modification percentages. While BLEU scores did show some decrease with modifications, the impact was significantly less pronounced than on FKGL. SARI scores, in particular, remained largely unaffected by the FKGL-manipulating techniques.
Limitations and Future Work
The paper acknowledges that its modifications were designed specifically to manipulate FKGL and thus do not serve as a definitive argument against the robustness of BLEU and SARI themselves. However, the primary limitation identified is the inherent flaw in using FKGL as a primary evaluation metric due to its susceptibility to manipulation.
The authors suggest that FKGL should be used for post-hoc analysis rather than as a direct evaluation metric. They advocate for reporting the component statistics of FKGL (average sentence length and average syllables per word) to provide a more nuanced understanding of system behavior. They also suggest supplementing these with other corpus statistics, such as the proportion of sentences split, to gain a comprehensive view of simplification strategies. Future work could involve exploring other "gameable" metrics and developing more robust evaluation frameworks that are less susceptible to artificial inflation.
Summary updated