Abstract
Evaluating Natural Language Generation (NLG) systems in production environments is challenging, as standard metrics often fail to capture practical utility. This paper explores using human post-editing as a quantitative framework for evaluation. By analyzing the modifications made by professional meteorologists to the output of the SUMTIME-MOUSAM weather forecasting system, the study provides insights into system performance and the nature of human-machine collaboration. Results indicate that while the system generates high-quality drafts, human edits are largely driven by stylistic preferences and linguistic ellipsis rather than factual inaccuracies.
Introduction
Evaluating NLG systems—which transform structured data into human-readable text—is a complex task. Standard metrics often rely on subjective human ratings or automated comparisons to a static "gold standard" corpus, neither of which fully reflects a system's operational value. This research focuses on the SUMTIME-MOUSAM system, used to generate marine weather forecasts for offshore oil rigs. In this production pipeline, the NLG system produces a draft that is subsequently refined by professional forecasters. The study explores whether the frequency and nature of these human edits can serve as a reliable proxy for system utility, a technique well-established in Machine Translation but relatively novel in NLG.
Methodology
The researchers developed an automated tool to compare "Pre-edit" (raw NLG output) and "Post-edit" (final human-approved) texts. The analysis followed a three-step process:
- Segmentation: Sentences were broken into individual phrases describing specific weather parameters.
- Alignment: Machine-generated phrases were matched with their human-edited counterparts.
- Labeling: Each pair was categorized as a match, replacement, addition, or deletion.
This granular approach allowed the team to pinpoint specific areas where the system required intervention or where human experts felt the need to adjust the output for clarity or style.
Results
The automated analysis of 73,041 phrases revealed that 60% were perfect matches, requiring no human intervention. This suggests a high baseline of performance for the SUMTIME-MOUSAM system. However, the remaining 30% of phrases that were modified provided critical insights into the types of "errors" or preferences encountered.
As shown in the results, the majority of edits (65%) were related to ellipsis—the addition or deletion of words to improve flow and naturalness. Data-related replacements (changes to specific ranges or directions) accounted for 20%, while lexical replacements (synonym swapping) made up the remaining 15%. The automated alignment tool achieved a 90% success rate, validating the feasibility of this evaluation method.
Discussion and Conclusion
A key finding of this study is that human post-editing is not solely a corrective measure for factual errors. Instead, a significant portion of edits represents "noise" caused by individual stylistic preferences or the linguistic phenomenon of ellipsis. The researchers argue that the volume of post-editing is a direct indicator of a system's "attractiveness" and cost-effectiveness; a system requiring minimal editing is more valuable in a professional workflow.
While post-editing offers a powerful quantitative metric, it is limited by the difficulty of determining the intentionality behind an edit. Future work should combine automated analysis with qualitative feedback from human editors to better distinguish between necessary corrections and stylistic choices. Ultimately, refining NLG systems to better handle stylistic nuances like ellipsis could significantly reduce the human workload in collaborative text generation.
Summary updated