IFlyEA: A Chinese Essay Assessment System with Automated Rating, Review Generation, and Recommendation

Jiefu Gong, Xiao Hu, Wei Song, Ruiji Fu, Zhichao Sheng, Bo Zhu, Shijin Wang, Ting Liu

Introduction

Automated Essay Assessment (AEA) has traditionally focused on "shallow" features like grammar and vocabulary, often tailored for English learners. However, native Chinese students in primary and junior high schools require more sophisticated feedback. IFlyEA addresses this by providing a comprehensive system that evaluates essays across grammar, rhetoric, and discourse levels. Unlike "black box" models that offer only a score, IFlyEA generates human-like reviews and personalized learning recommendations, making it a pedagogical tool for the classroom.

System Architecture

The IFlyEA system is built on a hierarchical three-tier analytical framework designed to mimic the evaluation process of a human teacher.

The multi-tier analytical architecture of the IFlyEA system.

1. Grammar Level

At the foundational level, the system handles spelling and grammatical errors. Spelling correction uses an unsupervised two-stage approach: a 5-gram language model identifies candidates from a "confusion set," which are then reranked using a BERT-based Masked Language Model (MLM). For broader grammatical diagnosis (redundancy, missing words, etc.), the system employs ResELECTRA, a ResNet-enhanced multi-layer bidirectional transformer.

2. Rhetoric Level

For native speakers, rhetorical flair is a key differentiator. IFlyEA identifies "beautiful sentences" using a BiLSTM with attention trained on 20,000+ crowd-sourced examples. It also recognizes figurative language like similes and personification, the latter of which integrates the Chinese CiLin knowledge base to understand human-to-object attribute mapping.

3. Discourse Level

The system evaluates logical flow and structure using hierarchical multi-task learning. It identifies discourse elements (e.g., thesis, support) and, for narrative essays, recognizes "modes" such as action or dialogue to assess whether a student is effectively "showing" rather than just "telling."

Explainable Feedback and Interface

A core innovation of IFlyEA is its focus on interpretability. Instead of a solitary grade, the system provides an interactive interface where different colors represent various discourse modes or rhetorical devices.

The IFlyEA user interface, demonstrating multi-level feedback including discourse modes, rhetorical devices, and grammatical corrections.

This visualization allows students to see exactly where their writing excels or needs improvement. Furthermore, the system "closes the loop" by generating personalized recommendations, such as cloze tests based on the student's own spelling errors, to reinforce correct usage in context.

Evaluation and Results

The system was evaluated using both public benchmarks (SIGHAN 2015, CGED 2020) and proprietary domain data from primary and junior high schools.

  • Spelling Correction: The unsupervised method achieved an $F_1$ score of 0.651 on SIGHAN 2015, rivaling supervised state-of-the-art models.
  • Grammar Diagnosis: The ResELECTRA model ranked 1st in the NLPTEA-2020 CGED task and achieved a 63% $F_1$ score at the position level on domain-specific data.
  • Rhetoric Recognition: The "beautiful sentence" classifier reached 81% accuracy, while simile and personification recognition achieved $F_1$ scores of 86% and 80%, respectively.

Conclusion

IFlyEA moves beyond simple grading to provide a nuanced, explainable assessment of Chinese essays. While challenges remain in identifying complex discourse elements like a "main idea" in sophisticated essays, the system's focus on rhetoric and pedagogical feedback represents a significant step forward for AEA in native-language contexts. Future work will involve extensive user studies to measure the system's impact on long-term student learning outcomes.

Summary updated

BibTeX
@inproceedings{gong2021iflyea,
  author = {Jiefu Gong and Xiao Hu and Wei Song and Ruiji Fu and Zhichao Sheng and Bo Zhu and Shijin Wang and Ting Liu},
  title = {{IFlyEA: A Chinese Essay Assessment System with Automated Rating, Review Generation, and Recommendation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-demo.29},
}
Figure