Abstract: This paper aims to investigate the main writing features in the English time-limited compositions by third-year English major students at Guangxi University in China. Three automated assessing tools were adopted: Juku automated scoring system designed in China; Coh-Metrix automated computational evaluation tool constructed by researchers in the USA; and Linguistic Inquiry Word Count (LIWC). The results showed that there are great differences in five dimensions in the writings of high and low score compositions based on the calculation of Juku and Coh-Metrix, and there exists a strong correlation between Chinese English majors' argumentative writings and formal texts standardized by LIWC.
Introduction
Evaluating student writing is notoriously subjective and labor-intensive for human instructors. While automated scoring systems like China’s "Juku" are widely used, there is often a "black box" element to how these scores are generated. This study utilizes a trio of automated NLP tools—Juku, Coh-Metrix, and LIWC—to dissect the linguistic "DNA" of high-scoring English argumentative essays. The research reveals that the secret sauce of a high-scoring composition isn't just correct spelling, but rather a sophisticated combination of referential cohesion, lexical diversity, and a stylistic alignment with formal academic prose.
Methodology
The researchers analyzed a corpus of 150 English argumentative compositions written by 45 third-year English major students at Guangxi University, China. The study employed a multi-tool analytical framework:
- Juku Automated Scoring System: Used to provide overall scores and categorize essays into high, intermediate, and low-score groups.
- Coh-Metrix (Version 3.0): Analyzed 108 indices related to text cohesion and coherence, such as "left embeddedness" and "referential cohesion."
- Linguistic Inquiry Word Count (LIWC): Categorized words into psychological and structural groups to determine stylistic tendencies.
Before analysis, all spelling mistakes were manually corrected to isolate structural and stylistic quality from simple orthographic errors.
Results and Analysis
The study identified five specific dimensions where high and low scores diverged significantly. Even with spelling errors removed, high-scoring essays exhibited superior structural and lexical sophistication.
The Five Dimensions of Divergence
- Referential Cohesion: High-scorers excelled at overlapping nouns and stems across sentences (Items 28–33). This maintains a consistent thread of thought, making the text easier to follow.
- Lexical Diversity: Measures like the Type-Token Ratio (TTR) and VOCD (vocabulary variety) were significantly higher in top-tier essays.
- Syntactic Complexity: High-score essays featured more modifiers per noun phrase and more words appearing before the main verb.
- Syntactic Pattern Density: Top essays had a much higher incidence of prepositional phrases and passive voice, hallmarks of objective academic writing.
- Word Information: High-scorers used words with higher "concreteness" and "imagability."
The "Formal" Signature
High-scoring students did not just write "better" English; they wrote different English. Their lexical choices showed a high correlation with formal academic texts rather than personal or conversational ones. They used more articles and "big words" (more than 6 letters).
As shown in the correlation matrix, the lexical use tendency of the high-score group had a 0.989 correlation with formal text standards, suggesting these students have successfully internalized the requirements of academic discourse.
Conclusion
The findings suggest that high-scoring argumentative writing is characterized by high referential cohesion and a stylistic alignment with formal prose. While this study is limited by its sample size and reliance on the Juku scoring benchmark, it provides a granular look at the linguistic features that define successful non-native academic writing. Future research should compare these automated indices against manual rubrics from expert instructors to further validate these computational findings.
Summary updated