Fine-tuning ChatGPT for Automatic Scoring of Written Scientific Explanations in Chinese

Jie Yang, Ehsan Latif, Yuze He, Xiaoming ZhaiJournal of Science Education and Technology 2025 · DOI · OpenReview

TL;DR

This paper explores the effectiveness of fine-tuning ChatGPT for automatically scoring scientific explanations written in Chinese, a logographic language. The study demonstrates that fine-tuned ChatGPT can achieve high accuracy, but its performance is influenced by the complexity of reasoning within student responses, with a nuanced relationship that differs between lower and higher-performing students.

What Problem Does This Paper Address?

Accurately scoring students' written scientific explanations is crucial for assessing their understanding and scientific literacy. However, this process is time-consuming and resource-intensive for human evaluators. While Large Language Models (LLMs) like ChatGPT have shown promise in automating this task, most research has focused on English, an alphabetic language. This paper addresses the gap in understanding whether these LLMs can be effectively adapted for scoring in logographic languages like Chinese, which possess distinct linguistic characteristics and reasoning styles that might pose unique challenges.

Methods

The core of this research involves fine-tuning ChatGPT on a dataset of Chinese scientific explanations. The authors employed a multi-stage approach:

  • Data Preprocessing: This included cleaning the data, ensuring privacy by anonymizing student information, and tokenizing the Chinese text to make it understandable for the LLM. Special attention was paid to preserving the nuances of Chinese language, such as implicit reasoning and idioms.
  • Fine-tuning: ChatGPT was fine-tuned using a regression-based loss function for continuous scores and a classification loss. The learning rate, epochs, and batch size were carefully optimized to handle the complexities of Chinese scientific explanations. Data augmentation techniques were also used to improve the model's ability to handle multi-step reasoning.
  • Evaluation: The fine-tuned model's performance was evaluated using metrics like Mean Absolute Error (MAE) and classification accuracy. A Kendall correlation analysis was used to assess the relationship between reasoning complexity and scoring accuracy.
  • Qualitative Analysis: A manual matching approach was used to identify linguistic features in student responses that might account for scoring differences, particularly in cases of misscoring.

Datasets

The study utilized a dataset of 7626 scientific explanations written by middle school and high school students in Beijing across seven different science tasks. These tasks were developed based on Chinese curriculum standards and varied in their scenario type (daily life vs. lab contexts) and task type (explanation vs. prediction and explanation). The dataset was split into training and testing samples for each item, with the number of labels varying per item.

Key Insights and Ideas

  • LLMs for Chinese Scientific Explanations: Fine-tuned ChatGPT demonstrates significant potential for automatically scoring scientific explanations in Chinese, a logographic language, bridging a gap in current research.
  • Reasoning Complexity Matters (and Differently): The relationship between reasoning complexity and scoring accuracy is not linear. For lower-performing students, higher reasoning complexity correlates with lower scoring accuracy, suggesting LLMs struggle with implicit or complex reasoning in these cases. Conversely, for higher-performing students, higher reasoning complexity correlates with higher scoring accuracy, indicating the model can better recognize sophisticated reasoning when it's well-articulated.
  • Linguistic Nuances are Key: The distinct linguistic features of Chinese, such as implicit expression, circularity, and a richer vocabulary, present challenges for LLMs. The study highlights how sentence structure, word choice, and the use of technical terms significantly influence scoring accuracy.
  • Simplicity vs. Comprehensiveness Trade-off: The paper suggests a tension between the simplicity and clarity of language (which aids LLM parsing) and the comprehensiveness required for complex scientific explanations. This trade-off impacts how accurately LLMs score responses.

Key Results

  • Scoring Accuracy: The fine-tuned ChatGPT models achieved scoring accuracy above 0.75 for all scoring categories across the seven items, with some items (Items 6 and 7) showing overall accuracy above 90%. However, performance varied by scoring element (Data, Theory, Reasoning, Phenomenon), with the Phenomenon element generally showing the highest average accuracy.
  • Reasoning Complexity Correlation:
    • Lower-level group: A moderate negative correlation (average Kendall's τ = -0.31) was found between reasoning complexity and scoring accuracy, indicating that as reasoning complexity increased, scoring accuracy decreased.
  • Higher-level group: A positive correlation (average significant Kendall's τ = 0.19) was observed, suggesting that higher reasoning complexity was associated with higher scoring accuracy.
  • Qualitative Findings: The analysis of linguistic features revealed that lower-level, correctly scored responses were shorter, used simpler language, and had straightforward sentence structures. In contrast, lower-level, misscored responses were longer, used more technical terms (sometimes inaccurately), and featured complex syntactic structures, posing parsing challenges. For higher-level responses, correctly scored ones were longer, used more technical terms and symbols, and had complex sentence structures with strong internal logic, while misscored ones were more concise but lacked depth or precise terminology.

Limitations and Future Work

  • Exploratory Nature: The study acknowledges that the analyses of linguistic features and their association with scoring accuracy were exploratory and calls for more structured quantitative frameworks.
  • Model Specificity: The findings are specific to ChatGPT. Future research should explore the performance of other LLMs and validate these findings across different models.
  • Cross-Cultural and Linguistic Generalization: While the study extends LLM scoring to Chinese, further research is needed to explore its applicability in other linguistic and cultural contexts.
  • Ethical Considerations: The authors emphasize the importance of addressing ethical concerns such as bias, privacy, and data security when integrating AI tools like ChatGPT into educational settings.
  • Rubric Design: The study suggests that the quality of human judgment in model training and rubric design is crucial, implying future work could focus on optimizing these aspects.

Summary updated

Abstract

The development of explanations for scientific phenomena is essential in science assessment, but scoring student-written explanations remains challenging and resource-intensive. Large language models (LLMs) have shown promise in addressing this issue, particularly in alphabetic languages like English. However, their applicability to logographic languages is less explored. This study investigates the potential of fine-tuning ChatGPT, a leading LLM, to automatically score scientific explanations written in Chinese. Student responses to seven scientific explanation tasks were collected and automatically scored, with scoring accuracy examined in relation to reasoning complexity using the Kendall correlation. A qualitative analysis explored how linguistic features influenced scoring accuracy. The results show that domain-specific adaptation enables ChatGPT to score Chinese scientific explanations with accuracy. However, scoring accuracy correlates with reasoning complexity: a negative correlation for lower-level responses and a positive one for higher-level responses. The model overrates complex reasoning in low-level responses with intricate sentence structures and underrates high-level responses using concise causal reasoning. These correlations stem from linguistic features--simplicity and clarity enhance accuracy for lower-level responses, while comprehensiveness improves accuracy for higher-level ones. Simpler, shorter responses tend to score more accurately at lower levels, whereas longer, information-rich responses yield better accuracy at higher levels. These findings demonstrate the effectiveness of LLMs in automatic scoring within a Chinese context and emphasize the importance of linguistic features and reasoning complexity in fine-tuning scoring models for educational assessments.
BibTeX
@article{yang2025fine,
  author = {Jie Yang and Ehsan Latif and Yuze He and Xiaoming Zhai},
  title = {{Fine-tuning ChatGPT for Automatic Scoring of Written Scientific Explanations in Chinese}},
  journal = {Journal of Science Education and Technology},
  year = {2025},
  doi = {10.1007/s10956-025-10199-z},
}
Figure