Representation-to-Creativity (R2C): Automated Holistic Scoring Model for Essay Creativity

Deokgi Kim, Joonyoung Jo, Byung-Won On, Ingyu LeeNAACL 2025 · DOI

Abstract

Automated Essay Scoring (AES) systems have reached high proficiency in evaluating technical writing skills such as grammar and structure but often fail to capture the "spark" of creativity. This paper introduces Representation-to-Creativity (R2C), a novel self-supervised framework designed to identify and score creativity by analyzing essay distributions in high-dimensional embedding space. By leveraging Longformer-based embeddings and Expectation-Maximization (EM) clustering, R2C distinguishes between conventional, creative, and monotonous writing styles. Experimental results demonstrate that R2C achieves up to a 58% improvement in F1-score over state-of-the-art models, suggesting that mathematical representations of originality can provide a more consistent evaluation than subjective human annotation.

Introduction

Traditional AES benchmarks, such as ASAP or AIHUB, utilize rubrics that prioritize technical proficiency over originality. Defining "creativity" mathematically is challenging; most existing AI approaches rely on word-level novelty (e.g., rare tokens), which fails to account for global context and narrative flow. The R2C framework addresses this gap by creating a systematic, self-supervised pipeline to quantify creativity and generate high-quality training data without the need for exhaustive, subjective human labeling.

Methodology

The R2C model follows a four-step self-supervised pipeline to transform raw text into creativity scores:

  1. Essay-to-Vector (E2V): Essays are converted into high-dimensional vectors using Longformer, which handles long sequences up to 4,096 tokens. These vectors are compressed via Principal Component Analysis (PCA) to highlight structural patterns.
  2. Clustering: Expectation-Maximization (EM) clustering groups essays. The largest, most central cluster typically represents "conventional" or average writing.
  3. Automatic Labeling: Creative and non-creative clusters are identified using a creativity-aware distance metric. This metric measures the cross-entropy distance from the "conventional" center, weighted by intra-cluster variance.
  4. Scoring: A synthetic training set is generated with scores from 1 to 5, which is then used to fine-tune standard language models like BERT or ELECTRA.
The R2C pipeline: from essay embedding to automatic scoring.

Experimental Setup

The study evaluates the R2C framework using three primary datasets:

  • ASAP: ~12,900 English essays (grades 7–10).
  • AIHUB: ~50,400 Korean essays (elementary to high school).
  • Ground Truth $\Lambda$: A curated set of 5,048 essays scored by experts using a 9-trait creativity rubric (novelty, richness, logicality, etc.), achieving a high human agreement (Kendall score of 0.827).

Results and Analysis

R2C-based models significantly outperformed supervised state-of-the-art (SOTA) models in binary classification (creative vs. non-creative), with accuracies reaching 0.951 (D1), 0.888 (D2), and 0.995 (D3).

The Variance Hypothesis

A key insight from the clustering analysis is the Variance Hypothesis: creative essays exhibit high intra-cluster variance. While both creative and non-creative essays appear as outliers relative to the conventional norm, creative essays are diverse in ideas and vocabulary. In contrast, non-creative essays tend to be monotonous and cluster tightly together.

Clustering results across datasets D1, D2, and D3, illustrating the distribution of essay types in embedding space.

Furthermore, models trained on R2C's automatically generated labels often surpassed those trained on human-annotated ground truth. This suggests that the clustering approach captures a more consistent mathematical representation of originality than subjective human graders.

Conclusion and Future Work

R2C represents a significant step toward automated holistic scoring of creativity. By focusing on the "shape" of an essay's meaning rather than local word frequency, it provides a robust metric for originality. Future work will focus on moving from holistic scores to trait-based feedback (e.g., identifying specific metaphors or structural innovations) and exploring deeper integration with Large Language Models (LLMs) to enhance the feedback loop for students.

Summary updated

BibTeX
@inproceedings{kim2025representation,
  author = {Deokgi Kim and Joonyoung Jo and Byung-Won On and Ingyu Lee},
  title = {{Representation-to-Creativity (R2C): Automated Holistic Scoring Model for Essay Creativity}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.292},
}
Figure