Abstract
Automated Essay Scoring (AES) systems have reached high proficiency in evaluating technical writing skills such as grammar and structure but often fail to capture the "spark" of creativity. This paper introduces Representation-to-Creativity (R2C), a novel self-supervised framework designed to identify and score creativity by analyzing essay distributions in high-dimensional embedding space. By leveraging Longformer-based embeddings and Expectation-Maximization (EM) clustering, R2C distinguishes between conventional, creative, and monotonous writing styles. Experimental results demonstrate that R2C achieves up to a 58% improvement in F1-score over state-of-the-art models, suggesting that mathematical representations of originality can provide a more consistent evaluation than subjective human annotation.
Introduction
Traditional AES benchmarks, such as ASAP or AIHUB, utilize rubrics that prioritize technical proficiency over originality. Defining "creativity" mathematically is challenging; most existing AI approaches rely on word-level novelty (e.g., rare tokens), which fails to account for global context and narrative flow. The R2C framework addresses this gap by creating a systematic, self-supervised pipeline to quantify creativity and generate high-quality training data without the need for exhaustive, subjective human labeling.
Methodology
The R2C model follows a four-step self-supervised pipeline to transform raw text into creativity scores:
- Essay-to-Vector (E2V): Essays are converted into high-dimensional vectors using Longformer, which handles long sequences up to 4,096 tokens. These vectors are compressed via Principal Component Analysis (PCA) to highlight structural patterns.
- Clustering: Expectation-Maximization (EM) clustering groups essays. The largest, most central cluster typically represents "conventional" or average writing.
- Automatic Labeling: Creative and non-creative clusters are identified using a creativity-aware distance metric. This metric measures the cross-entropy distance from the "conventional" center, weighted by intra-cluster variance.
- Scoring: A synthetic training set is generated with scores from 1 to 5, which is then used to fine-tune standard language models like BERT or ELECTRA.
Experimental Setup
The study evaluates the R2C framework using three primary datasets:
- ASAP: ~12,900 English essays (grades 7–10).
- AIHUB: ~50,400 Korean essays (elementary to high school).
- Ground Truth $\Lambda$: A curated set of 5,048 essays scored by experts using a 9-trait creativity rubric (novelty, richness, logicality, etc.), achieving a high human agreement (Kendall score of 0.827).
Results and Analysis
R2C-based models significantly outperformed supervised state-of-the-art (SOTA) models in binary classification (creative vs. non-creative), with accuracies reaching 0.951 (D1), 0.888 (D2), and 0.995 (D3).
The Variance Hypothesis
A key insight from the clustering analysis is the Variance Hypothesis: creative essays exhibit high intra-cluster variance. While both creative and non-creative essays appear as outliers relative to the conventional norm, creative essays are diverse in ideas and vocabulary. In contrast, non-creative essays tend to be monotonous and cluster tightly together.
Furthermore, models trained on R2C's automatically generated labels often surpassed those trained on human-annotated ground truth. This suggests that the clustering approach captures a more consistent mathematical representation of originality than subjective human graders.
Conclusion and Future Work
R2C represents a significant step toward automated holistic scoring of creativity. By focusing on the "shape" of an essay's meaning rather than local word frequency, it provides a robust metric for originality. Future work will focus on moving from holistic scores to trait-based feedback (e.g., identifying specific metaphors or structural innovations) and exploring deeper integration with Large Language Models (LLMs) to enhance the feedback loop for students.
Summary updated