Abstract
Automated Creative Essay Scoring (ACES) presents a unique challenge due to the subjective nature of "originality" in writing. Traditional Large Language Models (LLMs) often overlook the infrequent, expressive tokens that characterize creative work. This paper introduces ramBERT (rare token masking based BERT), a model designed to prioritize these "rare tokens" during pre-training. By evaluating five distinct rare token extraction methods, we demonstrate that latent semantic clustering ($f_3$) provides the strongest signal for creativity, yielding a classification accuracy of 84.2%—a significant improvement over standard random masking baselines.
Introduction
While Automated Essay Scoring (AES) has matured significantly, the evaluation of creativity remains elusive. Standard models like BERT utilize Masked Language Modeling (MLM) with random token selection. However, creativity often resides in the "long tail" of language—the unique word choices and semantic nuances that are statistically rare. The authors propose that by forcing a model to predict these rare tokens during pre-training, it develops a heightened sensitivity to the expressive "spark" that distinguishes creative writing from standard prose.
Methodology: The ramBERT Framework
The core innovation of this research is the ramBERT architecture. Unlike standard BERT, which masks tokens at random, ramBERT identifies and masks specific "rare tokens" to enhance the model's understanding of originality.
To identify these tokens, five extraction strategies were tested:
- BPE ($f_1$): Selecting subword units with the lowest global frequency.
- IDF ($f_2$): Identifying words that appear rarely across a large corpus (Wikipedia).
- Latent Semantic Clustering ($f_3$): Using internal representations to cluster words and picking those in isolated clusters, representing semantic uniqueness.
- SpanBERT ($f_4$): Identifying rare multi-word phrases or "spans."
- Expert Dictionaries ($f_5$): Using pre-existing lists of rare words (e.g., Stanford or Cambridge sets).
Experimental Setup
The model was pre-trained on a corpus of over 4 million Wikipedia documents to establish a baseline for linguistic rarity. For evaluation, the researchers utilized Prompt 1 of the ASAP (Automated Student Assessment Prize) dataset, with 800 essays labeled as "creative" or "non-creative" by domain experts. Higher-scoring essays in the original dataset were found to correlate strongly with these expert creativity labels.
Results and Performance Analysis
The experimental results confirm that the choice of masking strategy significantly impacts the model's ability to detect creativity. The baseline BERT model achieved an accuracy of 74%. In contrast, the ramBERT variants showed marked improvements across nearly all extraction methods.
The latent semantic clustering method ($f_3$) emerged as the most effective, reaching 84.2% accuracy. This suggests that creativity is best captured not just by word frequency, but by the unique contextual use of language. Interestingly, the expert dictionary approach ($f_5$) provided minimal gains (74.3%), highlighting the limitations of static word lists compared to dynamic, contextualized representations.
Discussion and Limitations
The study establishes a clear correlation between the density of rare tokens and human-rated creativity. However, several limitations remain. The current model focuses primarily on expression (how things are said) rather than content (what is said). Furthermore, the reliance on a relatively small dataset of 800 essays and the inherent subjectivity of human raters suggest that further scaling and cross-genre validation are necessary. The computational cost of clustering-based pre-processing also remains higher than standard random masking.
Conclusion
By shifting the focus of pre-training toward the "long tail" of language, ramBERT provides a robust framework for quantifying originality in student writing. Future work will explore the integration of content-based metrics and the application of rare token masking to broader natural language understanding tasks.
Summary updated