An Analysis of Scoring Methods for Reranking in Large Language Model Story Generation

Michael Deering, Gerald Penn2025 · DOI

Abstract

Outline-conditioned story generation using Large Language Models (LLMs) offers a promising approach for automating narrative creation. Some outline-conditioned story generation methods use automatic scoring during the generation process in order to improve the story quality. However, current research has shown that automatic scoring is not ideal for assessing story quality. This paper evaluates three proposed automatic story-scoring methods to improve the reranking of outputs during the generation process. These scoring methods leverage different prompting strategies and finetuning techniques to enhance the accuracy and relevance of the assessments. By experimenting with these approaches within a beam search framework, we aim to identify the most effective methods for optimizing story-generation outcomes. While we have found no significant overall difference between these methods in terms of their agreement with human ratings during story generation, the overall story ratings by human evaluators are average. These findings motivate the need for improved automatic scoring techniques and datasets while also indicating that simpler, more easily implementable scoring methods for reranking perform comparably to more complex approaches.

Introduction

Generating long, coherent narratives remains a significant challenge for Large Language Models (LLMs). A prevalent strategy to address this is outline-conditioned generation, where a model expands a series of plot points into a full story. To maintain narrative integrity and prevent "drift," researchers often employ reranking during the generation process. In a typical beam search setup, the system generates multiple candidate passages for each plot point and uses an automatic scorer to select the most suitable one before proceeding.

However, traditional automatic metrics like BLEU or ROUGE are inadequate for evaluating creative writing. While LLM-based judges are becoming more common, it remains unclear which scoring strategy—probabilistic, generative, or supervised—most effectively steers the generation toward high-quality results.

Methodology

This study evaluates three distinct automatic scoring techniques integrated into a beam search framework:

  1. Log-likelihood-based scoring (Prompt A): The model assesses a "Yes/No" question (e.g., "Did this event happen in the passage?"). The score is derived from the log-probability assigned to the token "Yes."
  2. Simple prompt-based scoring (Prompt B): The model is directly prompted to provide a numerical rating (1–5) based on specific criteria like relevance and coherence.
  3. Fine-tuned Scorer (Prompt C): A Llama-2-7b model was fine-tuned using LoRA (Low-Rank Adaptation) on the HANNA dataset, which contains 1,050 human-rated stories. This model was specifically trained to predict human scores across dimensions such as empathy, engagement, and complexity.
Comparison of the three prompting strategies used for automatic story scoring: Log-likelihood (A), Simple Prompt (B), and the Target Story format used for the fine-tuned model (C).

The generation process follows a "passages-per-plot-point" approach, where each narrative segment is ranked based on its fit with the current outline point and the preceding context.

Results and Analysis

The evaluation utilized the OpenMEVA benchmark for validation and WritingPrompts (with GPT-4o generated outlines) for human evaluation.

The Training vs. Generation Gap

A key finding is the discrepancy between a model's performance as a judge and its effectiveness as a guide. While the fine-tuned model achieved a Pearson correlation of 0.2281 on OpenMEVA—outperforming BERTScore—it did not result in superior stories when used for reranking. This suggests that the ability to evaluate a completed story does not necessarily translate to the ability to steer a story during the incremental generation process.

Human Evaluation and Preference

Human ratings on Likert scales (1–5) showed no statistically significant difference between the three methods across coherence (~3.37), relevance (~3.21), and empathy (~3.29). However, when forced to rank the outputs, a clear preference emerged for the Simple Prompt-based method (scale 0.884) over Log-likelihood (0.464) and the Fine-tuned model (-0.1349). This indicates that simpler, "off-the-shelf" prompting can be more effective than specialized, expensive fine-tuning for reranking tasks.

The Optimization Paradox

The study observed that different story qualities peak at different training stages. While relevance and coherence generally improve with more training epochs, metrics like complexity and engagement often decline after a certain point due to overfitting.

Trends in various story quality scores across training epochs, illustrating how different narrative dimensions respond to increased training.

Conclusion

The results highlight a significant gap between automatic scoring for evaluation and automatic scoring for reranking. The preference for simpler prompting methods suggests that current fine-tuning approaches, often trained on full stories, may lack the granularity required to score individual passages effectively. Future research should focus on passage-level annotation and exploring techniques like Reinforcement Learning from Human Feedback (RLHF) to better capture the nuances of creative narrative flow.

Summary updated

Abstract

Outline-conditioned story generation using Large Language Models (LLMs) offers a promising approach for automating narrative creation.Some outline-conditioned story generation methods use automatic scoring during the generation process in order to improve the story quality.However, current research has shown that automatic scoring is not ideal for assessing story quality.This paper evaluates three proposed automatic story-scoring methods to improve the reranking of outputs during the generation process.These scoring methods leverage different prompting strategies and finetuning techniques to enhance the accuracy and relevance of the assessments.By experimenting with these approaches within a beam search framework, we aim to identify the most effective methods for optimizing story-generation outcomes.While we have found no significant overall difference between these methods in terms of their agreement with human ratings during story generation, the overall story ratings by human evaluators are average.These findings motivate the need for improved automatic scoring techniques and datasets while also indicating that simpler, more easily implementable scoring methods for reranking perform comparably to more complex approaches.
BibTeX
@misc{deering2025analysis,
  author = {Michael Deering and Gerald Penn},
  title = {{An Analysis of Scoring Methods for Reranking in Large Language Model Story Generation}},
  year = {2025},
  doi = {10.18653/v1/2025.in2writing-1.10},
}
Figure