Abstract
Open-source Large Language Models (LLMs) frequently suffer from "repetition loops," a phenomenon often mitigated by the Repetition Penalty Parameter (RPP). However, aggressive RPP tuning can degrade model performance. This paper introduces Repetition-Aware Performance (RAP), a novel metric that integrates a repetition penalty into standard evaluation scores. By utilizing the Repetition Detection Algorithm (ReDA), RAP enables the identification of an optimal RPP that minimizes repetition while preserving accuracy across various tasks and models.
Introduction
The "repetition problem" is a well-known failure mode in LLMs, where models output redundant characters or sequences indefinitely. While the Repetition Penalty Parameter (RPP) is used during sampling to discourage token reuse, its application involves a delicate trade-off. Low RPP values lead to coherent but repetitive outputs, whereas high values can cause fragmentation, loss of fluency, or failure to generate complete answers. Standard evaluation metrics like F1, BLEU, or BERTScore are "repetition-blind," failing to penalize models that produce correct answers buried within repetitive noise. RAP addresses this by providing a unified score that accounts for both accuracy and repetition.
Methodology
The RAP framework consists of two primary components: the Repetition Detection Algorithm (ReDA) and the RAP metric itself.
- ReDA: This algorithm employs regular expressions to detect two types of repetition: Non-Word Character (NWC) repetitions (e.g., strings of punctuation or whitespace) and alphanumeric text repetitions of five or more characters. It calculates a Repetition Ratio (RR), representing the proportion of repeated content relative to the total text length.
- RAP Metric: The core metric is defined as $RAP = P \times (1 - RR)^3$, where $P$ is a standard performance score (like F1 or BERTScore).
The cubic penalty ensures that even moderate repetition significantly impacts the final score, effectively guiding researchers toward the "Goldilocks zone" where repetition is minimized without sacrificing accuracy.
Experimental Setup
The researchers evaluated 12 open-source LLMs, including the Llama-2, Llama-3, Gemma, Phi, and Mistral families, ranging from 2B to 70B parameters. Performance was assessed across three primary datasets:
- WebQSP: Knowledge Base Question Answering.
- MS MARCO: Reading comprehension and QA.
- MAC: Chinese-English machine translation.
The study also compared generic prompts against model-specific chat templates and explored the impact of Retrieval-Augmented Generation (RAG).
Analysis and Results
A key finding is the existence of a clear "sweet spot" for RPP. For instance, on the WebQSP dataset, Llama-3-8B achieved a 93.1% reduction in repetition with only a 3.7% decrease in raw performance.
Key insights from the evaluation include:
- Chat Templates as De-loopers: Using a model's specific chat template significantly reduces repetition compared to generic prompts, likely due to instruction-tuning effects.
- Dataset Sensitivity: Entity-heavy tasks like WebQSP trigger higher repetition levels than reading comprehension tasks like MS MARCO.
- Cubic Advantage: Ablation studies confirmed that the cubic penalty function $(1-RR)^3$ provides the most robust balance across all datasets for identifying optimal RPP settings.
Conclusion
RAP provides a systematic, automated way to tune LLM sampling parameters. While currently focused on exact character matches, it represents a significant advancement in evaluating model reliability. Future work will explore semantic repetition detection and the interaction of RAP with other sampling parameters such as temperature and top-p.
Summary updated