Abstract
In the landscape of informal online communication, users frequently employ Repetitive Lengthening Forms (RLF)—such as "loooove" or "great!!!!"—to convey heightened sentiment and nuance. Despite their prevalence, RLF has remained a neglected feature in natural language processing (NLP) research compared to emojis or memes. This paper addresses this gap by introducing Lengthening, the first large-scale multi-domain dataset specifically focused on RLF, and ExpInstruct, an explainable instruction-tuning framework. Our results demonstrate that while standard models often achieve high accuracy through superficial patterns, ExpInstruct enables open-source models like LLaMA2 to match the performance and explainability of GPT-4, bridging the "explainability gap" in informal sentiment analysis.
Introduction
Online text is rarely "standard." To express intensity, users often deviate from formal grammar through RLF. While these forms are intuitive to humans, they pose a challenge for automated systems: is the lengthening a significant sentiment signal or merely linguistic noise? Furthermore, do current Language Models (LMs) truly understand the emphatic nature of these expressions, or are they exhibiting "Clever Hans" effects—arriving at the correct sentiment label without identifying the actual source of intensity?
The authors argue that the lack of specialized datasets and evaluation metrics has hindered our ability to assess whether a model understands that the extra characters in "sooooo" are the primary drivers of sentiment. To solve this, they present a comprehensive study involving a massive new dataset and a novel tuning framework designed for explainability.
Methodology and Framework
The research introduces a multi-stage approach to quantify and improve model understanding of RLF.
The Lengthening Dataset Pipeline
The researchers developed a regex-based extraction pipeline to identify RLF instances across diverse web corpora. This pipeline identifies lengthened words, maps them back to their root forms (e.g., "goooood" to "good"), and pairs them with control sentences from the same context that lack RLF. This allows for a controlled study of the specific impact of lengthening on sentiment.
ExpInstruct: Explainable Instruction Tuning
To move beyond "black-box" predictions, the authors proposed ExpInstruct, a two-stage framework for Large Language Models (LLMs):
- Chain-of-Thought (CoT) Generation: Using GPT-4, the system generates Word Importance Scores (WIS). The model is prompted to decompose sentences and assign scores (1-5) to words based on their contribution to the overall sentiment.
- Instruction Fine-tuning: Open-source models (like LLaMA2) are fine-tuned on these explanation-sentiment pairs. This forces the model to learn not just the label, but the rationale behind it.
Unified Explainability Metric ($S_{exp}$)
To compare different model architectures (e.g., gradient-based PLMs vs. prompt-based LLMs), the authors introduced $S_{exp}$. This normalized metric quantifies the "importance" a model assigns specifically to the RLF word within a sentence, providing a direct measure of linguistic understanding.
The Lengthening Dataset
The Lengthening dataset is a massive benchmark containing 850,000 samples. It draws from four major sources: Amazon Reviews (Books & Electronics), Yelp (Restaurants), TripAdvisor (Hotels), and Twitter.
The dataset categorizes RLF into two primary types:
- Repetitive Letters: (e.g., "loooove", "sooooo")
- Repetitive Punctuation: (e.g., "wow!!!!", "really??")
Analysis of the dataset reveals that RLF appears in approximately 5.8% of documents on average, with some domains like Twitter or Yelp exceeding 13%, highlighting its significance in real-world data.
Key Insights and Results
The study yielded several critical findings regarding how models process informal emphasis.
The Explainability Gap
A significant discovery was the "explainability gap." While smaller fine-tuned models like RoBERTa-Large achieved high accuracy (~86%), their $S_{exp}$ scores were often low. This suggests they were making correct predictions based on context rather than identifying the RLF word as the source of emphasis. In contrast, GPT-4 showed high zero-shot explainability ($S_{exp}$ of 0.38).
Performance of ExpInstruct
By applying the ExpInstruct framework with only 1,600 high-quality explainable samples, the authors enabled LLaMA2-13B to reach an $S_{exp}$ of 0.35, nearly matching GPT-4. This demonstrates that targeted instruction tuning can significantly enhance a model's ability to "reason" about informal linguistic features.
Punctuation vs. Letters
The results indicate that repetitive punctuation generally provides a stronger and more consistent sentiment signal across all tested models compared to repetitive letters. Furthermore, sentences containing RLF were found to be highly expressive, often serving as accurate proxies for the sentiment of an entire multi-sentence document.
Conclusion and Future Work
This research highlights RLF as a vital but overlooked component of sentiment analysis. By providing the Lengthening dataset and the ExpInstruct framework, the authors offer a path toward more robust and explainable NLP models. Future work includes expanding the study to multilingual contexts (e.g., Spanish or Romanian) and incorporating human-in-the-loop corrections to further refine the quality of generated explanations.
Summary updated