TL;DR
This paper investigates the effectiveness of prompt engineering for sequence labeling tasks in Computational Literary Studies (CLS). The authors find that LLM performance is sensitive to data splits and prompt phrasing, and that common prompt components don't consistently improve results, casting doubt on the immediate utility of zero-shot LLM prompting for these tasks.
What Problem Does This Paper Address?
Computational Literary Studies (CLS) aims to apply data-driven methods, including NLP and machine learning, to analyze literary texts. A recent development is "prompt engineering," where large language models (LLMs) are instructed via natural language prompts to perform tasks without explicit fine-tuning. This approach promises to make CLS more accessible by allowing researchers to express complex questions in natural language. However, the reliability and generalizability of this method, especially for sequence labeling tasks common in CLS (like identifying emotions or events in text), remain underexplored. This paper addresses the critical question of whether prompt engineering is a robust and dependable method for CLS, or if its performance is too brittle and task-specific.
Methods
The authors employ a systematic evaluation of LLMs on two CLS-relevant sequence classification tasks: emotion detection and event detection. Their methodology focuses on three key research questions:
- Generalizability of Performance Measurements: They test how consistent performance is across different data splits of the same dataset.
- Robustness against Meaning-Preserving Prompt Variations: They investigate how sensitive LLM performance is to minor changes in prompt wording (paraphrasing).
- Generalizability of Prompt Component Optimization: They analyze whether commonly recommended prompt components (like assigning a role to the LLM or using phrases like "think step-by-step") consistently improve performance across different models and tasks.
They use a batch-use scenario, where prompts are applied to a large quantity of data for automatic detection, adhering to machine learning best practices like splitting data into train and test sets.
Datasets
The study uses two publicly available datasets:
- REMAN (Relational Emotion Annotation for Fiction): Contains annotations of emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, trust) in English literary texts.
- Event Dataset: Contains annotations of event types (process, stative, change of state, non-event) in German literary texts from the TextGrid and d-prose repositories.
The authors note that these datasets are publicly available, and it's possible they were included in the pre-training data of the LLMs used.
Key Insights and Ideas
- Prompt Brittleness is Real: LLM performance can be surprisingly sensitive to small, meaning-preserving changes in prompts, leading to inconsistent results and potentially different model rankings.
- Data Splits Matter: Even with the same model and prompt, performance can vary significantly across different test data splits, suggesting that LLMs may not be generalizing as robustly as hoped.
- "Magic" Prompt Components Aren't So Magical: Commonly suggested prompt components (like "think step-by-step" or "bribe" prompts) do not consistently improve performance across all models and tasks. Their effectiveness appears highly context-dependent.
- CLS Tasks are Challenging for Zero-Shot LLMs: The LLMs tested struggled to achieve high performance on these CLS sequence labeling tasks in a zero-shot setting, questioning the immediate applicability of this approach for complex literary analysis.
Key Results
- Performance Variability: All tested LLMs showed statistically significant differences in performance between the two test data splits for both emotion and event tasks. Mean differences in F1 scores ranged from 4.2% to 10.9%.
- Robustness Findings: Standard deviations of F1 scores across prompt variations (for the emotion task) ranged from 1.96% to 5.31%, and for the event task, from 2.48% to 6.53%. While seemingly small, these deviations can lead to substantial differences in performance. LLAMA showed the most robustness for the emotion task, and SAUERKRAUT for the event task.
- Prompt Component Effectiveness: The analysis of prompt components revealed that only a few components appeared frequently in the best-performing prompts. For instance, the "steps" component was beneficial for LLAMA but detrimental for SAUERKRAUT. Overall, no generalizable recommendation for prompt components could be derived.
- Overall Performance: Performance scores for the emotion task were generally lower than for the event task. GPT performed best for emotion, and MIXTRAL for event. The best F1 scores achieved were around 27-32%, which are lower than typical benchmarks on more general NLP tasks.
Limitations and Future Work
- Model Architectures: The proprietary nature of many LLM architectures makes it difficult to fully understand their internal workings and limitations.
- Theory Dependency of Data: The evaluation data and annotation guidelines are influenced by theoretical perspectives, and alternative guidelines could lead to different results.
- Statistical Representativeness of Data Splits: The study used only two test splits, and it's unclear if more splits would yield different conclusions.
- Training Data Contamination: The possibility that the LLMs' training data included the datasets used for evaluation cannot be ruled out.
- Answer-Space Mapping: It's unclear how the LLMs' internal representations map to the predefined class labels used by domain specialists.
- Task Relevance: The tasks (emotion and event detection) are foundational but not direct literary research questions. Future work could explore more abstract, complex CLS questions.
- Need for Annotated Data: The authors conclude that for specific CLS tasks, using annotated reference data and experimenting with different prompt configurations remains the most reliable way to find optimal solutions, despite the impossibility of exhaustive searching.
Summary updated