This paper explores a "data-centric AI" approach to empathy computing, demonstrating that Large Language Models (LLMs) are often more reliable than crowdsourced human workers for labeling subjective emotional data. By using LLMs in vitro—as high-quality teachers to clean and augment training data for smaller, more efficient models—the authors achieved state-of-the-art performance on empathy detection benchmarks.
TL;DR
Instead of using LLMs for direct inference, this paper uses them to "de-noise" messy crowdsourced datasets and generate synthetic training labels. By applying a psychologically grounded "scale-aware" prompting strategy, the authors fine-tuned smaller models (like RoBERTa) to reach a state-of-the-art Pearson correlation of 0.648 on the NewsEmp empathy benchmark.
What Problem Does This Paper Address?
Empathy detection—predicting how much empathy a person feels based on their written response to a news story—is notoriously difficult because it is highly subjective. Most existing datasets rely on crowdsourced "self-annotations" (e.g., from Amazon Mechanical Turk), which are often plagued by noise due to inattentive workers or "garbage in, garbage out" data collection.
While LLMs are great at zero-shot tasks, deploying them "in vivo" (directly for every request) is computationally expensive and slow. The authors address two main challenges:
- Label Noise: How to identify and correct incorrect human labels in existing datasets.
- Data Scarcity: How to expand training sets without expensive human intervention.
Methods
The authors propose an "in vitro" framework where LLMs (Llama 3 70B and GPT-4) act as expert annotators during the preprocessing phase.
- Scale-Aware Prompting: Rather than asking the LLM for a single "empathy score," the authors designed prompts based on Batson’s Empathy Scale. The LLM rates six specific sub-dimensions (sympathetic, moved, compassionate, tender, warm, and soft-hearted). These are averaged to produce a final label, mirroring the rigorous protocol used in psychology.
- Application 1: Noise Mitigation: They compared LLM labels to human labels. If the difference exceeded a threshold ($\alpha$), the human label was replaced by the LLM label.
- Application 2: Data Augmentation: They used the LLM to label unlabelled essays, effectively expanding the training pool for smaller Pre-trained Language Models (PLMs).
- Metric Advocacy: The authors argue that the standard Pearson Correlation Coefficient (PCC) is insufficient because it only measures trends, not absolute error. They advocate for the Concordance Correlation Coefficient (CCC), which penalizes predictions that are far from the actual magnitude of the target value.
Datasets
The study utilizes the NewsEmp series (NewsEmp22, NewsEmp23, and NewsEmp24). These datasets consist of essays written by people in response to news articles about harm or distress, paired with empathy scores.
- Key Finding on Data: The authors discovered significant data leakage in previous studies (overlap between NewsEmp23 test sets and NewsEmp24 validation sets) and provided a corrected evaluation framework to ensure fair comparison.
Key Insights and Ideas
- LLMs as Consistent Teachers: LLMs showed much higher inter-rater reliability (Krippendorff’s Alpha of 0.80–0.99) compared to the low agreement between LLMs and crowdsourced humans (0.19–0.27). This suggests the human labels were the primary source of noise.
- The "Aha" Moment in Clustering: Using t-SNE visualizations, the authors showed that models trained on LLM-refined labels produced much tighter, more distinct feature clusters (higher Silhouette scores) than those trained on raw crowdsourced data.
- Expert Validation: In a blind human preference study, experts preferred LLM-generated labels over the original crowdsourced labels in 18 out of 20 high-discrepancy cases.
- Demographic Bias: Even "demographic-unaware" prompts resulted in varying performance across different races and income levels, highlighting that LLMs may inherit systemic biases from their training data that propagate into the labels they generate.
Key Results
- State-of-the-Art (SOTA): A RoBERTa model trained on the LLM-corrected NewsEmp24 dataset achieved a PCC of 0.648, outperforming the previous best of 0.629.
- Efficiency: By using the LLM in vitro, the final deployment model is a standard RoBERTa-base (125M parameters), which is orders of magnitude smaller and faster than the LLMs used to train it.
- Threshold Sensitivity: The best results were found when replacing labels only when the LLM and human disagreed by a moderate amount ($\alpha \approx 3.5$ to $4.5$ on a 7-point scale), suggesting that some human subjectivity is worth keeping, but extreme outliers are likely errors.
Limitations and Future Work
- Reasoning: The current approach only asks for scores. Future work could involve Chain-of-Thought (CoT) prompting to make the empathy labels more explainable.
- Language: The study was limited to English-language datasets; the authors suggest exploring the multilingual capabilities of LLMs for empathy in other cultures.
- Bias Mitigation: While the "in vitro" approach improves accuracy, it doesn't automatically solve demographic bias. The authors call for more research into equitable empathy computing that accounts for diverse human backgrounds.
Summary updated