Annotation of Positive vs Negative User Interactions for Social Sign Prediction

Biancamaria Bombino, Chiara Boldrini, Andrea Passarella, Marco ContiarXiv 2026 · arXiv

TL;DR

Traditional social network analysis uses text sentiment to guess whether online interactions are positive or negative, often confusing what people are talking about with how they treat each other. This paper demonstrates that zero-shot Large Language Models (LLMs) can successfully decouple relationship dynamics from topic sentiment by explicitly identifying interaction-level relational signals: personal praise and personal attacks.

What Problem Does This Paper Address?

To map signed social networks (graphs showing who trusts, cooperates, or clashes with whom), researchers typically rely on sentiment analysis of user messages. However, this introduces the relation-content conflation problem. Sentiment analysis measures content valence (the emotional tone of the topic) rather than relational valence (the nature of the exchange between users).

For example, two close friends passionately discussing a tragic news event will trigger a "negative" sentiment label, while passive-aggressive rivals might use polite, neutral language to mask hostility. To date, no systematic methodology has successfully isolated the actual social relationship from the semantic content flowing through it. This paper tackles this gap by shifting the focus from general sentiment to direct interpersonal gestures: personal praise and personal attacks.

Methods

The authors frame sign prediction as two independent binary classification tasks: determining whether a textual interaction constitutes (1) a personal praise or (2) a personal attack directed at the interlocutor.

They evaluate four LLMs in a zero-shot setting to see if they can capture these subtle linguistic cues without task-specific fine-tuning. The models span both proprietary architectures (GPT-4o and GPT-5.4-mini) and open-weight alternatives (Qwen2.5:7b and Gemma2:9b).

To understand how instruction detail impacts performance, they test three prompt designs of increasing complexity:

  1. Minimal: Contains only the core task definition.
  2. Intermediate: Adds explicit instructions regarding the directionality of the gesture (e.g., distinguishing an attack on the interlocutor from an attack on a third party).
  3. Structured: Provides detailed instructions alongside explicit positive and negative examples.

Additionally, they explore a cascading ensemble strategy that routes uncertain predictions through a chain of different models based on their error complementarity.

Datasets

Because standard sentiment datasets fail to isolate relational gestures, the authors built two new balanced benchmarks using human annotators and majority voting:

  • Praise Dataset (298 texts): Sourced from the Reddit-based GoEmotions corpus. The authors filtered for comments expressing positive sentiment and "admiration," then had 7 human annotators re-label them based on whether they contained direct personal praise (149 praise, 149 non-praise).
  • Personal Attacks Dataset (340 texts): Compiled from Wikipedia talk pages (editor criticisms) and specific Reddit subreddits like r/roastme. 5 human annotators labeled the texts (170 attacks, 170 non-attacks).

An interesting note on annotation complexity: Humans achieved a complete consensus on 57.35% of the personal attack cases but only 20.80% of the praise cases, confirming that praise is inherently more subjective and context-dependent.

Key Insights and Ideas

  • The Asymmetry of Social Gestures: Attack detection and praise detection behave entirely differently. Hostility carries explicit linguistic markers that LLMs easily recognize regardless of prompt framing. Praise is highly subjective, making its detection sensitive to both model choice and prompt engineering.
  • More Instructions $\neq$ Better Performance: For praise detection, the intermediate prompt performed best. The highly detailed structured prompt actually introduced noise and degraded the models' ability to generalize. For attack detection, the minimal prompt reigned supreme, proving that extra constraints can restrict an LLM's natural classification strengths.
  • Prompting Bridges the Capacity Gap: In praise detection, the smaller open-weight model Qwen2.5:7b achieved highly competitive results when paired with the intermediate prompt, proving that smart prompt engineering can partially compensate for smaller model capacity.
  • Ensembles Aren't Worth the Complexity: The cascading ensemble strategy yielded only marginal improvements for attack detection and failed to improve praise detection. The authors conclude that the added engineering overhead is rarely justified in practice.

Key Results

  • Zero-Shot Viability: Without any task-specific training data, zero-shot LLMs established a strong, practical baseline for relational annotation.
  • Praise Detection: GPT-5.4-mini achieved the best overall results across all prompt types. Model errors heavily clustered around low-human-agreement instances; for instance, 50% of GPT-4o's total errors occurred on borderline cases where human annotators themselves disagreed.
  • Attack Detection: Proprietary models maintained a distinct advantage over open-weight alternatives. GPT-4o and GPT-5.4-mini dominated the task, performing best under the minimal prompt configuration. Unlike the praise task, errors were evenly distributed across clear and borderline cases, meaning model limitations weren't strictly driven by text ambiguity.

Limitations and Future Work

  • Dataset Scale: Both evaluation datasets are relatively small and restricted to Reddit and Wikipedia. Testing on larger, more diverse corpora is necessary to ensure these interaction styles generalize across other social platforms.
  • Prompting Strategies: The study focused strictly on predefined zero-shot prompts. The authors suggest that exploring few-shot prompting, chain-of-thought (CoT) reasoning, or lightweight fine-tuning could unlock higher accuracy, particularly for the nuanced praise detection task.
  • Downstream Integration: The ultimate goal is integrating this interaction-level relational annotation into a broader sign prediction pipeline. Assessing how these LLM-generated labels impact graph-level tasks (like community detection or signed link prediction) remains an open and crucial next step.

Summary updated

Abstract

Inferring the sign of social relationships from online interactions is a fundamental challenge in social network analysis. Existing approaches typically rely on sentiment analysis to label individual interactions as positive or negative, then aggregate these labels to assign a sign to the relationship. However, sentiment analysis captures the valence of the content being discussed rather than the nature of the relational exchange itself, a conflation that can lead to systematic misclassification. In this paper, we propose a methodology that addresses this limitation by leveraging Large Language Models (LLMs) in a zero-shot setting to identify interaction-level relational signals (specifically, personal praise and personal attacks directed at the interlocutor) as more direct indicators of positive and negative social ties. We evaluate four models spanning open-weight and proprietary architectures (Qwen2.5:7b, Gemma2:9b, GPT-4o, GPT-5.4-mini) across three prompt designs of increasing complexity, on two human-annotated datasets of approximately 298 and 340 texts respectively. Results show that zero-shot LLMs achieve good classification performance on both tasks without any task-specific training data, establishing a practical baseline for relational annotation. Performance differs across tasks: attack detection is robust to prompt design and model choice, while praise detection is more sensitive to both, reflecting the greater subjectivity of positive relational gestures. These findings lay the groundwork for integrating LLM-based relational annotation into sign prediction pipelines.
BibTeX
@misc{bombino2026annotation,
  author = {Biancamaria Bombino and Chiara Boldrini and Andrea Passarella and Marco Conti},
  title = {{Annotation of Positive vs Negative User Interactions for Social Sign Prediction}},
  year = {2026},
  eprint = {2606.06425},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.06425},
}

Loading related content...

Figure