TL;DR
Researchers have introduced FoodReact-AR, a new dataset and evaluation protocol that highlights a significant blind spot in Large Language Models (LLMs): the ability to detect derisive audience reactions. While zero-shot Arabic LLMs are excellent at identifying "Grief" in food-insecurity discourse, they almost entirely fail to recognize "Dismissal" (sarcasm or mockery), a signal that simple supervised models can easily learn from text and metadata.
What Problem Does This Paper Address?
In social media analysis, we often assume that the sentiment of a post tells us how people feel. However, "what is said" is not always "how it is received." A government ministry might post a neutral, factual update about a food budget, but the audience might respond with a flood of "Haha" (laughing) reactions—a clear sign of derision or institutional distrust.
Current NLP pipelines, especially those relying on zero-shot LLMs for annotation, tend to read text at face value. This paper investigates whether these models miss the "hidden" signal of audience reception in Arabic food-insecurity discourse, specifically across the Middle East and North Africa (MENA) region during recent global shocks like the COVID-19 pandemic, the 2022 wheat price spike, and the war in Gaza.
Methods
The authors treat Facebook reactions (Love, Care, Wow, Haha, Sad, Angry) as compositional data. Because these reactions are proportions that sum to 100%, an increase in one mechanically decreases the others. To handle this, they used a Centered Log-Ratio (CLR) transformation, mapping the proportions into Euclidean space where standard statistical methods apply.
Using k-means clustering on these transformed reactions, they identified four distinct "audience profiles":
- Solidarity: Dominated by "Love" reactions.
- Dismissal: Dominated by "Haha" reactions (often indicating sarcasm).
- Grief: Dominated by "Sad" reactions.
- Ambivalent: A flat profile with no dominant reaction.
They then tested whether a state-of-the-art Arabic-centric LLM (Fanar) and simple supervised classifiers could predict these profiles using only the post's text and metadata (like the country of the page and category).
Datasets
The study introduces the FoodReact-AR resource, which includes:
- 74,241 Arabic Facebook posts (2020–2024) related to food security, clustered into the four profiles.
- 6,225 reaction-blind LLM labels for zero-shot evaluation.
- 800 posts with both reaction-informed LLM labels and independent native-speaker human annotations.
- A 294-post matched ablation set to test if showing the LLM the reaction percentages actually helps its prediction.
Key Insights and Ideas
- The "Haha" Signal: The "Dismissal" profile is the most elusive. It peaks during periods of economic stress (like the 2022 wheat shock) and is most common on posts from government organizations and TV channels. It represents a "derisive laughter" that text-only sentiment analysis usually misses.
- LLMs are Too Literal: Zero-shot LLMs are surprisingly bad at "reading the room." While the LLM recovered "Grief" for 83.8% of posts, it caught "Dismissal" only 3.4% of the time. Even when the researchers showed the LLM that 60% of the audience reacted with "Haha," the model still refused to label the post as "Dismissal."
- Derision is Predictable, but not from Tone: Simple supervised models (Logistic Regression) trained on reaction labels achieved 60% recall on Dismissal. These models didn't look for "sarcastic tone" but rather learned associations between specific actors (ministers, institutions) and the likely derisive response they provoke.
Key Results
- Temporal Shifts: The study quantitatively confirmed that audience moods shifted with global events. Dismissal reactions doubled in 2022 during the wheat price shock, while Grief reactions peaked in 2024 during the Gaza food emergency.
- Model Performance: The supervised baseline (text + metadata) significantly outperformed the zero-shot LLM, improving the macro-F1 score from 0.43 to 0.57.
- Engagement: "Solidarity" posts are shared the most, while "Dismissal" posts generate the most comments—suggesting that derision drives discussion but not necessarily "travel" or virality in the same way as positive affiliation.
Limitations and Future Work
The authors acknowledge that their supervised baselines are lexical models that might be picking up on "outlet signatures" (e.g., knowing a certain page always gets mocked) rather than deep sarcasm. Additionally, the study used a single human annotator for the gold-standard set, which limits inter-annotator reliability checks.
Future work suggests that fine-tuning Arabic encoders (like MARBERT) or using few-shot prompting with reaction-derived examples could bridge the gap for LLMs. The authors have released their evaluation protocol to encourage the community to treat audience response as a distinct prediction target, separate from text sentiment.
Summary updated