The “r” in “woman” stands for rights. Auditing LLMs in Uncovering Social Dynamics in Implicit Misogyny

Arianna Muti, Chris Emmery, Debora Nozza, Alberto Barrón‐Cedeño, Tommaso Caselli2025 · DOI

TL;DR

This paper investigates how well Large Language Models (LLMs) can detect subtle, implicit misogyny in text, moving beyond overt hate speech. The authors developed a new taxonomy of "social dynamics" to categorize these implicit biases and found that current LLMs struggle to accurately identify them, often relying on superficial word associations rather than deep understanding.

What Problem Does This Paper Address?

Misogyny, a pervasive societal bias, often manifests subtly through implicit language rather than explicit slurs. Traditional approaches to detecting misogyny in text have focused on explicit forms, overlooking these more insidious expressions. This leaves online spaces vulnerable to the perpetuation of harmful stereotypes and social injustices. The authors aim to bridge this gap by auditing LLMs' ability to identify these nuanced, implicit misogynistic social dynamics, which are deeply embedded in cultural norms and power structures.

Methods

The core of this research lies in a novel, literature-grounded taxonomy of "social dynamics" that capture the underlying communicative intent behind implicit misogynistic statements. This taxonomy was developed through a mixed-method approach, combining inductive observation of social media data with deductive insights from feminist and gender studies.

The researchers then evaluated nine LLMs (including models like Llama2, Llama3, Mistral, Qwen, and GPT-40-mini) on two tasks:

  1. Multi-label classification: Identifying relevant social dynamics from a predefined list.
  2. Text span selection: Pinpointing the exact text segments that triggered the identified social dynamics.

These evaluations were conducted across three learning settings: zero-shot, few-shot, and prescriptive prompting.

Datasets

The study utilized two existing corpora, enriched with the newly developed social dynamics annotations:

  • ImplicIT-Mis: A corpus of 1,120 Italian Facebook comments specifically targeting implicit misogyny.
  • SBIC+: A subset of the Social Bias Inference Corpus (SBIC) containing 2.4k English social media posts filtered for implicit misogyny.

Key Insights and Ideas

  • Implicit Misogyny is Tricky: Detecting subtle misogyny is challenging not only for humans but also for LLMs, as it relies on understanding underlying assumptions and social context, not just surface-level language.
  • LLMs Rely on Semantics, Not Understanding: The study suggests that LLMs often fail to grasp the true meaning of implicit misogyny, instead relying on semantic associations between words. This means they might correctly identify a category because certain words are present, but without understanding why those words contribute to misogyny.
  • Instruction Following is a Hurdle: LLMs demonstrated a struggle to consistently follow instructions, particularly in accurately identifying text spans and adhering to the specified output format.
  • Language Differences Matter: The research observed distinct patterns in how implicit misogyny is expressed in English and Italian, with English leaning towards more direct and confrontational forms, while Italian is more subtly embedded in cultural norms.

Key Results

  • Overall Low Performance: LLMs generally performed poorly across all settings, indicating a significant gap in their ability to understand and identify implicit misogyny.
  • Italian Data Yields Better Results (for some models): Models like GPT-40-mini and Qwen showed higher performance on Italian data compared to English, suggesting some models might be better attuned to certain linguistic nuances or cultural expressions.
  • Explicit Cases are Easier: Models were more successful at identifying overtly hostile or objectifying language that might have slipped through the implicit-only filtering, highlighting the difficulty of truly implicit content.
  • Free-text vs. Categorical Labels: While free-text explanations from LLMs often captured underlying reasoning better, they lacked the precision of structured categories. The authors suggest a combination of both approaches is ideal for comprehensive evaluation.

Limitations and Future Work

  • Limited Model Scope: The study primarily used smaller, more accessible LLMs (7B parameter range) due to computational constraints. This limits the generalizability of findings to larger, more capable models.
  • Need for Fine-tuning: The results suggest that current LLMs lack the necessary social knowledge for this task and would likely benefit from additional training, fine-tuning, or knowledge-augmentation approaches.
  • External Knowledge: The authors propose exploring the use of external knowledge sources as a future direction to enhance LLMs' understanding of implied societal assumptions.

Summary updated

Abstract

Persistent societal biases like misogyny express themselves more often implicitly than through openly hostile language.However, previous misogyny studies have focused primarily on explicit language, overlooking these more subtle forms. We bridge this gap by examining implicit misogynistic expressions in English and Italian. First, we develop a taxonomy of social dynamics, i.e., the underlying communicative intent behind misogynistic statements in social media data. Then, we test the ability of nine LLMs to identify the social dynamics as a multi-label classification and text span selection: first LLMs must choose social dynamics given a prefixed list, then they have to explicitly identify the text spans that triggered their decisions. We also investigate the extent of using different learning settings: zero and few-shot, and prescriptive. Our analysis suggests that LLMs struggle to follow instructions and reason in all settings, mostly relying on semantic associations, recasting claims of emergent abilities.
BibTeX
@misc{muti2025r,
  author = {Arianna Muti and Chris Emmery and Debora Nozza and Alberto Barrón‐Cedeño and Tommaso Caselli},
  title = {{The “r” in “woman” stands for rights. Auditing LLMs in Uncovering Social Dynamics in Implicit Misogyny}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.292},
}
Figure