“Feels Feminine to Me”: Understanding Perceived Gendered Style through Human Annotations

Hongyu Chen, Neele Falk, Michael Roth, Agnieszka FalenskaEMNLP 2025

Sure, here's a blog-style summary of the paper "“Feels Feminine to Me”: Understanding Perceived Gendered Style through Human Annotations":

TL;DR

This paper introduces a new dataset and methodology for understanding how humans perceive gendered language style, moving beyond author identity to focus on subjective interpretation. They found that perceived gendered style is influenced by specific linguistic features and annotator demographics, offering a more nuanced approach to modeling gender in NLP.

What Problem Does This Paper Address?

In Natural Language Processing (NLP), the association between language and gender has often been framed by inferring an author's gender identity from their writing. This approach risks reinforcing stereotypes and misgendering individuals, especially those who don't conform to traditional linguistic patterns. The authors argue that this identity-based framing conflates gender identity with gender expression. To address this, they propose shifting the focus to perceived gendered style – how language is externally interpreted by humans, independent of the author's actual gender identity. This allows for a more inclusive understanding of how gender is expressed and perceived in text.

Methods

The core of the study involves a human annotation task. They collected 5,100 annotations from 130 participants who rated texts on a 5-point scale from "very feminine" to "very masculine," including a "neutral" option. To understand what drives agreement and disagreement, they employed statistical analysis, including linear regression models, to identify textual features that correlate with perceived gendered style and annotator agreement. They also analyzed annotator demographics to see how they influence perceptions.

Datasets

The study utilized texts from three established datasets:

  • PAN13-EN: A large-scale dataset for authorship verification, containing conversational texts from social media.
  • BLOG: The Blog Authorship Corpus, comprising blog posts from blogger.com.
  • PASTEL: A parallel stylistic language dataset designed for research on persona-conditioned language variation.

The authors sampled texts from these datasets, ensuring a diverse range of content and style.

Key Insights and Ideas

  • Perception is Subjective: The study highlights that perceived gendered style is inherently subjective, with significant variation in annotator agreement. This challenges the idea of a single "correct" gendered style for a given text.
  • Annotator Demographics Matter: Annotators' own gender significantly influences their perceptions. Women were more likely to label texts as feminine, while men and non-binary annotators leaned towards masculine.
  • Linguistic Cues are Key: Specific textual features strongly influence perceived gendered style. Formal expressions and lower emotional intensity were associated with higher annotator agreement.
  • Feminine vs. Masculine vs. Neutral: Distinct linguistic patterns characterize each style. Feminine style is linked to positive emotions and pronouns, masculine style to directness and dominance, and neutral style to a balance of emotional intensity and complexity.
  • Beyond Identity: The findings underscore the importance of distinguishing between gender identity and gender expression, suggesting that NLP models should account for the social perception of gender in language.

Key Results

  • The study collected 5,100 human annotations of perceived gendered style.
  • Neutral style received the highest number of annotations, followed by "somewhat feminine" and "somewhat masculine."
  • Inter-annotator agreement (IAA) was moderate overall (Krippendorff's alpha = 0.22), but higher confidence levels among annotators correlated with higher agreement.
  • Textual features explained 11.5% of the variance in annotator agreement. Features like temporal entities and high-intensity trust were associated with lower agreement, while low arousal and dependency markers were linked to higher agreement.
  • In terms of perceived style, feminine style was associated with features like joy intensity and pronouns, masculine style with directness and adverbs, and neutral style with balanced emotional intensity and readability.

Limitations and Future Work

  • Dataset Size: The dataset of 5,100 annotations across 510 texts is considered preliminary. A larger, more diverse dataset is needed for better generalizability.
  • Disaggregation of Agreement: Future work could explore agreement levels within specific style categories (feminine, masculine, neutral).
  • Comparison with Models: The study could benefit from comparing human perceptions with automatic annotations from state-of-the-art language models.
  • Downstream Task Utility: The dataset's utility in downstream NLP tasks (e.g., bias detection, style transfer) needs to be evaluated.
  • Simplification of Gender: The paper acknowledges that gender is more complex than a binary spectrum and that the annotation task might have encouraged reliance on stereotypes. Future work should consider the intersectionality of gender with other social factors.
  • Author Identity Comparison: Direct comparison with author-identity-based patterns would provide deeper insights into how perceived style aligns with actual gender expression.

Summary updated

BibTeX
@inproceedings{chen2025feels,
  author = {Hongyu Chen and Neele Falk and Michael Roth and Agnieszka Falenska},
  title = {{“Feels Feminine to Me”: Understanding Perceived Gendered Style through Human Annotations}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure