HebID: Detecting Social Identities in Hebrew-language Political Text

Guy Mor-Lan, Naama Rivlin-Angert, Yael R. Kaplan, Tamir Sheafer, Shaul R. ShenhavEMNLP 2025

TL;DR

This paper introduces HEBID, the first comprehensive Hebrew-language dataset for detecting nuanced social identities in political text. They demonstrate that Hebrew-tuned large language models (LLMs) significantly outperform existing methods, enabling deeper analysis of how political actors express identities across different platforms and how these dynamics relate to public opinion.

What Problem Does This Paper Address?

Understanding how individuals and groups express their social identities is crucial for analyzing political discourse. However, existing computational tools for this task are largely English-centric, focus on coarse-grained categories, and are often single-label. This leaves a significant gap in our ability to analyze identity expression in non-English political contexts, particularly for nuanced, multi-faceted identities. The authors aim to bridge this gap by creating a new dataset and benchmark for Hebrew, a language with unique socio-political characteristics.

Methods

The core of this work is the creation of the HEBID dataset, which involves a novel approach to annotation. They first identified salient social identities in Israel through a large-scale national survey. These survey-derived identities were then used to guide the manual annotation of political texts. For the computational analysis, they benchmarked various models, including multilabel and single-label encoder models, as well as decoder-based Large Language Models (LLMs). They specifically highlight the effectiveness of Hebrew-tuned LLMs, particularly DICTALM2.0, for this task.

Datasets

  • HEBID Dataset: A newly introduced, multilabel Hebrew corpus of 5,536 sentences from Israeli politicians' Facebook posts (Dec 2018–Apr 2021), annotated for twelve nuanced social identities.
  • Israeli Panel Survey Data: Used to empirically ground the selection of social identities for annotation.
  • IsraParlTweet Dataset: Used for cross-genre evaluation of the model's generalization to parliamentary speeches.
  • CHES-Israel Expert Survey Data: Used for external validation by correlating predicted identity discourse with party policy rankings.

Key Insights and Ideas

  • Survey-grounded annotation is key: Using survey data to identify salient identities significantly improves the relevance and comprehensiveness of the annotated dataset.
  • Hebrew-tuned LLMs excel: Large Language Models specifically fine-tuned on Hebrew data (like DICTALM2.0) significantly outperform general multilingual models and encoder-only architectures for social identity detection in this language.
  • Identity discourse is platform-dependent: The popularity and expression of certain identities differ notably between politicians' Facebook posts, parliamentary speeches, and public survey responses.
  • Electoral cycles amplify identity discourse: Identity-related language in political discourse, particularly on social media, tends to surge around election periods.
  • Identities form "bundles": Certain identities tend to co-occur, forming distinct ideological groups (e.g., a left-wing cluster and a right-wing cluster), which can be observed across different data sources.

Key Results

  • The best-performing model, DICTALM2.0, achieved a macro-F1 score of 0.743 on the HEBID test set.
  • The model demonstrated strong generalization capabilities, achieving a comparable macro-F1 score of 0.72 on parliamentary speeches.
  • External validation showed a strong alignment between predicted identity discourse and expert-coded party policy stances, with 16 out of 21 correlations being statistically significant.
  • Analysis revealed that "Socially-oriented," "Rightist," and "Democrat" identities are generally popular across platforms, while "Leftist" is less so.
  • Identities like "Honest" and "Zionist" are significantly more popular among the public than in elite discourse, while "Socially-oriented" is more prominent in parliament.
  • Women tend to express more identities than men, and specific identities show gendered leanings (e.g., "Socially-oriented" leans towards women, while "Rightist" and "Security-oriented" lean towards men).

Limitations and Future Work

  • Temporal and Platform Scope: The study is limited to a specific time period and platform (Facebook), potentially missing evolving identity discourse on other platforms (e.g., Twitter, news media).
  • Survey Population: The panel survey only included Jewish citizens of Israel, excluding other significant populations like Palestinians, whose identity salience might differ.
  • Annotation Granularity: While multilabel, the scheme uses twelve categories, potentially excluding less frequent but important identities. Negative or critical references to identities were also excluded.
  • Model Biases: Classifiers may inherit biases from training data and pre-trained models, potentially affecting performance on dialectal or informal text.
  • Cross-Genre Validation: The parliamentary speech evaluation used a limited sample, suggesting a need for broader validation.
  • Methodological Considerations: Comparing survey self-reports with text analysis requires caution due to inherent differences in measurement approaches.

Future work should extend HEBID to more platforms and populations, refine the annotation taxonomy to include emerging identities, and explore methods to mitigate model bias in identity detection.

Summary updated

BibTeX
@inproceedings{morlan2025hebid,
  author = {Guy Mor-Lan and Naama Rivlin-Angert and Yael R. Kaplan and Tamir Sheafer and Shaul R. Shenhav},
  title = {{HebID: Detecting Social Identities in Hebrew-language Political Text}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure