Improving Neural Political Statement Classification with Class Hierarchical Information

Erenay Dayanik, André Blessing, Nico Blokker, Sebastian Haunss, Jonas Kuhn, Gabriella Lapesa, Sebastian PadóACL 2022 · DOI

Introduction

In Computational Social Science (CSS), researchers frequently classify political statements into highly specific, fine-grained categories. While these labels are essential for nuanced analysis, they present a significant "long-tail" problem for machine learning: a few categories appear frequently, while the vast majority are extremely rare. Standard neural models, such as BERT, often struggle with these infrequent classes because they treat every label as an independent entity, failing to leverage the inherent relationships between them.

However, human-designed political codebooks are almost always hierarchical. For instance, "Free Market" and "Market Regulation" both fall under the broader supercategory of "Economy." This research explores how to integrate this hierarchical "prior knowledge" into neural networks to improve representations for rare classes by "borrowing" strength from their more frequent siblings in the hierarchy.

Methodology

The authors propose several lightweight methods to integrate hierarchical information into standard transformer-based (BERT) fine-tuning. These methods are categorized into end-to-end training techniques and post-processing steps.

Taxonomy of methods for encoding hierarchical structure.

Key methods include:

  • Hierarchical Label Encoding (HLE): Decomposes the representation of a subcategory into a sum of vectors representing itself and its parent supercategories.
  • Class Representation Regularization (CRR): Adds a penalty to the loss function to encourage weight vectors of subcategories within the same supercategory to be more similar.
  • Instance Representation Regularization (IRR): Penalizes the model if it produces significantly different embeddings for two statements belonging to the same supercategory.
  • Integer Linear Programming (ILP): A post-processing step that enforces hard constraints, ensuring that if a subcategory is predicted, the corresponding supercategory is also accounted for.

These methods add virtually no new parameters, making them highly suitable for the relatively small datasets typical in social science research.

Experimental Setup

The models were evaluated on two major benchmarks:

  1. DebateNet-mig15: A German corpus regarding the 2015 refugee crisis (8 supercategories, 72 subcategories), treated as a multi-label task.
  2. Comparative Manifesto Project (CMP): A massive dataset of political party manifestos across five languages (Finnish, German, Hungarian, Turkish, and English), treated as a single-label task.

Key Results and Analysis

Performance on Rare Classes

The most significant finding is the massive gain in performance for low-frequency categories. On the DebateNet dataset, the base model performed poorly on rare classes with an F1 score of only 9.6. By applying the combined hierarchical approach (HLE+CRR+ILP), this score jumped to 38.3—a nearly fourfold improvement.

Performance comparison across frequency bands on the DebateNet dataset.

Overall Gains and Data Efficiency

Beyond rare classes, the hierarchical models improved overall performance. On DebateNet, the best combined model raised the overall F1 score from 47.0 to 60.8. Furthermore, the benefit of hierarchical information is most pronounced when training data is scarce. Experiments on the Manifesto corpus showed that hierarchical models (specifically CRR+IRR) consistently outperformed the base model across all five languages, with the largest performance gap occurring when only 25% of the training data was used.

Conclusion

This research demonstrates that incorporating hierarchical priors through simple regularization and encoding techniques is more effective than building complex, dedicated architectures for small-to-medium CSS datasets. By improving performance on low-frequency categories, these methods not only enhance accuracy but also promote algorithmic fairness by ensuring that niche political viewpoints and minority voices are accurately captured in automated analyses.

Summary updated

BibTeX
@inproceedings{dayanik2022improving,
  author = {Erenay Dayanik and André Blessing and Nico Blokker and Sebastian Haunss and Jonas Kuhn and Gabriella Lapesa and Sebastian Padó},
  title = {{Improving Neural Political Statement Classification with Class Hierarchical Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.186},
}
Figure