Using Hierarchical Class Structure to Improve Fine-Grained Claim Classification

Erenay Dayanik, Andre Blessing, Nico Blokker, Sebastian Haunss, Jonas Kuhn, Gabriella Lapesa, Sebastian PadóACL 2021

TL;DR

This paper tackles the challenge of fine-grained claim classification in political debates, where many specific categories are rare. They propose and evaluate methods (Hierarchical Label Encoding and Integer Linear Programming) to leverage the inherent hierarchy of claim ontologies, showing significant improvements in classification accuracy, especially for infrequent categories.

What Problem Does This Paper Address?

Analyzing public debates often involves identifying specific "claims" within text and assigning them to a detailed ontology of categories. This is crucial for understanding policy discussions, but a major hurdle is the sheer number of fine-grained categories, many of which appear very infrequently in real-world data. This data sparsity makes it difficult for standard classification models to learn effectively, leading to poor performance, especially for the less common but potentially important claims. The paper aims to improve the classification of these fine-grained categories by explicitly using the hierarchical structure of the ontologies.

Methods

The authors propose two main methods to incorporate the hierarchical structure of claim ontologies into classification models:

  • Hierarchical Label Encoding (HLE): This method injects the relationship between super- and subcategories directly into the model's representation learning. It achieves this by creating a matrix that encodes subclass relationships and then multiplying it with the weight matrix of the final fully connected layer. This effectively introduces parameter sharing between related classes, encouraging consistency.
  • Integer Linear Programming (ILP): This approach treats classification as a binary optimization problem. It uses the predicted probabilities from a standard claim classifier as coefficients and then applies "hard" constraints to ensure that predictions are logically consistent with the ontology. Specifically, it enforces that a subcategory can only be predicted if its parent supercategory is also predicted, and that a supercategory must have at least one subcategory predicted.

These methods can be applied to various neural network architectures, including LSTMs, BiLSTMs, and BERT.

Datasets

The experiments were conducted on an extended version of the DebateNet-mig15 corpus. This corpus contains 1361 articles from the German quality newspaper "taz" published in 2015, annotated with a two-level claim ontology. The ontology consists of 8 supercategories and 100 subcategories, with the dataset containing 3827 annotated textual spans. For their experiments, they filtered out very infrequent subcategories (below 20 instances) and merged smaller categories into a catch-all category, resulting in 8 supercategories and 72 subcategories.

Key Insights and Ideas

  • Hierarchy is Key: Explicitly leveraging the hierarchical structure of claim ontologies significantly boosts performance, especially for rare categories. This is a crucial insight for domains with sparse, fine-grained labels.
  • Complementarity of HLE and ILP: HLE and ILP offer complementary benefits. HLE improves representation learning by encouraging consistency during training, while ILP enforces logical consistency as hard constraints post-prediction. Combining them yields the best results.
  • Low-Frequency Classes Benefit Most: The most substantial improvements were observed for low-frequency categories, which are typically the hardest to classify. This suggests the proposed methods are particularly effective in addressing data sparsity.
  • Concreteness Matters: The effectiveness of the hierarchical approach seems to be linked to the "concreteness" of the policy measures. More concrete subcategories showed larger gains, while less concrete ones (e.g., related to "Society") did not benefit as much.

Key Results

  • Overall Improvement: Both HLE and ILP individually improved classification performance across all tested architectures. The combination (HLE+ILP) consistently outperformed either method alone.
  • BERT's Strong Performance: The BERT model, when enhanced with HLE+ILP, achieved the highest overall Recall (0.59) and F-Score (0.60), representing a 13-point F1 improvement over the plain BERT model.
  • Frequency Band Analysis: For the BERT model, the plain classifier performed poorly on low-frequency classes (F1=0.1). HLE and ILP, especially HLE, significantly boosted performance in this band (+27 F1 points for HLE).
  • Recall Focus: The F1 score improvements were largely driven by gains in Recall, which is particularly valuable for semi-automatic annotation tasks.

Limitations and Future Work

  • Dataset Specificity: The study was conducted on a specific German immigration debate corpus. While the methods are general, their performance might vary on different domains or languages.
  • Concreteness Effect: The observation that more concrete subcategories benefit more than less concrete ones suggests that future work could explore how to better model abstract or nuanced policy claims.
  • Further Exploration: The authors suggest investigating these methods on a wider range of datasets and evaluating their utility for constructing semi-automatic discourse networks. They also hint at potential for further improvements by exploring better combinations of HLE and ILP.

Summary updated

BibTeX
@inproceedings{dayanik2021using,
  author = {Erenay Dayanik and Andre Blessing and Nico Blokker and Sebastian Haunss and Jonas Kuhn and Gabriella Lapesa and Sebastian Padó},
  title = {{Using Hierarchical Class Structure to Improve Fine-Grained Claim Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure