A Dataset for Explainable Sentiment Analysis in the German Automotive Industry

Andrea Zielinski, Calvin Spolwind, Henning Kroll, Anna GrimmACL 2023

TL;DR

This paper introduces "InfoBarometer," the first German dataset for explainable sentiment analysis in the automotive industry, complete with sentiment labels, topic categories, and human-annotated rationales. They found that while BERT excels at sentiment classification, current explainability methods struggle to align with human interpretations, highlighting a critical gap in bridging model predictions with human understanding.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, is crucial for understanding public opinion and market trends. While deep learning models have achieved impressive accuracy in this area, they often operate as "black boxes," making it difficult to understand why they make certain predictions. This lack of transparency is a significant hurdle, especially in high-stakes domains like financial analysis, where decisions need to be justifiable. Furthermore, existing datasets for explainable sentiment analysis are often limited in scope or language. This paper addresses these issues by creating a specialized dataset for German automotive news and evaluating how well current explainability techniques can shed light on sentiment analysis models trained on this domain.

Methods

The researchers employed a multi-faceted approach:

  • Dataset Creation: They scraped German news articles related to automotive innovations, meticulously annotating them with sentiment polarity (positive, negative, neutral), topic categories (e.g., e-mobility, autonomous driving), and crucially, human-provided "rationales" – textual spans explaining the sentiment.
  • Sentiment Classification Models: They evaluated several state-of-the-art models, including SVM, CNN, LSTM, and a fine-tuned BERT model, to determine the best performer for sentiment classification on their dataset.
  • Explainability Techniques: They applied various post-hoc explainability methods, such as LIME, SHAP, Integrated Gradients, and Saliency maps, to understand the internal workings of the trained models. These methods aim to highlight which parts of the input text are most influential for a given prediction.
  • Evaluation Metrics: They used metrics like plausibility (agreement with human rationales) and faithfulness (how well explanations reflect the model's internal logic) to assess the quality of the explanations.

Datasets

The primary contribution is the InfoBarometer dataset, a newly introduced benchmark comprising 1,039 German news articles from the automotive sector. Each article is annotated with:

  • Sentiment Polarity: Positive, Negative, Neutral.
  • Topic Category: e.g., autonomous driving, electromobility, hydrogen, Diesel, Synfuel, misc.
  • Human Rationales: Text spans identified by human annotators as justification for the sentiment label.

This dataset is split into training, validation, and testing sets. The authors note that it differs from existing datasets by including a neutral class and focusing on the specific domain of German automotive innovation news.

Key Insights and Ideas

  • BERT Dominates Classification, But Explanations Lag: While BERT achieved the highest accuracy in sentiment classification, the explainability methods applied to it did not consistently align with human-annotated rationales. This suggests a disconnect between high predictive performance and true interpretability.
  • Human Rationales are Key, But Hard to Replicate: The inclusion of human rationales is vital for evaluating explainability. However, the study found that XAI methods often failed to capture the same textual evidence that humans used, indicating that current XAI techniques may not fully grasp the nuances of human reasoning in sentiment analysis.
  • Domain-Specific Language is a Challenge: The automotive industry's specialized vocabulary and the potential for mixed sentiments within a single article (e.g., positive about products, negative about investments) make sentiment analysis inherently complex.
  • Global Explanations Offer Some Value: While local explanations (explaining individual predictions) struggled, global explanations (like SHAP feature importance) proved useful for identifying potential biases or spurious features in the dataset and model.

Key Results

  • Classification Performance: The fine-tuned BERT model achieved the highest macro F1-score of 73.8% for sentiment classification.
  • Inter-Annotator Agreement (IAA): High agreement was observed among human annotators, with an average F1 score of 0.829 for topic classification and 0.774 for sentiment annotation, indicating reliable gold-standard data.
  • Plausibility: There was only modest agreement between XAI-generated explanations and human rationales. Simpler models like CNN showed slightly better agreement, suggesting a potential trade-off between model complexity and explainability alignment with human judgment.
  • Faithfulness: Simpler models (CNN) generally exhibited higher faithfulness scores, meaning their predictions were more strongly influenced by the selected salient tokens.
  • Computational Efficiency: LIME, IG, and Saliency methods were found to be computationally efficient enough for real-time applications, whereas SHAP suffered from longer computation times.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that their dataset, while valuable, is relatively small, which can limit the robustness of deep learning models and the effectiveness of some explainability techniques.
  • Explanation Alignment: A significant limitation is the observed discrepancy between XAI explanations and human rationales, suggesting that current XAI methods are not yet fully capable of replicating human reasoning in this domain.
  • Future Work: The researchers plan to investigate influence functions to identify specific training data points responsible for model misclassifications and biases. They also aim to explore the impact of pre-trained embeddings and model architectures on explainability.

Summary updated

BibTeX
@inproceedings{zielinski2023dataset,
  author = {Andrea Zielinski and Calvin Spolwind and Henning Kroll and Anna Grimm},
  title = {{A Dataset for Explainable Sentiment Analysis in the German Automotive Industry}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure