Enhancing Aspect Extraction for Hindi

Arghya Bhattacharya, Alok Debnath, Manish ShrivastavaOpenReview 2021 · OpenReview

Abstract: Aspect extraction is not a well-explored topic in Hindi, with only one corpus having been developed for the task. In this paper, we discuss the merits of the existing corpus in terms of quality, size, sparsity, and performance in aspect extraction tasks using established models. To provide a better baseline corpus for aspect extraction, we translate the SemEval 2014 aspect-based sentiment analysis dataset and annotate the aspects in that data. We provide rigorous guidelines and a replicable methodology for this task. We quantitatively evaluate the translations and annotations using inter-annotator agreement scores. We also evaluate our dataset using state-of-the-art neural aspect extraction models in both monolingual and multilingual settings and show that the models perform far better on our corpus than on the existing Hindi dataset. With this, we establish our corpus as the gold-standard aspect extraction dataset in Hindi.

Introduction

Aspect Extraction (AE) is a critical component of sentiment analysis, involving the identification of specific attributes (e.g., "service" in a restaurant review) of an entity. While AE is well-established for English, Hindi has lacked high-quality, standardized benchmarks. Prior work, notably by Akhtar et al. (2016), suffered from significant sparsity and inconsistency issues. For instance, technical terms were inconsistently transliterated or kept in Roman script, and the distribution across 12 domains led to insufficient data per domain for effective model training. Consequently, state-of-the-art models that typically achieve ~80% F1 scores in English struggled to reach 40% on existing Hindi datasets.

Methodology

To address these limitations, we developed a new "gold-standard" parallel corpus by manually translating the SemEval 2014 English dataset (covering Laptops and Restaurants) into Hindi. Our methodology followed a rigorous two-phase process:

  1. Translation Strategy: We compared "aspect-aware" translations (preserving aspect terms) with "aspect-blind" translations (prioritizing natural flow), selecting the optimal version for each sentence.
  2. Standardization Guidelines: We established strict rules for handling "Hinglish" and technical terms. Proper nouns remained in Roman script, while technical terms were transliterated into Devanagari. Idiomatic expressions were simplified to their literal meanings to maintain clarity.
  3. Annotation: Aspects were annotated using the standard BIO (Begin, Inside, Outside) sequence labeling format.
  4. Evaluation: We benchmarked the dataset using traditional Conditional Random Fields (CRF) and advanced neural architectures, including DeCNN and Seq2Seq4ATE. We also explored cross-lingual transfer using multilingual BERT (mBERT).

Dataset Analysis

Our dataset comprises 5,989 sentences with 5,864 annotated aspects. By focusing on two domains rather than twelve, we significantly reduced the sparsity that plagued previous efforts.

Table 1: Comparison of dataset metrics between Akhtar et al. (2016), Pontiki et al. (2014), and our new Hindi dataset.

As shown in Table 1, while our total sentence count is comparable to Akhtar et al. (2016), the concentration of data into fewer domains provides a much denser signal for machine learning models. Furthermore, our inter-annotator agreement reached a Fleiss’ Kappa of 0.91, confirming high reliability.

Results and Discussion

The transition to a high-quality, standardized dataset yielded dramatic improvements in model performance across all tested architectures.

Table 2: Performance comparison (F1-score) of various models across different datasets.

The results in Table 2 highlight a massive performance jump. The Seq2Seq4ATE model, for example, improved from an F1 score of 35.04 on the Akhtar et al. dataset to 68.61 on our new corpus. Additionally, the parallel nature of the dataset allowed for effective fine-tuning of mBERT, which achieved an F1 of 66.28 by leveraging both English and Hindi data.

Conclusion

This work establishes a new gold standard for Hindi Aspect Extraction. By addressing the fundamental issues of sparsity and inconsistency in previous benchmarks, we have enabled a nearly 40% improvement in model performance. Future work will involve expanding this methodology to additional domains and integrating sentiment polarity annotation to facilitate full Aspect-Based Sentiment Analysis (ABSA) in Hindi.

Summary updated

Abstract

Aspect extraction is not a well-explored topic in Hindi, with only one corpus having been developed for the task. In this paper, we discuss the merits of the existing corpus in terms of quality, size, sparsity, and performance in aspect extraction tasks using established models. To provide a better baseline corpus for aspect extraction, we translate the SemEval 2014 aspect-based sentiment analysis dataset and annotate the aspects in that data. We provide rigorous guidelines and a replicable methodology for this task. We quantitatively evaluate the translations and annotations using inter-annotator agreement scores. We also evaluate our dataset using state-of-the-art neural aspect extraction models in both monolingual and multilingual settings and show that the models perform far better on our corpus than on the existing Hindi dataset. With this, we establish our corpus as the gold-standard aspect extraction dataset in Hindi.
BibTeX
@misc{bhattacharya2021enhancing,
  author = {Arghya Bhattacharya and Alok Debnath and Manish Shrivastava},
  title = {{Enhancing Aspect Extraction for Hindi}},
  year = {2021},
  url = {https://openreview.net/forum?id=7sNot5zF7oN},
  howpublished = {OpenReview},
}
Figure