Classifier Combination Approach for Question Classification for Bengali Question Answering System

Somnath Banerjee, Sudip Kumar Naskar, Paolo Rosso, Sivaji BandyopadhyayarXiv 2020 · DOI · arXiv

TL;DR

Researchers have significantly improved question classification for Bengali—a language with complex syntax and limited digital resources—by moving beyond single machine learning models. By combining multiple classifiers through techniques like stacking and voting, and introducing a new two-layer taxonomy of 69 fine-grained categories, they achieved up to 91.65% accuracy, outperforming individual models by a notable margin.

What Problem Does This Paper Address?

Question Answering (QA) systems rely heavily on Question Classification (QC) to understand what kind of answer a user is looking for (e.g., a person, a location, or a definition). While QC is well-studied for English, it remains a challenge for Indic languages like Bengali.

Bengali presents unique hurdles: it has relatively free word order, complex morphology, and a lack of large-scale labeled datasets. Previous attempts at Bengali QC were mostly limited to simple rule-based systems or single classifiers using a small set of broad categories. Furthermore, deep learning—the current "gold standard" in NLP—often fails when training data is scarce. This paper tackles these issues by developing a robust framework for Bengali QC that works effectively with small datasets.

Methods

The authors employed a "strength in numbers" strategy, using three primary classifier combination techniques:

  1. Ensemble (Bagging and Boosting): Using multiple versions of the same model (like Decision Trees) to reduce variance and bias.
  2. Stacking: Training a "meta-learner" to combine the predictions of several different base models.
  3. Voting: Taking a majority vote from a pool of different classifiers.

They used four base learners: Naïve Bayes, kernel Naïve Bayes, Rule Induction, and Decision Trees. To feed these models, they engineered a rich set of features:

  • Lexical: Interrogative words (wh-words), their positions, question length, and word shapes.
  • Syntactic: Part-of-Speech (POS) tags and "head words" (the most informative word in the question).
  • Semantic: Related words from a Bengali synonym dictionary and Named Entities (NER).

Crucially, they expanded the existing Bengali question taxonomy from 8 coarse-grained classes to a hierarchical two-layer system featuring 69 fine-grained classes.

Table 1: Two-layer Bengali Question Taxonomies

Datasets

The study utilized a dataset of 1,100 Bengali questions spanning domains like history, geography, and science.

  • Training set: 769 questions (70%)
  • Testing set: 331 questions (30%)
  • Annotation: Questions were manually labeled by native Bengali specialists, achieving a high inter-annotator agreement (Kappa = 0.85).

Key Insights and Ideas

  • Ensembles Beat Deep Learning on Small Data: With only 1,100 samples, traditional machine learning combinations proved more effective than data-hungry deep learning frameworks.
  • Feature Synergy: Accuracy consistently improved as the models moved from using only lexical features to incorporating syntactic and semantic information.
  • The Power of the Head Word: Identifying the "head word" (e.g., "city" in "Which is the largest city?") is vital but difficult in Bengali due to free word order. The authors used specific heuristics based on the position of the interrogative word to solve this.
  • Stability in Numbers: For ensemble methods like Bagging, the authors found that accuracy stabilizes after a certain number of iterations (typically between 7 and 20), preventing overfitting.
Figure 1: Size and Accuracy variation in Bagging with {fL, fS, fM}

Key Results

The combination approaches consistently outperformed individual classifiers across the board:

  • Coarse-Grained Classification: The Voting approach achieved the highest accuracy at 91.65%, a 4.02% improvement over the best single classifier.
  • Fine-Grained Classification: The Stacking approach (using a Decision Tree as the meta-learner) proved superior, reaching 87.79% accuracy.
  • Base Learner Performance: Among individual models, Decision Trees (DT) were the strongest performers, while Naïve Bayes served as a reliable baseline.
Detailed results show that combined approaches (bottom) consistently beat individual models (top).

Limitations and Future Work

The primary limitation is the dataset size. While 1,100 questions provided a solid foundation, a larger corpus would allow for even more granular classification and the potential use of neural networks.

The authors suggest several future directions:

  • Expanding the dataset to include more diverse question types.
  • Incorporating state-of-the-art neural network techniques as more data becomes available.
  • Applying this classifier combination methodology to other Indo-Aryan or Indic languages that face similar resource constraints.

Summary updated

Abstract

Question classification (QC) is a prime constituent of automated question answering system. The work presented here demonstrates that the combination of multiple models achieve better classification performance than those obtained with existing individual models for the question classification task in Bengali. We have exploited state-of-the-art multiple model combination techniques, i.e., ensemble, stacking and voting, to increase QC accuracy. Lexical, syntactic and semantic features of Bengali questions are used for four well-known classifiers, namely Na\"ıve Bayes, kernel Na\"ıve Bayes, Rule Induction, and Decision Tree, which serve as our base learners. Single-layer question-class taxonomy with 8 coarse-grained classes is extended to two-layer taxonomy by adding 69 fine-grained classes. We carried out the experiments both on single-layer and two-layer taxonomies. Experimental results confirmed that classifier combination approaches outperform single classifier classification approaches by 4.02% for coarse-grained question classes. Overall, the stacking approach produces the best results for fine-grained classification and achieves 87.79% of accuracy. The approach presented here could be used in other Indo-Aryan or Indic languages to develop a question answering system.
BibTeX
@misc{banerjee2020classifier,
  author = {Somnath Banerjee and Sudip Kumar Naskar and Paolo Rosso and Sivaji Bandyopadhyay},
  title = {{Classifier Combination Approach for Question Classification for Bengali Question Answering System}},
  year = {2020},
  eprint = {2008.13597},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2008.13597},
}
Figure