TL;DR
Researchers have significantly improved question classification for Bengali—a language with complex syntax and limited digital resources—by moving beyond single machine learning models. By combining multiple classifiers through techniques like stacking and voting, and introducing a new two-layer taxonomy of 69 fine-grained categories, they achieved up to 91.65% accuracy, outperforming individual models by a notable margin.
What Problem Does This Paper Address?
Question Answering (QA) systems rely heavily on Question Classification (QC) to understand what kind of answer a user is looking for (e.g., a person, a location, or a definition). While QC is well-studied for English, it remains a challenge for Indic languages like Bengali.
Bengali presents unique hurdles: it has relatively free word order, complex morphology, and a lack of large-scale labeled datasets. Previous attempts at Bengali QC were mostly limited to simple rule-based systems or single classifiers using a small set of broad categories. Furthermore, deep learning—the current "gold standard" in NLP—often fails when training data is scarce. This paper tackles these issues by developing a robust framework for Bengali QC that works effectively with small datasets.
Methods
The authors employed a "strength in numbers" strategy, using three primary classifier combination techniques:
- Ensemble (Bagging and Boosting): Using multiple versions of the same model (like Decision Trees) to reduce variance and bias.
- Stacking: Training a "meta-learner" to combine the predictions of several different base models.
- Voting: Taking a majority vote from a pool of different classifiers.
They used four base learners: Naïve Bayes, kernel Naïve Bayes, Rule Induction, and Decision Trees. To feed these models, they engineered a rich set of features:
- Lexical: Interrogative words (wh-words), their positions, question length, and word shapes.
- Syntactic: Part-of-Speech (POS) tags and "head words" (the most informative word in the question).
- Semantic: Related words from a Bengali synonym dictionary and Named Entities (NER).
Crucially, they expanded the existing Bengali question taxonomy from 8 coarse-grained classes to a hierarchical two-layer system featuring 69 fine-grained classes.
Datasets
The study utilized a dataset of 1,100 Bengali questions spanning domains like history, geography, and science.
- Training set: 769 questions (70%)
- Testing set: 331 questions (30%)
- Annotation: Questions were manually labeled by native Bengali specialists, achieving a high inter-annotator agreement (Kappa = 0.85).
Key Insights and Ideas
- Ensembles Beat Deep Learning on Small Data: With only 1,100 samples, traditional machine learning combinations proved more effective than data-hungry deep learning frameworks.
- Feature Synergy: Accuracy consistently improved as the models moved from using only lexical features to incorporating syntactic and semantic information.
- The Power of the Head Word: Identifying the "head word" (e.g., "city" in "Which is the largest city?") is vital but difficult in Bengali due to free word order. The authors used specific heuristics based on the position of the interrogative word to solve this.
- Stability in Numbers: For ensemble methods like Bagging, the authors found that accuracy stabilizes after a certain number of iterations (typically between 7 and 20), preventing overfitting.
Key Results
The combination approaches consistently outperformed individual classifiers across the board:
- Coarse-Grained Classification: The Voting approach achieved the highest accuracy at 91.65%, a 4.02% improvement over the best single classifier.
- Fine-Grained Classification: The Stacking approach (using a Decision Tree as the meta-learner) proved superior, reaching 87.79% accuracy.
- Base Learner Performance: Among individual models, Decision Trees (DT) were the strongest performers, while Naïve Bayes served as a reliable baseline.
Limitations and Future Work
The primary limitation is the dataset size. While 1,100 questions provided a solid foundation, a larger corpus would allow for even more granular classification and the potential use of neural networks.
The authors suggest several future directions:
- Expanding the dataset to include more diverse question types.
- Incorporating state-of-the-art neural network techniques as more data becomes available.
- Applying this classifier combination methodology to other Indo-Aryan or Indic languages that face similar resource constraints.
Summary updated