TL;DR
This paper benchmarks various Arabic BERT and BART models for financial intent detection in the AraFinNLP 2024 shared task. They found that a two-step hierarchical classification approach, grouping intents into broader categories before fine-grained classification, generally improved performance over direct fine-tuning, with MARBERTv2 showing strong results.
What Problem Does This Paper Address?
The financial sector is experiencing rapid growth in Middle Eastern stock markets, creating a pressing need for specialized Arabic Natural Language Processing (NLP) tools. However, existing Arabic NLP models often struggle with the nuances of diverse Arabic dialects and the specific terminology of the financial domain. This paper addresses this gap by benchmarking several state-of-the-art transformer-based language models on the Multi-dialect Intent Detection task within the banking sector, aiming to identify the most effective approaches for this challenging area.
Methods
The researchers explored two primary approaches for intent detection:
- Direct Fine-tuning: Standard fine-tuning of pre-trained BERT and BART models directly on the 77 distinct intents of the ArBanking77 dataset.
- Hierarchical Classification: A novel two-step approach. First, intents were grouped into 11 broader "parent classes" (e.g., "Cards," "Charges," "Transactions"). A model was trained to predict these parent classes. Second, for each predicted parent class, a specialized sub-model was trained to classify the specific intent within that group. This approach was implemented by fine-tuning 12 models for each base model (11 parent class models + 1 overall model for the top-level classification).
Additionally, they investigated the impact of dialect translation by translating queries from various Arabic dialects to Modern Standard Arabic (MSA) before classification, using the AraT5-Arabic-Dialects-Translation model.
Datasets
- ArBanking77: This is the primary dataset used for the AraFinNLP 2024 shared task. It's an Arabic version of the English Banking77 dataset, containing 39,315 annotated queries across 77 intents in the banking domain. It includes various Arabic dialects such as MSA, Moroccan, Saudi, Palestinian, and Tunisian. The dataset was split into training, validation, and test sets for the shared task.
Key Insights and Ideas
- Hierarchical Classification Wins: Grouping intents into broader categories before fine-grained classification significantly improved performance for most models compared to direct fine-tuning. This suggests that tackling the problem in stages, leveraging shared context within intent groups, is more effective.
- Dialect Translation's Mixed Impact: Translating queries to MSA before classification showed promise, particularly when the translated queries were combined with original queries during fine-tuning. However, evaluating on translated queries sometimes led to worse results, possibly due to propagated translation errors.
- MARBERTv2 Shines: The MARBERTv2 model, a further fine-tuned version of MARBERT, demonstrated strong performance, especially with the hierarchical approach.
- Domain-Specific Models Matter: The performance of models like MARBERT and QARIB, which have been pre-trained or fine-tuned on Arabic data, aligns with previous studies and suggests their suitability for Arabic NLP tasks.
Key Results
The team's official submission, using the MARBERTv2 model with the two-step hierarchical classification approach (trained on original queries), achieved a macro F1 score of 0.7866, ranking them 9th in the AraFinNLP 2024 shared task.
On the Palestinian validation set, the hierarchical approach generally improved F1 scores. For instance, MARBERTv2 saw an improvement from 0.8961 (direct fine-tuning) to 0.9470 (two-step hierarchical). When translation was incorporated into the two-step approach and evaluated on the original PAL validation set, MARBERT achieved the highest F1 score of 0.9473, followed closely by MARBERTv2 at 0.9320.
Limitations and Future Work
The authors acknowledge that while the two-step approach improved performance, further refinement is still needed. They suggest that future work could explore:
- Using different models for different groups: Instead of a single hierarchical structure, specialized models could be chosen for each parent class based on its characteristics.
- Experimenting with different grouping criteria: The current grouping was based on observed semantic similarity; exploring alternative grouping strategies might yield better results.
- Addressing time constraints: Testing all possible permutations of models and groupings is time-consuming, but could lead to further performance gains.
Summary updated