dzFinNlp at AraFinNLP: Improving Intent Detection in Financial Conversational Agents

Mohamed Lichouri, Khaled Lounnas, Amziane ZakariaACL 2024

TL;DR

This paper introduces dzFinNlp, a system for intent detection in financial conversational agents for Arabic dialects. They explored various machine learning and deep learning models, achieving a strong micro F1-score of 93.02% on the ArBanking77 dataset, demonstrating the effectiveness of traditional TF-IDF with weighted features for this challenging task.

What Problem Does This Paper Address?

The financial sector in the Arab world is rapidly expanding, yet sophisticated Natural Language Processing (NLP) tools tailored for Arabic are still developing. Conversational agents in finance need to accurately understand customer intents, which is complicated by the linguistic diversity and dialectal variations within Arabic. The AraFinNLP shared task aims to address this by focusing on intent detection in financial contexts across different Arabic dialects, highlighting the need for robust models that can handle this complexity.

Methods

The authors experimented with a range of approaches:

  • Traditional Machine Learning: They started with LinearSVC combined with TF-IDF (Term Frequency-Inverse Document Frequency) features. This is a standard baseline for text classification.
  • Feature Engineering: They explored different n-gram lengths (1-grams, 3-grams, 5-grams) for words and characters, including word boundaries (char_wb), to capture richer linguistic patterns. They also investigated weighting these TF-IDF features to emphasize certain n-gram types.
  • Deep Learning:
    • LSTM (Long Short-Term Memory) Networks: These were used to capture sequential information in text, a common technique for sequence modeling.
  • Transformer-based Models: They leveraged pre-trained models like XLM-ROBERTa and Sentence Transformers to generate sentence embeddings, aiming to capture the overall semantic meaning of queries.

Datasets

The primary dataset used is ArBanking77, provided by the AraFinNLP shared task.

  • It's derived from the English Banking77 dataset and translated into Modern Standard Arabic (MSA) and Palestinian Arabic (PAL).
  • It's expanded to include other Arabic dialects (Gulf, Levantine, North African Arabic).
  • The dataset contains 31,404 queries categorized into 77 intent classes.
  • For their experiments, they focused on the Palestinian Arabic (PAL) subset for training and validation, aiming to specialize their model.

Key Insights and Ideas

  • Traditional Methods Still Shine: Surprisingly, the best-performing model used a traditional TF-IDF approach with carefully tuned n-grams and class weights, outperforming more complex deep learning models in their experiments. This suggests that for specific tasks like intent detection in a domain with rich linguistic features, well-engineered traditional features can be highly effective.
  • Dialect Specialization Matters: Focusing on a specific dialect (Palestinian Arabic) for training allowed their model to be finely tuned, potentially leading to better performance on that dialect.
  • Transformer Potential: While their initial experiments with pre-trained Transformers didn't surpass the best traditional model, the authors note their strong performance (around 75.76% F1-score) and suggest that fine-tuning these models specifically for the Arabic financial domain could unlock their full potential.

Key Results

The authors reported the following key results on the development set:

  • Best Traditional Model: Achieved a micro F1-score of 93.02% using a combination of 3, 4, and 5-grams for text features, class weights, and a specific regularization parameter (C=5).
  • LSTM Performance: Achieved competitive F1-scores, with the best reaching 79.84%.
  • Transformer Performance: Pre-trained Transformer models achieved an F1-score of 75.76%.

On the test set, their best model achieved a micro F1-score of 67.21%.

Limitations and Future Work

  • Transformer Fine-tuning: The authors acknowledge that their Transformer models were not specifically fine-tuned for the Arabic intent detection task. They suggest that further investigation into fine-tuning these models could lead to improved performance.
  • Hybrid Approaches: They propose exploring hybrid approaches that combine the strengths of traditional and deep learning methods to potentially achieve even better results.
  • Statistical Significance: While they observed slight improvements with hyperparameter tuning, further analysis is needed to determine statistical significance.

Summary updated

BibTeX
@inproceedings{lichouri2024dzfinnlp,
  author = {Mohamed Lichouri and Khaled Lounnas and Amziane Zakaria},
  title = {{dzFinNlp at AraFinNLP: Improving Intent Detection in Financial Conversational Agents}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure