CultranAI at PalmX 2025: Data Augmentation for Cultural Knowledge Representation

TL;DR

CultranAI tackles the challenge of cultural knowledge representation in Arabic LLMs for the PalmX 2025 shared task. By leveraging LLM-driven data augmentation to scale training data with over 22,000 culturally grounded multiple-choice questions and fine-tuning the Arabic-centric Fanar-7B model, the system achieved 5th place on the blind test set with a 70.50% accuracy rate.

What Problem Does This Paper Address?

While large language models (LLMs) excel at standard NLP tasks, they frequently stumble on queries requiring deep, region-specific cultural knowledge, local customs, or dialectal nuances. This is especially true for Arabic, where rich cultural diversity spans 22 countries with significant dialectal variations.

The PalmX 2025 Shared Task (Subtask 1) addresses this gap by benchmarking models on multiple-choice questions (MCQs) covering Arab history, geography, literature, and customs in Modern Standard Arabic (MSA). The authors aim to build a system that accurately captures this cultural literacy under a strict 13-billion parameter constraint, making it viable for low-compute environments.

Methods

The authors' approach, named CultranAI, combines zero-shot model selection, automated data augmentation, and Parameter-Efficient Fine-Tuning (PEFT).

• Model Selection: They first benchmarked several multilingual and Arabic-centric models in a zero-shot setup to find the strongest base model.
• Data Augmentation Pipeline: To expand the limited training data, they used a two-pronged strategy:
• Format Conversion: Converted the existing free-form Palm dataset into an MCQ format using GPT-4 to generate three plausible distractors per question.
• NativQA Framework Expansion: Used GPT-4 to identify the target country for each question in the training set, then fed this country data into the NativQA framework to retrieve location-specific QA pairs from verified web domains. These were filtered for cultural relevance and converted into ~22,000 new MCQs (PalmX-ext).
• Fine-Tuning: They compared full-precision LoRA against 4-bit quantized QLoRA on the best-performing base model, followed by extensive hyperparameter tuning on learning rates, dropout, and LoRA rank/alpha parameters.

Expand

Datasets

The study utilizes three main datasets:

• PalmX: The official shared task dataset consisting of 2,000 training, 500 development, and 2,000 blind test MCQs in MSA.
• Palm: A community-curated dataset spanning MSA and various dialects across 22 Arab countries. The authors adapted 1,926 available QA pairs, splitting them into 950 for training and 950 for evaluation.
• PalmX-ext: A newly curated synthetic dataset of 22,000 MCQs generated via the NativQA framework to boost regional diversity.

Expand

Key Insights and Ideas

• Arabic-Centric Pre-training Wins: In zero-shot evaluations, models specifically pre-trained or instruction-tuned on Arabic data (like Fanar-7B and ALLaM-7B) consistently outperformed broader multilingual models (like Llama-3.1-8B), highlighting the necessity of language-specific foundation training for cultural tasks.
• Quantization Has Minimal Cost: The performance gap between full-precision LoRA (80.60%) and 4-bit quantized QLoRA (80.00%) was remarkably small. This demonstrates that QLoRA is a highly viable strategy for culturally aligning models in low-compute environments.
• Data Augmentation Drives Generalization: Adding synthetic, location-specific data (PalmX-ext) and cross-dialect data (Palm) provided a significant performance boost (+4.0% accuracy on the dev set), proving that LLM-driven data curation can successfully fill regional knowledge gaps.
• The "Hidden" Domain Shift: The authors noticed a sharp drop in performance on the final blind test set. Analysis revealed a domain shift: while the training/dev sets focused heavily on contemporary social and institutional knowledge, the test set introduced entirely new domains like ancient history, legal systems, and zoology.

Key Results

• Base Model Selection: Fanar-7B achieved the highest zero-shot accuracy on the PalmX Dev set at 72.40%.

Expand

• PEFT Comparison: Full-precision LoRA slightly edged out QLoRA (80.60% vs 80.00%) under identical settings.

Expand

• Ablation of Augmentation: Fine-tuning on PalmX alone yielded 76.6% accuracy on the Dev set, whereas incorporating the augmented datasets boosted accuracy to 80.6%.

Expand

• Hyperparameter Tuning: The optimal configuration (3 epochs, LR $2 \times 10^{-4}$, LoRA rank 64, dropout 0.05, alpha 16) achieved 80.5% on Dev before incorporating the dev set into final training.

Expand

• Final Standings: The final CultranAI system achieved 84.1% accuracy on the Palm set and ranked 5th on the official PalmX blind test set with an accuracy of 70.50%.

Limitations and Future Work

The authors identified several data-quality bottlenecks:

• Token Limits: Directly converting instructional QA pairs from the Palm dataset occasionally caused sequences to exceed PalmX's 512-token limit.
• Distractor Quality: The synthetic distractors generated by GPT-4 were frequently shorter than the correct answers, introducing a superficial length bias that models could exploit rather than relying on actual cultural knowledge.
• Domain Generalization: The model struggled with out-of-distribution cultural domains present in the test set.

Future work will focus on refining the data augmentation pipelines (e.g., adding truncation steps and prompt engineering for better distractor balance) and exploring broader model generalizability across unseen cultural topics.