TL;DR
This paper tackles the challenge of generating coherent and diverse Arabic stories using Large Language Models (LLMs). The authors introduce novel datasets and fine-tuning strategies for the Arabic LLM AraLLaMA, demonstrating its competitive performance against larger models and achieving strong results in both Modern Standard Arabic (MSA) and two major dialects.
What Problem Does This Paper Address?
While LLMs have revolutionized text generation in many languages, their application to Arabic story generation has lagged due to a scarcity of relevant data and limited research focus. Existing Arabic NLP efforts often rely on cross-lingual transfer or struggle with dialectal nuances. This paper aims to bridge this gap by developing and evaluating LLM-based story generation specifically for Arabic, including its diverse dialects, and establishing a framework for its evaluation.
Methods
The core of this work involves fine-tuning the Arabic LLM, AraLLaMA-2-base. Two primary fine-tuning strategies were explored:
- Direct Fine-tuning: Using a synthetic dataset generated by GPT-4, tailored to Arabic contexts and dialects.
- Two-Step Fine-tuning: First, fine-tuning on a translated dataset (TinyStories) and then further fine-tuning on the GPT-4 generated data.
The authors also developed a sophisticated prompt template incorporating 12 features (e.g., age, setting, plot twist) to guide GPT-4 in generating high-quality and diverse stories across MSA and two dialects (Egyptian and Moroccan). Evaluation was conducted using both GPT-4 as an automated judge and human evaluators.
Datasets
- Translated TinyStories Dataset: 1.13 million English stories from the TinyStories dataset were translated into Arabic using Google Translate API. A rigorous filtering strategy based on multilingual sentence embeddings was applied to ensure high-quality translations, resulting in 545K samples.
- GPT-4 Generated Dataset: 3,000 stories were generated using GPT-4 with custom prompts designed for MSA, Moroccan, and Egyptian dialects. This dataset was crucial for fine-tuning and evaluation.
- Evaluation Prompts: 20 additional prompts were created for evaluating the models.
Key Insights and Ideas
- Dialectal Nuance Matters: While GPT-4 can generate dialectal content, direct prompting in the target dialect (rather than MSA) leads to more uniform and natural-sounding stories, as evidenced by their prompt translation strategy.
- Data Quality Over Quantity (to an extent): The authors' fine-tuned models, despite being significantly smaller than some commercial baselines (e.g., Command-R), achieved competitive performance, suggesting the importance of curated, high-quality Arabic-specific data.
- GPT-4 as a Reliable Evaluator: GPT-4 proved to be a capable judge for Arabic story generation, scoring models on fluency, coherence, instruction following, consistency, and variety. However, it showed a tendency to score MSA content higher, necessitating the explicit "Variety" criterion for dialectal evaluation.
- Two-Step Fine-tuning Benefits: The two-step fine-tuning approach (translated data then GPT-4 data) generally yielded better results, indicating that a broader initial exposure to translated stories can enhance performance.
Key Results
- Competitive Performance: The fine-tuned AraLLaMA models (Model A and Model B) demonstrated strong performance across MSA, Egyptian, and Moroccan dialects, often rivaling or exceeding larger models like GPT-3.5 and Command-R in specific metrics.
- Instruction Following: Model A showed particular strength in instruction following across all varieties, while Model B excelled in consistency and fluency for Egyptian.
- Human Evaluation Agreement: Human evaluations largely aligned with GPT-4's assessments, confirming the models' ability to produce coherent and fluent narratives that adhere to instructions.
- Dialectal Challenges: While models performed well, GPT-4 noted that dialectal generation still lagged behind MSA, and some models (like AceGPT-7B-Chat and GPT-3.5) struggled significantly with dialectal content.
Limitations and Future Work
- Computational Constraints: The study was limited to models with a maximum of 7 billion parameters or those with available APIs due to computational limitations.
- Limited Training Data: The initial training dataset comprised only 3,000 GPT-4 generated samples. Future work plans to generate more data using newer models like GPT-4o.
- Lack of Error Analysis: The authors acknowledge that a detailed error analysis of GPT-4's output would be beneficial to understand its limitations in adhering to instructions.
- Future Directions: The authors plan to fine-tune larger models on more extensive datasets and incorporate a wider range of Arabic dialects to improve coverage and performance. They also suggest future work should address GPT-4's instruction adherence issues.
Summary updated