Introduction
Automatic image captioning has seen significant progress, yet most advancements remain centered on English. Applying these models to Semitic languages like Arabic is challenging due to complex morphology. Unlike English, where words are often distinct units, Arabic words are typically derived from a core three-letter "root" by adding specific patterns of vowels and consonants. Simple translation from English often yields grammatically incorrect or unnatural results. This paper proposes a framework that generates captions directly in Arabic by leveraging its unique root-word structure.
Methodology
The proposed system follows a three-stage pipeline designed to respect Arabic linguistic rules, moving from visual fragments to structured linguistic roots and finally to full sentences.
- Stage I: Mapping Fragments to Roots: Object fragments are extracted using pre-trained networks (Caffe and ImageNet). Instead of mapping these to full English words, they are mapped directly to Arabic root words.
- Stage II: Morphological Expansion: A Deep Belief Network (DBN), pre-trained with Restricted Boltzmann Machines (RBMs), predicts the necessary vowels and consonants to transform roots into contextually appropriate words (e.g., turning the root k-t-b into "book" or "writer").
- Stage III: Sentence Formation: Dependency tree relations are used to rank and organize these words into syntactically coherent Arabic sentences.
Experimental Setup
The model was trained and tested on two major datasets:
- ImageNet (Arabic Subset): 10,000 images with professional Arabic captions.
- Al-Jazeera News Dataset: 100,000 images with real-world news captions.
The evaluation utilized 80,000 images for training and 30,000 for testing, employing 10-fold cross-validation to ensure robustness.
Results and Discussion
The performance was evaluated using the BLEU-1 score, a standard metric for machine-generated text quality. The results demonstrate that generating captions directly from roots significantly outperforms the common approach of translating English captions via tools like Google Translate.
While the English baseline remains higher (48.4) due to more mature datasets and models, the proposed method's score of 34.8 is a substantial improvement over the 27.2 achieved by translation-based methods. This confirms that a "linguistic-first" design, treating words as Root + Pattern, better captures Arabic's nuances than generic translation tools. Using roots as semantic anchors allows the model to handle a wide variety of derived words while keeping the initial search space for the neural network manageable.
Conclusion and Future Work
This research marks a significant milestone in leveraging Arabic root-word dependency for captioning. By mimicking the actual grammatical logic of the language, the model produces more natural results. Future work will focus on addressing regional dialectal variations and scaling the dependency tree logic to handle more complex, multi-clause sentences. Additionally, integrating these root-word insights into modern Transformer-based architectures represents a promising next step for the field.
Summary updated