Introduction
Multimodal AI systems, which process both visual and textual information, have seen rapid advancement in recent years. However, these models are predominantly trained on high-resource Western languages, leading to a significant "digital divide." African languages, representing hundreds of millions of speakers, are often excluded due to a lack of curated datasets and models tailored to their unique linguistic structures. AfriCaption addresses this gap by introducing the first large-scale image captioning framework specifically designed for 20 African languages.
Methodology
The researchers developed a dual-pronged approach involving a robust data generation pipeline and a specialized model architecture.
Context-Preserving Data Pipeline
To generate high-quality captions without the prohibitive costs of manual translation, the team implemented a "Cascade Ensemble" of Machine Translation (MT) models, including NLLB-200 and Azure Translate. To ensure semantic fidelity between the original image and the translated text, they utilized Language-Agnostic BERT Sentence Embeddings (LaBSE) for similarity checking. This pipeline is designed to be adaptive, allowing for the substitution of improved MT models as they become available.
The AfriCaption Model Architecture
The core model is a 0.5B parameter vision-to-text system. It leverages SigLIP as a state-of-the-art vision encoder and NLLB-200 as a text decoder, connected via a linear projector that aligns visual features with the text space. The training process involved two stages: initial modality alignment followed by full multimodal pretraining. Notably, the researchers chose to unfreeze the vision encoder during the second stage, allowing the model to better capture spatial and relational signals specific to the target languages.
The AfriCaption Dataset
The AfriCaption dataset is built upon the Flickr8k benchmark, extending it to 8,000 images with semantically aligned captions across 20 African languages, including Yoruba, Igbo, Hausa, and Luganda. This corpus covers a broad linguistic spectrum, including the Afro-Asiatic, Niger-Congo, and Nilo-Saharan families. Unlike existing datasets like WIT, which have unspecified African language coverage, AfriCaption provides a curated, high-quality corpus specifically optimized for image-to-text tasks.
Results and Discussion
AfriCaption significantly outperforms existing state-of-the-art multilingual benchmarks. When compared to Pangea, a leading open-weight model, AfriCaption achieved a CIDEr score of 8.40 for Igbo, whereas Pangea's performance was near zero. The model showed particularly strong results in widely spoken languages like Yoruba and Hausa.
Human evaluation revealed that while many translations were excellent, low-resource MT can be "hit-or-miss," sometimes failing on idiomatic expressions (e.g., literal translations of English idioms). Furthermore, the decision to unfreeze the vision encoder during training proved crucial for the model's ability to interpret relational signals in the target languages. The dataset also maintains a consistent average caption length across all 20 languages, preventing model bias.
Conclusion and Future Work
While AfriCaption marks a major step forward, challenges remain regarding translation consistency across all 20 languages and the need for culturally specific imagery beyond Western-centric benchmarks like Flickr8k. Future work will focus on a "participatory approach" with native speaker communities to further refine captions and incorporate deeper cultural nuances that resonate with the lived realities of African communities.
Summary updated