Abstract
For people with visual impairments (PVI), navigating personal smartphone image galleries is a significant challenge due to the lack of descriptive alternative (alt) text. Current accessibility tools often provide only basic metadata, leaving the visual content of memories inaccessible. This paper introduces SaGol, an AI-powered mobile application that utilizes the MiniGPT-4 vision-language model to generate detailed, paragraph-length descriptions of images. By combining these rich descriptions with a semantic search engine based on TF-IDF and cosine similarity, SaGol enables users to find specific photos using natural language queries. Experimental results using the COCO dataset demonstrate that SaGol significantly outperforms traditional models in both descriptive depth and search accuracy.
Introduction
Digital images are often "black boxes" for individuals with visual impairments. While web accessibility standards like WCAG promote the use of alt text, personal photo galleries rarely contain such descriptions. Existing solutions, such as Android's Talkback, typically provide only rudimentary metadata like the date or file name, which fails to help users locate specific memories or understand the visual context of a scene.
Previous automated captioning attempts relied on basic object detection or small-scale AI models. These systems often produced brief, one-sentence descriptions (e.g., "a bicycle") that lacked the environmental context, mood, and specific details necessary for a PVI user to distinguish between similar images. SaGol addresses this gap by leveraging Large Language Models (LLMs) to provide human-like visual synthesis.
System Architecture and Methods
SaGol operates through a two-phase pipeline designed to bridge the gap between high-power vision-language models (VLMs) and mobile accessibility.
Phase I: Alt Text Generation
When a user uploads images to the application, the system sends them to a server running MiniGPT-4. Unlike standard models, MiniGPT-4 is prompted to provide a comprehensive description. This includes the primary subjects, their spatial relationships, the background, and the overall atmosphere of the photo.
Phase II: Semantic Search
To retrieve an image, the user provides a natural language query (e.g., "Find the blue bus"). The system then processes the stored alt text using the following workflow:
- Tokenization: Using NLTK to break down the text into searchable units.
- Vectorization: Applying TF-IDF (Term Frequency-Inverse Document Frequency) to convert descriptions into mathematical vectors.
- Similarity Calculation: Using Cosine Similarity to measure the distance between the user's query and the image descriptions, returning the top 10 most relevant results.
User Interface and Interaction
The SaGol mobile interface is designed for ease of use, focusing on voice-driven interactions. Users can initiate a search via voice recognition, confirm the interpreted search term, and then browse a filtered gallery of the most relevant images. Once an image is selected, the app provides the full, detailed description generated by the AI, allowing the user to "see" the photo through text-to-speech.
Experimental Evaluation
The researchers evaluated SaGol using a subset of 300 images from the COCO (Common Objects in Context) dataset. The study focused on two primary metrics: search accuracy and descriptive power.
Key Results
- Search Accuracy: SaGol achieved a 68.7% Top-10 accuracy rate, significantly higher than traditional models like InceptionV3Small (24.3%) and VGG16 (13.3%), and slightly better than the BLIP model (64.3%).
- Descriptive Power: A major highlight of the study was the verbosity of the descriptions. MiniGPT-4 averaged 110 words per image, whereas the BLIP model averaged only 11 words. This tenfold increase in detail provides the "surface area" necessary for the search engine to match complex queries and offers PVI users a much richer understanding of their photos.
Discussion and Future Work
SaGol demonstrates that LLMs are transformative tools for assistive technology. However, several areas for future development remain. The current study was purely algorithmic; future work must include usability studies with PVI users to determine if 110-word descriptions are optimally helpful or potentially overwhelming. Additionally, the researchers aim to optimize the system for larger galleries (thousands of images) and explore "prompt engineering" to allow users to ask specific questions about their photos, such as "Is the person in this photo smiling?"
While the reliance on a remote AI server introduces some latency, the significant gain in accessibility and searchability marks a major step forward in making personal digital media inclusive for everyone.
Summary updated