Abstract
Visually impaired people are a large group who can only use braille for reading and writing. However, the lack of special educational resources is the bottleneck for educating them. Educational equity is a reflection of the level of social civilization, cultural equality, and individual dignity. Facilitating and improving lifelong learning channels for the visually impaired is of great significance. Their written braille homework or exam papers cannot be understood by sighted teachers, because of the lack of a highly accurate braille translation system, especially in Chinese which has tone marks. braille writers often omit tone marks to save space, leading to confusion when braille with the same consonants and vowels is translated into Chinese. Previous algorithms were insufficient in extracting contextual information, resulting in low accuracy of braille translations into Chinese. This project informatively fine-tuned the mT5 model with an Encoder-decoder architecture for braille to Chinese character conversion. This research created a training set of braille and corresponding Chinese text from the Leipzig Corpora. This project significantly reduced the confusion in braille, achieving 62.4 and 62.3 BLEU scores in the validation and test sets, with a curriculum learning fine-tuning method. By incorporating the braille recognition algorithm, this project is the first publicly available braille translation system and can benefit lots of visually impaired students and families who are preparing for the Chinese College Test and help to propel their college dreams in the future.
1. Introduction
In China, approximately 150,000 school-age blind children face a significant educational barrier. While specialized college entrance exams (GaoKao) exist for blind students, participation is remarkably low—often fewer than 20 students nationwide in a given year. A primary cause is the "double-blind" communication gap: sighted teachers in mainstream or even some special education settings cannot easily read or grade Braille homework and exams.
The technical challenge is rooted in the phonetic nature of Chinese Braille. It represents consonants, vowels, and tones; however, to save space, writers omit tone marks up to 90% of the time. This leads to a "homophone explosion" where a single phonetic string (e.g., "yi") can correspond to over a hundred different Chinese characters. Previous translation attempts failed to capture sufficient context to disambiguate these characters effectively.
2. Methods
Vision-Braille addresses this through an end-to-end pipeline that converts physical Braille images into digital Chinese text. The system consists of two primary modules:
- Braille OCR (Image-to-Braille): The system utilizes RetinaNet, a one-stage object detection framework enhanced with a Feature Pyramid Network (FPN). This model identifies individual dots within Braille cells from images and converts them into digital Braille characters.
- Braille Translation (Braille-to-Text): The core translation engine is a fine-tuned mT5-Small model. mT5 is a multilingual text-to-text transformer that excels at sequence-to-sequence tasks. To optimize it for this task, the researchers expanded the tokenizer vocabulary with 63 specific Braille characters as special tokens.
2.1 Curriculum Learning Strategy
To handle the tone-omission problem, the authors developed a novel three-stage curriculum learning approach:
- Stage 1 (Easy): Training on "Full-Tone" Braille to establish basic translation rules.
- Stage 2 (Medium): Training on "No-Tone" Braille to force the model to rely entirely on context for character selection.
- Stage 3 (Hard/Real-world): Fine-tuning on a dataset where only 10% of tones are present, mimicking actual writing habits.
3. Datasets
Due to the lack of publicly available Chinese Braille datasets, the authors constructed a large-scale corpus using one million sentences from the Leipzig Corpora Collection (news media from 2007–2009). These were converted into Braille using the "Braille Online Platform of China."
The final dataset includes over 525,000 training samples. The researchers generated three variants—Full-Tone, No-Tone, and 10%-Tone—to support the curriculum learning strategy and provide a realistic benchmark for performance evaluation.
4. Key Results and Insights
The Vision-Braille system achieved a BLEU score of 62.4 on the validation set and 62.3 on the test set for the 10%-tone real-world simulation. These results demonstrate that the transformer-based mT5 model effectively captures long-range contextual dependencies required for disambiguation, outperforming simpler RNN or N-gram models.
Beyond technical metrics, the project emphasizes accessibility as a complete pipeline. By integrating OCR with translation, the tool allows a teacher to simply photograph a student's work to receive an accurate translation, bridging the gap between Braille writers and sighted educators.
5. Limitations and Future Work
While Vision-Braille represents a significant step forward, several areas for improvement remain:
- Granularity: The current model is optimized for sentence-level translation. Future iterations aim to support passage-level processing to handle long-form essays and full exam papers.
- Model Scaling: The researchers utilized the "Small" version of mT5 (~300M parameters). Scaling to larger models or utilizing few-shot learning with state-of-the-art LLMs could further enhance accuracy.
- Environmental Robustness: Future work will focus on improving OCR performance under non-ideal conditions, such as poor lighting or worn-down Braille dots on physical paper.
Summary updated