MathReader : Text-to-Speech for Mathematical Documents

Sieun Hyeon, Kyudan Jung, Nam-Joon Kim, Hyun Gon Ryu, Jaeyoung Do

For researchers and students with visual impairments, the "Read Aloud" feature in a PDF viewer is a lifeline. However, anyone who has tried to listen to a math-heavy paper knows the frustration: the software either skips the equations entirely or reads out a garbled mess of LaTeX commands like "backslash frac" and "caret."

A new paper titled "MathReader: Text-to-Speech for Mathematical Documents" introduces a specialized pipeline designed to bridge this gap, turning complex mathematical notation into natural, spoken English.

The MathReader Pipeline

MathReader is an end-to-end pipeline that integrates specialized OCR, a fine-tuned T5 language model, and a TTS engine to accurately read mathematical documents. By treating LaTeX-to-speech as a translation task, it achieves significantly better performance than industry standards.

The five-stage MathReader pipeline: OCR, Formula Extraction, T5 Translation, Replacement, and TTS.

The pipeline consists of five distinct stages:

  1. OCR (Nougat-small): Instead of standard OCR, the authors use Nougat, a transformer-based model specifically trained to recognize academic documents and output them in a markup format (.mmd) that preserves LaTeX formulas.
  2. Formula Extraction: The system identifies mathematical blocks using standard LaTeX delimiters (like \[ ... \]).
  3. Translation (Fine-tuned T5-small): This is the core innovation. The authors treated the problem as a translation task. They fine-tuned a T5-small model to "translate" raw LaTeX code into natural spoken English.
  4. Replacement: The original LaTeX code in the document is replaced with the natural language strings generated by the T5 model.
  5. TTS (VITS): The final, fully English text is fed into VITS, a state-of-the-art end-to-end TTS model, to produce the final audio.

Math as a Language Translation Task

The most significant conceptual shift in this research is the realization that math-to-speech is essentially a translation problem. By using a sequence-to-sequence model (T5), the system can handle the semantic context of an equation—knowing when a symbol represents a power, a fraction, or a specific function.

By opting for T5-small rather than a massive LLM, the researchers proved that a specialized, smaller model can achieve high accuracy while remaining fast enough for practical use. Furthermore, the choice of Nougat as the OCR engine is critical. Standard OCR often mangles mathematical symbols before a TTS system even gets a chance to read them; starting with high-fidelity LaTeX extraction is half the battle.

Evaluation and Results

MathReader was compared against the "Read Aloud" features of Microsoft Edge and Adobe Acrobat using Word Error Rate (WER) and Character Error Rate (CER).

Performance comparison of MathReader against industry standards and an ablation study.

The results demonstrate a substantial improvement in accuracy. MathReader achieved a WER of 0.281, compared to Microsoft Edge (0.510) and Adobe Acrobat (0.617). An ablation study showed that when the T5 translation step was removed (reading raw OCR output), the WER jumped to 0.663, confirming that the "translation" step is the most vital part of the pipeline.

In terms of speed, on an NVIDIA H100 GPU, the system processes a page in roughly 23.62 seconds. Since the average page generates about 134 seconds of audio, the system is more than fast enough to support real-time reading with a small initial buffer.

Conclusion

While MathReader represents a significant leap forward, some limitations remain. The current performance metrics rely on high-end GPUs (H100), suggesting a need for future optimization for consumer-grade hardware. Additionally, extremely non-standard document layouts or very low-resolution scans may still pose challenges for the initial OCR stage. Nevertheless, by treating mathematical notation as a translatable language, MathReader provides a robust framework for making academic literature accessible to all.

Summary updated

BibTeX
@inproceedings{hyeon2025mathreader,
  author = {Sieun Hyeon and Kyudan Jung and Nam-Joon Kim and Hyun Gon Ryu and Jaeyoung Do},
  title = {{MathReader : Text-to-Speech for Mathematical Documents}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=i2d8TEJl0f},
}
Figure