In plain words: MathReader reads text and formula images in a document, then uses a trained language model to rewrite each formula as ordinary words before speaking it aloud. On documents full of formulas, its spoken-word errors fell from 0.510 to 0.281 compared with Microsoft Edge's reader.
Abstract · MathReader : Text-to-Speech for Mathematical Documents
TTS (Text-to-Speech) document reader from Microsoft, Adobe, Apple, and OpenAI have been serviced worldwide. They provide relatively good TTS results for general plain text, but sometimes skip contents or provide unsatisfactory results for mathematical expressions. This is because most modern academic papers are written in LaTeX, and when LaTeX formulas are compiled, they are rendered as distinctive text forms within the document. However, traditional TTS document readers output only the text as it is recognized, without considering the mathematical meaning of the formulas. To address this issue, we propose MathReader, which effectively integrates OCR, a fine-tuned T5 model, and TTS. MathReader demonstrated a lower Word Error Rate (WER) than existing TTS document readers, such as Microsoft Edge and Adobe Acrobat, when processing documents containing mathematical formulas. MathReader reduced the WER from 0.510 to 0.281 compared to Microsoft Edge, and from 0.617 to 0.281 compared to Adobe Acrobat. This will significantly contribute to alleviating the inconvenience faced by users who want to listen to documents, especially those who are visually impaired. The code is available at https://github.com/hyeonsieun/MathReader.
Sieun Hyeon, Kyudan Jung, Nam-Joon Kim, Hyun Gon Ryu, Jaeyoung Do
arXiv:2501.07088 · cs.AI, cs.SD, eess.AS · submitted Jan 13, 2025 · updated Jan 19, 2025
abstract · pdf · html · Accepted at ICASSP 2025