In plain words: Instead of automatic splitting, people pick meaningful word chunks to build the vocabulary for training a Fon–French translator, since Fon's tones and marks trip up standard rules. The approach is compared with standard tokenization on translation in both directions.
Abstract · Crowdsourced Phrase-Based Tokenization for Low-Resourced Neural Machine Translation: The Case of Fon Language
Building effective neural machine translation (NMT) models for very low-resourced and morphologically rich African indigenous languages is an open challenge. Besides the issue of finding available resources for them, a lot of work is put into preprocessing and tokenization. Recent studies have shown that standard tokenization methods do not always adequately deal with the grammatical, diacritical, and tonal properties of some African languages. That, coupled with the extremely low availability of training samples, hinders the production of reliable NMT models. In this paper, using Fon language as a case study, we revisit standard tokenization methods and introduce Word-Expressions-Based (WEB) tokenization, a human-involved super-words tokenization strategy to create a better representative vocabulary for training. Furthermore, we compare our tokenization strategy to others on the Fon-French and French-Fon translation tasks.
Bonaventure F. P. Dossou, Chris C. Emezue
arXiv:2103.08052 · cs.CL, cs.AI · submitted Mar 14, 2021 · updated Mar 17, 2021
abstract · pdf