about
Subword Regularization: Improving Neural Network Translation Models (arxiv.org)
1 point by rerx on May 3, 2018 | hide | past | pdf | discuss on HN

In plain words: Words are split into reusable letter chunks before translation, usually one fixed way; here the model trains on many random splits of the same sentence to stay robust. It translated better than fixed splitting, with the biggest gains on small training sets and unfamiliar text.

Abstract · Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

Taku Kudo
arXiv:1804.10959 · cs.CL · submitted Apr 29, 2018
abstract · pdf · html · Accepted as a long paper at ACL2018

add comment on HN