about
Multi-Criteria Chinese Word Segmentation with Transformer (arxiv.org)
1 point by sel1 on Jul 2, 2019 | hide | past | pdf | discuss on HN

In plain words: Instead of training separate models for each Chinese word-splitting standard, one shared model reads a special tag saying which standard to use and splits the text accordingly. Across eight datasets it beat both single-standard and other multi-standard systems.

Abstract · A Concise Model for Multi-Criteria Chinese Word Segmentation with Transformer Encoder

Multi-criteria Chinese word segmentation (MCCWS) aims to exploit the relations among the multiple heterogeneous segmentation criteria and further improve the performance of each single criterion. Previous work usually regards MCCWS as different tasks, which are learned together under the multi-task learning framework. In this paper, we propose a concise but effective unified model for MCCWS, which is fully-shared for all the criteria. By leveraging the powerful ability of the Transformer encoder, the proposed unified model can segment Chinese text according to a unique criterion-token indicating the output criterion. Besides, the proposed unified model can segment both simplified and traditional Chinese and has an excellent transfer capability. Experiments on eight datasets with different criteria show that our model outperforms our single-criterion baseline model and other multi-criteria models. Source codes of this paper are available on Github https://github.com/acphile/MCCWS.

Xipeng Qiu, Hengzhi Pei, Hang Yan, Xuanjing Huang
arXiv:1906.12035 · cs.CL, cs.AI · submitted Jun 28, 2019 · updated Oct 5, 2020
abstract · pdf · html · Findings of EMNLP 2020

add comment on HN