ML Newsnew | past | best | rss | submit about
Advances in Pre-Training Distributed Word Representations (arxiv.org)
59 points by mpweiher on Dec 30, 2017 | hide | past | pdf | 4 comments on HN

In plain words: Word vectors are lists of numbers that capture what words mean, learned from lots of text. Combining several known training tricks that are rarely used together produced new public word vectors that beat the best current ones by a large margin on many tasks.

Abstract

Many Natural Language Processing applications nowadays rely on pre-trained word representations estimated from large text corpora such as news collections, Wikipedia and Web Crawl. In this paper, we show how to train high-quality word vector representations by using a combination of known tricks that are however rarely used together. The main result of our work is the new set of publicly available pre-trained models that outperform the current state of the art by a large margin on a number of tasks.

Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, Armand Joulin
arXiv:1712.09405 · cs.CL · submitted Dec 26, 2017
abstract · pdf · html

add comment on HN
Also discussed: Dec 2017 (3 points, 0 comments)
thomasahle on Dec 30, 2017 [–]
This is describing the new fasttext vectors available at fasttext.cc.

As they write in the abstract, they don't introduce any new techniques, but I think the paper reads really well as a concise tour for people interested in the current state of the art.

reply
rspeer on Dec 31, 2017 | parent [–]
I'm going to say this is probably state of the art among methods that learn from scratch from a corpus of text. I say "probably" because they skipped most of the usual word-similarity evaluations, only using RW.

The RW results are still lower than any results for ConceptNet Numberbatch [1] in the case where you are able to use a knowledge graph. This is still an advance -- from my point of view, these vectors are an improved input for Numberbatch -- but it continues to surprise me how the possibility of learning from a knowledge graph is not even mentioned when the big players write about these evaluations.

It also pains me that the only analogies evaluated are Mikolov et al. (2013), which is a huge huge case of Not Invented Here just because Mikolov is the first author. It is not a good evaluation [2]. It is the same 10 or so boring analogies over and over. I would much rather see BATS [3] or SemEval-2012, or even Turney's set of real SAT analogies despite their non-free status. But this would require Mikolov to admit that he did not come up with the perfect semantic evaluation off the top of his head, on his first try, without referring to any of the work already done by Turney and others.

[1] https://github.com/commonsense/conceptnet-numberbatch

[2] https://www.aclweb.org/anthology/N/N16/N16-2002.pdf

[3] https://aclweb.org/aclwiki/Bigger_analogy_test_set_(State_of...

reply
gre on Dec 31, 2017 | parent [–]
Aka the old fasttext vectors, aka the fasttext vectors.
reply
thomasahle on Dec 31, 2017 | parent [–]
Oh, I thought that given the article date (Dec 26, 2017) that these were upgraded fasttext vectors. Is this just the first time they've released a paper on them?
reply
About | RSS | RSS (all) | HN arXiv