In plain words: Word vectors trained to predict nearby words answer analogies like man:woman :: king:? by addition and subtraction, but nobody could prove why. A mathematical analysis shows the training itself builds straight-line relation patterns and automatically shrinks frequent words' influence, without earlier explanations' strong assumptions.
Abstract
A surprising property of word vectors is that word analogies can often be solved with vector arithmetic. However, it is unclear why arithmetic operators correspond to non-linear embedding models such as skip-gram with negative sampling (SGNS). We provide a formal explanation of this phenomenon without making the strong assumptions that past theories have made about the vector space and word distribution. Our theory has several implications. Past work has conjectured that linear substructures exist in vector spaces because relations can be represented as ratios; we prove that this holds for SGNS. We provide novel justification for the addition of SGNS word vectors by showing that it automatically down-weights the more frequent word, as weighting schemes do ad hoc. Lastly, we offer an information theoretic interpretation of Euclidean distance in vector spaces, justifying its use in capturing word dissimilarity.
Kawin Ethayarajh, David Duvenaud, Graeme Hirst
arXiv:1810.04882 · cs.CL · submitted Oct 11, 2018 · updated Sep 7, 2026
abstract · pdf · html · Accepted to ACL 2019
TL;DR: We prove that linear word analogies hold over a set of ordered pairs (e.g., {(Paris, France), (Ottawa, Canada), ...}) in an SGNS or GloVe embedding space with no reconstruction error when PMI(x,y) + log p(x,y) is the same for every word pair (x,y). We call this term the csPMI (co-occurrence shifted PMI). This has a number of interesting implications:
1. It implies that Pennington et al. (authors of GloVe) had the right intuition about why these analogies hold.
2. Adding two word vectors together to compose them makes sense, because you're implicitly downweighting the more frequent word -- like TF-IDF or SIF would do explicitly.
3. Using Euclidean distance to measure word dissimilarity make sense because the Euclidean distance is a linear function of the negative csPMI.