On SkipGram Word Embedding Models with Negative Sampling: Unified
Framework and Impact of Noise Distributions
PRESENTER:
Ziqiao Wang
University of Ottawa
ABSTRACT:
SkipGram word embedding models with negative sampling, or SGN in short, is
an elegant family of word embedding models. In this paper, we formulate a
framework for word embedding, referred to as Word-Context Classification
(WCC), that generalizes SGN to a wide family of models. The framework,
utilizing some "noise examples'', is justified through a theoretical
analysis. The impact of noise distribution on the learning of the WCC
embedding models is studied experimentally, suggesting that the best
noise distribution is in fact the data distribution, in terms of both the
embedding performance and the speed of convergence during training. Along
our way, we discover several novel embedding models that outperform the
existing WCC models. This is joint work with Yongyi Mao (University of
Ottawa), Hongyu Guo (National Research Council Canada) and Richong Zhang
(Beihang University).