MixUp as Locally Linear Out-Of-Manifold Regularization
PRESENTER:
Harry Guo
NRC
ABSTRACT:
MixUp is a recently proposed data-augmentation scheme, which linearly
interpolates a random pair of training examples and correspondingly the
one-hot representations of their labels. Training deep neural networks
with such additional data is shown capable of significantly improving the
predictive accuracy of the current art. The power of MixUp, however, is
primarily established empirically and its working and effectiveness have
not been explained in any depth. In this paper, we develop an
understanding for MixUp as a form of "out-of-manifold regularization",
which imposes certain "local linearity" constraints on the model's input
space beyond the data manifold. This analysis enables us to identify a
limitation of MixUp, which we call "manifold intrusion". In a nutshell,
manifold intrusion in MixUp is a form of under-fitting resulting from
conflicts between the synthetic labels of the mixed-up examples and the
labels of original training data. Such a phenomenon usually happens when
the parameters controlling the generation of mixing policies are not
sufficiently fine-tuned on the training data. To address this issue, we
propose a novel adaptive version of MixUp, where the mixing policies are
automatically learned from the data using an additional network and
objective function designed to avoid manifold intrusion. The proposed
regularizer, AdaMixUp, is empirically evaluated on several benchmark
datasets. Extensive experiments demonstrate that AdaMixUp improves upon
MixUp when applied to the current art of deep classification models.
URL: https://www.aaai.org/Papers/AAAI/2019/AAAI-GuoH.5633.pdf
This is joint work with Yongyi Mao at the University of Ottawa and Richong
Zhang at Beihang University, China