Capturing Reliable Fine-Grained Sentiment Associations by
Crowdsourcing and Best-Worst Scaling
PRESENTER:
Svetlana Kiritchenko
NRC
ABSTRACT:
Access to word-sentiment associations is useful for many applications,
including sentiment analysis, stance detection, and linguistic analysis.
However, manually assigning fine-grained sentiment association scores to
words has many challenges with respect to keeping annotations consistent.
We apply the annotation technique of Best-Worst Scaling to obtain
real-valued sentiment association scores for words and phrases in four
different domains: English Twitter, Arabic Twitter, English sentiment
modifiers, and English opposing polarity phrases. We show that on all four
domains the ranking of words by sentiment remains remarkably consistent
even when the annotation process is repeated with a different set of
annotators.
We use these fine-grained word-sentiment associations in three ways.
First, we analyze human perception of sentiment and calculate the minimal
difference in sentiment that is detectable by native speakers. This least
perceptible difference helps in our second objective: studying sentiment
composition in phrases that include common sentiment modifiers (such as
negators, modals, and degree adverbs) and in phrases that include words of
opposing polarities. Changes in sentiment incurred in phrases are
considered significant only if they exceed the least perceptible
difference. Finally, as part of a SemEval-2016 shared task, we use the
manually determined sentiment associations to evaluate automatically
generated sentiment lexicons.