Annotating and Identifying Emotions in Text Page: 25
18 p.View a full description of this chapter.
Extracted Text
The following text was automatically extracted from the image on this page using optical character recognition software:
Annotating and Identifying Emotions in Text 25
Two data sets were made available: a development data set consisting of 250
annotated headlines, and a test data set consisting of 1,000 annotated headlines.4
3.2 Data Annotation
To perform the annotations, we developed a Web-based annotation interface that
displayed one headline at a time, together with six slide bars for emotions and one
slide bar for valence. The interval for the emotion annotations was set to [0, 100],
where 0 means the emotion is missing from the given headline, and 100 repre-
sents maximum emotional load. The interval for the valence annotations was set to
[-100, 100], where 0 represents a neutral headline, -100 represents a highly nega-
tive headline, and 100 corresponds to a highly positive headline.
Unlike previous annotations of sentiment or subjectivity [45, 32], which typi-
cally rely on binary 0/1 annotations, we decided to use a finer-grained scale, hence
allowing the annotators to select different degrees of emotional load.
The test data set was independently labeled by six annotators. The annotators
were instructed to select the appropriate emotions for each headline based on the
presence of words or phrases with emotional content, as well as the overall feeling
invoked by the headline. Annotation examples were also provided, including exam-
ples of headlines bearing two or more emotions to illustrate the case where several
emotions were jointly applicable. Finally, the annotators were encouraged to follow
their "first intuition," and to use the full-range of the annotation scale bars.
The final annotation labels were created as the average of the six independent
annotations, after normalizing the set of annotations provided by each annotator for
each emotion to the 0-100 range. Table 1 shows three sample headlines in our data
set, along with their final gold standard annotations.
Table 1 Sample headlines and manual annotations of emotions
EMOTIONS
Anger Disgust Fear Joy Sadness Surprise Valence
Inter Milan set Serie A win record 2 0 0 50 0 9 50
Cisco sues Apple over iPhone name 48 8 10 0 11 19 -56
Planned cesareans not risk-free, group 0 0 61 0 15 11 -60
warns
3.3 Inter-annotator Agreement
We conducted inter-tagger agreement studies for each of the six emotions. The
agreement evaluations were carried out using the Pearson correlation measure, and
are shown in Table 2. To measure the agreement among the six annotators, we first
measured the agreement between each annotator and the average of the remaining
five annotators, followed by an average over the six resulting agreement figures.
4 The data set and more information about the task can be found at the SEMEVAL 2007 web
sitehttp://nlp.cs. swarthmore.edu/semeval
Upcoming Pages
Here’s what’s next.
Search Inside
This chapter can be searched. Note: Results may vary based on the legibility of text within the document.
Tools / Downloads
Get a copy of this page or view the extracted text.
Citing and Sharing
Basic information for referencing this web page. We also provide extended guidance on usage rights, references, copying or embedding.
Reference the current page of this Chapter.
Strapparava, Carlo, 1962- & Mihalcea, Rada, 1974-. Annotating and Identifying Emotions in Text, chapter, 2010; [Berlin, Germany]. (https://digital.library.unt.edu/ark:/67531/metadc31010/m1/5/: accessed April 23, 2024), University of North Texas Libraries, UNT Digital Library, https://digital.library.unt.edu; crediting UNT College of Engineering.