Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation

Description:

In this paper, the authors discuss research on whether they can use Mechanical Turk (MTurk) to acquire goo annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task.

Creator(s):
Creation Date: June 2010
Partner(s):
UNT College of Engineering
Collection(s):
UNT Scholarly Works
Usage:
Total Uses: 297
Past 30 days: 5
Yesterday: 0
Creator (Author):
Akkaya, Cem

University of Pittsburgh

Creator (Author):
Conrad, Alexander

University of Pittsburgh

Creator (Author):
Wiebe, Janyce M.

University of Pittsburgh

Creator (Author):
Mihalcea, Rada, 1974-

University of North Texas

Publisher Info:
Place of Publication: [Stroudsburg, Pennsylvania]
Date(s):
  • Creation: June 2010
Description:

In this paper, the authors discuss research on whether they can use Mechanical Turk (MTurk) to acquire goo annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task.

Degree:
Note:

Abstract: Amazon Mechanical Turk (MTurk) is a marketplace for so-called "human intelligence tasks" (HITs), or tasks that are easy for humans but currently difficult for automated processes. Providers upload tasks to MTurk which workers then complete. Natural language annotation is one such human intelligence task. In this paper, the authors investigate using MTurk to collect annotations for Subjectivity Word Sense Disambiguation (SWSD), a course-grained word sense disambiguation task. The authors investigate whether they can use MTurk to acquire good annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task. While our results with respect to spammers are inconclusive, the authors are able to obtain high-quality annotations for the SWSD task. These results suggest a greater role for MTurk with respect to constructing a large scale SWSD system in the future, promising substantial improvement in subjectivity and sentiment analysis.

Physical Description:

9 p.

Language(s):
Subject(s):
Keyword(s): word sense disambiguations | natural language annotations
Source: North American Chapter of the Association for Computational Linguistics Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010, Los Angeles, California, United States
Contributor(s):
Partner:
UNT College of Engineering
Collection:
UNT Scholarly Works
Identifier:
  • ARK: ark:/67531/metadc31023
Resource Type: Paper
Format: Text
Rights:
Access: Public