Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation

PDF Version Also Available for Download.

Description

In this paper, the authors discuss research on whether they can use Mechanical Turk (MTurk) to acquire goo annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task.

Physical Description

9 p.

Creation Information

Akkaya, Cem; Conrad, Alexander; Wiebe, Janyce M. & Mihalcea, Rada, 1974- June 2010.

Context

This paper is part of the collection entitled: UNT Scholarly Works and was provided by UNT College of Engineering to Digital Library, a digital repository hosted by the UNT Libraries. It has been viewed 384 times . More information about this paper can be viewed below.

Who

People and organizations associated with either the creation of this paper or its content.

Authors

Publisher

Provided By

UNT College of Engineering

The UNT College of Engineering promotes intellectual and scholarly pursuits in the areas of computer science and engineering, preparing innovative leaders in a variety of disciplines. The UNT College of Engineering encourages faculty and students to pursue interdisciplinary research among numerous subjects of study including databases, numerical analysis, game programming, and computer systems architecture.

Contact Us

What

Descriptive information to help identify this paper. Follow the links below to find similar items on the Digital Library.

Degree Information

Description

In this paper, the authors discuss research on whether they can use Mechanical Turk (MTurk) to acquire goo annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task.

Physical Description

9 p.

Notes

Abstract: Amazon Mechanical Turk (MTurk) is a marketplace for so-called "human intelligence tasks" (HITs), or tasks that are easy for humans but currently difficult for automated processes. Providers upload tasks to MTurk which workers then complete. Natural language annotation is one such human intelligence task. In this paper, the authors investigate using MTurk to collect annotations for Subjectivity Word Sense Disambiguation (SWSD), a course-grained word sense disambiguation task. The authors investigate whether they can use MTurk to acquire good annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task. While our results with respect to spammers are inconclusive, the authors are able to obtain high-quality annotations for the SWSD task. These results suggest a greater role for MTurk with respect to constructing a large scale SWSD system in the future, promising substantial improvement in subjectivity and sentiment analysis.

Source

  • North American Chapter of the Association for Computational Linguistics Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010, Los Angeles, California, United States

Language

Item Type

Collections

This paper is part of the following collection of related materials.

UNT Scholarly Works

Materials from the UNT community's research, creative, and scholarly activities and UNT's Open Access Repository. Access to some items in this collection may be restricted.

What responsibilities do I have when using this paper?

When

Dates and time periods associated with this paper.

Creation Date

  • June 2010

Added to The UNT Digital Library

  • Jan. 31, 2011, 2:01 p.m.

Description Last Updated

  • July 12, 2013, 4:32 p.m.

Usage Statistics

When was this paper last used?

Yesterday: 0
Past 30 days: 1
Total Uses: 384

Interact With This Paper

Here are some suggestions for what to do next.

Start Reading

PDF Version Also Available for Download.

Citations, Rights, Re-Use

Akkaya, Cem; Conrad, Alexander; Wiebe, Janyce M. & Mihalcea, Rada, 1974-. Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation, paper, June 2010; [Stroudsburg, Pennsylvania]. (digital.library.unt.edu/ark:/67531/metadc31023/: accessed September 24, 2017), University of North Texas Libraries, Digital Library, digital.library.unt.edu; crediting UNT College of Engineering.