Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation

PDF Version Also Available for Download.

Description

In this paper, the authors discuss research on whether they can use Mechanical Turk (MTurk) to acquire good annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task.

Physical Description

9 p.

Creation Information

Akkaya, Cem; Conrad, Alexander; Wiebe, Janyce M. & Mihalcea, Rada, 1974- June 2010.

Context

This paper is part of the collection entitled: UNT Scholarly Works and was provided by the UNT College of Engineering to the UNT Digital Library, a digital repository hosted by the UNT Libraries. It has been viewed 458 times, with 5 in the last month. More information about this paper can be viewed below.

Who

People and organizations associated with either the creation of this paper or its content.

Authors

Publisher

Provided By

UNT College of Engineering

The UNT College of Engineering strives to educate and train engineers and technologists who have the vision to recognize and solve the problems of society. The college comprises six degree-granting departments of instruction and research.

Contact Us

What

Descriptive information to help identify this paper. Follow the links below to find similar items on the Digital Library.

Degree Information

Description

In this paper, the authors discuss research on whether they can use Mechanical Turk (MTurk) to acquire good annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task.

Physical Description

9 p.

Notes

Abstract: Amazon Mechanical Turk (MTurk) is a marketplace for so-called "human intelligence tasks" (HITs), or tasks that are easy for humans but currently difficult for automated processes. Providers upload tasks to MTurk which workers then complete. Natural language annotation is one such human intelligence task. In this paper, the authors investigate using MTurk to collect annotations for Subjectivity Word Sense Disambiguation (SWSD), a course-grained word sense disambiguation task. The authors investigate whether they can use MTurk to acquire good annotations with respect to gold-standard data, whether they can filter out low-quality workers (spammers), and whether there is a learning effect associated with repeatedly completing the same kind of task. While our results with respect to spammers are inconclusive, the authors are able to obtain high-quality annotations for the SWSD task. These results suggest a greater role for MTurk with respect to constructing a large scale SWSD system in the future, promising substantial improvement in subjectivity and sentiment analysis.

Source

  • North American Chapter of the Association for Computational Linguistics Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk, 2010, Los Angeles, California, United States

Language

Item Type

Identifier

Unique identifying numbers for this paper in the Digital Library or other systems.

Collections

This paper is part of the following collection of related materials.

UNT Scholarly Works

Materials from the UNT community's research, creative, and scholarly activities and UNT's Open Access Repository. Access to some items in this collection may be restricted.

What responsibilities do I have when using this paper?

When

Dates and time periods associated with this paper.

Creation Date

  • June 2010

Added to The UNT Digital Library

  • Jan. 31, 2011, 2:01 p.m.

Description Last Updated

  • Nov. 27, 2017, 11:05 a.m.

Usage Statistics

When was this paper last used?

Yesterday: 0
Past 30 days: 5
Total Uses: 458

Interact With This Paper

Here are some suggestions for what to do next.

Start Reading

PDF Version Also Available for Download.

International Image Interoperability Framework

IIF Logo

We support the IIIF Presentation API

Akkaya, Cem; Conrad, Alexander; Wiebe, Janyce M. & Mihalcea, Rada, 1974-. Amazon Mechanical Turk for Subjectivity Word Sense Disambiguation, paper, June 2010; [Stroudsburg, Pennsylvania]. (https://digital.library.unt.edu/ark:/67531/metadc31023/: accessed September 26, 2020), University of North Texas Libraries, UNT Digital Library, https://digital.library.unt.edu; crediting UNT College of Engineering.