Finding Patterns in Noisy Crowds: Regression-based Annotation Aggregation for Crowdsourced Data

PDF Version Also Available for Download.

Description

This paper presents an aggregation approach that learns a regression model from crowdsourced annotations to predict aggregated labels for instances that have no expert adjudications.

Physical Description

6 p.

Creation Information

Nielsen, Rodney D. & Parde, Natalie September 2017.

Context

This paper is part of the collection entitled: UNT Scholarly Works and was provided by UNT College of Engineering to Digital Library, a digital repository hosted by the UNT Libraries. More information about this paper can be viewed below.

Who

People and organizations associated with either the creation of this paper or its content.

Authors

Publisher

Provided By

UNT College of Engineering

The UNT College of Engineering strives to educate and train engineers and technologists who have the vision to recognize and solve the problems of society. The college comprises six degree-granting departments of instruction and research.

Contact Us

What

Descriptive information to help identify this paper. Follow the links below to find similar items on the Digital Library.

Degree Information

Description

This paper presents an aggregation approach that learns a regression model from crowdsourced annotations to predict aggregated labels for instances that have no expert adjudications.

Physical Description

6 p.

Notes

Abstract: Crowdsourcing offers a convenient means
of obtaining labeled data quickly and inexpensively.
However, crowdsourced labels
are often noisier than expert-annotated
data, making it difficult to aggregate them
meaningfully. We present an aggregation
approach that learns a regression model
from crowdsourced annotations to predict
aggregated labels for instances that have
no expert adjudications. The predicted labels
achieve a correlation of 0.594 with
expert labels on our data, outperforming
the best alternative aggregation method by
11.9%. Our approach also outperforms the
alternatives on third-party datasets.

Source

  • 2017 Conference on Empirical Methods in Natural Language Processing, September 7-11, 2017. Copenhagen, Denmark.

Language

Item Type

Publication Information

  • Publication Title: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
  • Pages: 1908-1913
  • Peer Reviewed: Yes

Collections

This paper is part of the following collection of related materials.

UNT Scholarly Works

Materials from the UNT community's research, creative, and scholarly activities and UNT's Open Access Repository. Access to some items in this collection may be restricted.

What responsibilities do I have when using this paper?

When

Dates and time periods associated with this paper.

Creation Date

  • September 2017

Added to The UNT Digital Library

  • Nov. 30, 2017, 9:17 a.m.

Usage Statistics

When was this paper last used?

Yesterday: 0
Past 30 days: 0
Total Uses: 6

Interact With This Paper

Here are some suggestions for what to do next.

Start Reading

PDF Version Also Available for Download.

International Image Interoperability Framework

IIF Logo

We support the IIIF Presentation API

Nielsen, Rodney D. & Parde, Natalie. Finding Patterns in Noisy Crowds: Regression-based Annotation Aggregation for Crowdsourced Data, paper, September 2017; Stroudsburg, Pennsylvania. (digital.library.unt.edu/ark:/67531/metadc1042611/: accessed December 18, 2018), University of North Texas Libraries, Digital Library, digital.library.unt.edu; crediting UNT College of Engineering.