- The Face of Quality in Crowdsourcing Relevance Labels
- Demographics, Personality and Labeling Accuracy
- International Conference on Information and Knowledge Management
- Book/source title
- CIKM’12: The Proceedings of the 21st ACM International Conference on Information and Knowledge Management
- Pages (from-to)
- New York: Association for Computing Machinery
- Document type
- Conference contribution
- Interfacultary Research Institutes
Faculty of Humanities (FGw)
- Institute for Logic, Language and Computation (ILLC)
Information retrieval systems require human contributed relevance labels for their training and evaluation. Increasingly such labels are collected under the anonymous, uncontrolled conditions of crowdsourcing, leading to varied output quality. While a range of quality assurance and control techniques have now been developed to reduce noise during or after task completion, little is known about the workers themselves and possible relationships between workers' characteristics and the quality of their work. In this paper, we ask how do the relatively well or poorly-performing crowds, working under specific task conditions, actually look like in terms of worker characteristics, such as demographics or personality traits. Our findings show that the face of a crowd is in fact indicative of the quality of their work.
- go to publisher's site
If you believe that digital publication of certain material infringes any of your rights or (privacy) interests, please let the Library know, stating your reasons. In case of a legitimate complaint, the Library will make the material inaccessible and/or remove it from the website. Please Ask the Library, or send a letter to: Library of the University of Amsterdam, Secretariat, Singel 425, 1012 WP Amsterdam, The Netherlands. You will be contacted as soon as possible.