Towards sound measurement of stereotypes and attitudes in humans & language models

Open Access
Authors
Supervisors
Award date 25 November 2026
ISBN
  • 9789493603196
Series ILLC Dissertation Series, DS-2026-15
Number of pages 277
Organisations
  • Faculty of Humanities (FGw)
  • Interfacultary Research - Institute for Logic, Language and Computation (ILLC)
Abstract
Conclusions drawn based on empirical data are valid only to the extent to which data mean what one thinks they mean. The goal of this dissertation is hence to contribute to the development of high-quality measurement tools.
Part 1 of this dissertation bridges Natural Language Processing (NLP; i.e., the subfield of artificial intelligence specialized in acting on human language) and Psychometrics (i.e., the subfield of psychology concerned with the theory and practice of measurement). We argue that psychometrics provides NLP with theoretical lenses (i.e., reliability and validity, Chapter 2, and item response theory, Chapter 3) that can help address shortcomings of contemporary NLP bias measurement tools. Additionally, to support the validation of NLP gender bias measurement tools, we provide a dataset of human ratings for occupational gender stereotypes (Chapter 4).
Part 2 explores theoretical assumptions underlying two approaches to measuring and approximating human opinions. In Chapter 5, we test whether evidence for gender stereotypes can be obtained with a sequential priming paradigm, if participants are unaware of what we attempt to measure. In Chapter 6, we discuss a pilot (questionnaire) study, in which we measured people’s news consumption and opinions. This was part of a project to evaluate silicon sampling – an approach to letting Large Language Models predict how groups of people respond to questionnaire items.
The main argument of this dissertation is that assumptions that underlie measurement tools should be explicated, communicated, scrutinized, and tested.
Document type PhD thesis
Language English
Downloads
Permalink to this page
cover
Back