Towards sound measurement of stereotypes and attitudes in humans & language models
| Authors | |
|---|---|
| Supervisors | |
| Award date | 25 November 2026 |
| ISBN |
|
| Series | ILLC Dissertation Series, DS-2026-15 |
| Number of pages | 277 |
| Organisations |
|
| Abstract |
Conclusions drawn based on empirical data are valid only to the extent to which data mean what one thinks they mean. The goal of this dissertation is hence to contribute to the development of high-quality measurement tools.
Part 1 of this dissertation bridges Natural Language Processing (NLP; i.e., the subfield of artificial intelligence specialized in acting on human language) and Psychometrics (i.e., the subfield of psychology concerned with the theory and practice of measurement). We argue that psychometrics provides NLP with theoretical lenses (i.e., reliability and validity, Chapter 2, and item response theory, Chapter 3) that can help address shortcomings of contemporary NLP bias measurement tools. Additionally, to support the validation of NLP gender bias measurement tools, we provide a dataset of human ratings for occupational gender stereotypes (Chapter 4). Part 2 explores theoretical assumptions underlying two approaches to measuring and approximating human opinions. In Chapter 5, we test whether evidence for gender stereotypes can be obtained with a sequential priming paradigm, if participants are unaware of what we attempt to measure. In Chapter 6, we discuss a pilot (questionnaire) study, in which we measured people’s news consumption and opinions. This was part of a project to evaluate silicon sampling – an approach to letting Large Language Models predict how groups of people respond to questionnaire items. The main argument of this dissertation is that assumptions that underlie measurement tools should be explicated, communicated, scrutinized, and tested. |
| Document type | PhD thesis |
| Language | English |
| Downloads | |
| Permalink to this page | |
