Rethinking the human-centered evaluation of conversational systems

This thesis focuses on evaluating and improving conversational systems in task-based dialogue and open-domain search. We begin by examining user satisfaction in task-oriented systems. We propose a model that captures how different dialogue aspects – such as relevance, task completion, and interest arousal – contribute to satisfaction at both turn and dialogue levels. Building on this, we study how the amount and form of previous dialogue context affect evaluation quality in crowdsourced settings. We introduce an LLM-as-an-assistant approach that provides annotators with either a summary or a user information need description, which improves evaluation label reliability while reducing annotator cognitive load.
We further investigate the role of user feedback in evaluation, showing that follow-up utterances influence judgments and reveal quality signals not evident from system responses alone. Comparisons between human and LLM assessments highlight alignment on factual criteria such as relevance, but expose limitations in capturing subjective aspects like usefulness.
In the second part of the thesis, we focus on clarifying questions as a means to reduce ambiguity in open-domain conversational search. We introduce an end-to-end framework for generating and evaluating clarifying questions using large language models, and demonstrate its scalability and effectiveness. Finally, we extend this work by incorporating images into clarifying questions. Through user studies and retrieval experiments, we show that multimodal clarification improves both user understanding and retrieval performance.
Together, these contributions advance the design and evaluation of conversational systems by providing scalable, context-sensitive, and user-centered methodologies.