Vision for action Locomotive affordance perception in brains, behavior, and machines

Human visual perception rapidly extracts behaviorally relevant information from complex visual input. From a single glance, we perceive not only objects and spatial layouts, but also possibilities for action, known as affordances. Yet how such affordances are represented and computed in the human brain remains poorly understood. This thesis investigates the where, when, and how of affordance perception by integrating behavioral experiments, functional magnetic resonance imaging (fMRI), electroencephalography (EEG), and deep neural networks (DNNs). Focusing on six common locomotive actions (walking, biking, driving, climbing, swimming, and boating), I introduce a new operationalization of affordances that captures how people can move through diverse real-world scenes. The results provide evidence that locomotive action affordances form a distinct representational space and are represented in the human visual system, particularly in the occipital place area (OPA) and parahippocampal place area (PPA), independently of object information, scene categories, and low-level visual features. Time-resolved analyses show that unique affordance representations emerge around 200 ms after stimulus onset, with spatiotemporal analyses suggesting temporally distinct contributions of OPA and PPA. In contrast, state-of-the-art DNNs only partially capture human locomotive action affordance representations and align more strongly with object- and low-level visual feature representations. Even explicit training on locomotive action affordance labels does not close this gap. This underscores the need for computational models that more explicitly integrate representations of action affordances, both to advance neuroscientific theories of visual perception and to guide the development of models that better bridge perception and action.