Modelling Visual Properties and Visual Context in Multimodal Semantics

Multimodal semantic models that extend linguistic representations with additional perceptual input have proved successful in a range of natural language processing (NLP) tasks. However, existing research has extracted visual features from complete images, and has not examined how different kinds of visual information impact performance. We construct multimodal models that differentiate between internal visual properties of the objects and their external visual context. We evaluate the models on the task of decoding brain activity associated with the meanings of nouns, demonstrating their advantage over those based on complete images.