Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information

ow do neural language models keep track of number agreement between subject and verb? We show that ‘diagnostic classifiers’, trained to predict number from the internal states of the language model, provide a detailed understanding of how, when, and where this information is represented. Moreover, they give us insight in when and where this information is corrupted in cases where the language model ends up making agreement errors. To demonstrate the causal role that the representations we find play, we then use this information to influence the course of the LSTM during the processing of difficult sentences. Results from such an intervention show a large increase in the language model’s accuracy. Together, these results show that diagnostic classifiers give us an unrivalled detailed look into the representation of linguistic information in neural models, and moreover demonstrate that this knowledge can be use to improve their performance.