An empirical analysis of phrase-based and neural machine translation

Two popular types of machine translation are phrase-based (PBMT) and neural machine translation (NMT) systems. Both of these types of machine translation systems are composed of multiple complex models or layers. Due to their complexity, it is difficult to explain why a system makes certain mistakes during translation or how it is capable of regenerating complex syntactic or semantic phenomena in the target language. Each of these models and layers learns different linguistic aspects of the source language. However, for some of these models and layers, it is not clear which linguistic phenomena are learned or how this information is learned.
To shed light on what linguistic phenomena are captured by these systems, we analyze the behavior of important models in both systems. We consider phrase reordering models from PBMT to investigate which words from inside of a phrase have the biggest impact on the reordering behavior. Additionally, to contribute to the interpretability of NMT systems we study the behavior of the attention model, which is a key component in NMT systems and the closest model in functionality to phrase reordering models in phrase-based systems. The attention model together with the encoder hidden state representations form the main components to encode source side linguistic information in NMT. To this end, we also analyze the information captured in the encoder hidden state representations of an NMT system. We investigate the extent to which syntactic and lexical-semantic information from the source side is captured by hidden state representations of different NMT architectures.