Combining images and text for improved medical image understanding

In recent years, the integration of artificial intelligence, especially deep learning models, in medical imaging has had a big impact on disease classification capabilities. However, relying solely on imaging data limits the ability to fully capture the complexities of patient diagnosis. Text data in the form of clinical reports provide additional valuable insights, offering rich clinical information that can complement image-based analysis. Despite their potential, incorporating text information into disease classification models poses significant challenges due to the variability in clinician input and differing information levels across medical text. This thesis addresses these challenges through new methods and approaches for multi-modal learning, aiming to enhance medical image analysis by integrating imaging data with additional sources of clinical information. The over-arching conclusion of the work presented in this thesis is that the combination of information across different modalities in the medical domain is not only possible, but works extremely well in a large number of scenarios. The addition of additional information brings artificial intelligence-based analysis conceptually closer to a clinician's analysis by considering a more complete data representation of a patient.