Generative AI methods for synthesis of image data to train AI for automated scene understanding in a military context: a review of opportunities

The rapid increase in sensors on manned and unmanned military platforms has led to a significant rise in image data (e.g., visible, infrared, sonar, radar), enabling extensive scene analysis. Thorough and real-time understanding of these scenes requires automated image analysis tools, for e.g. object detection, traversability analysis, and threat classification. However, the development of artificial intelligence (AI) models for automated scene understanding is constrained by limited access to relevant military training data due to its restricted nature, high acquisition costs, and evolving threat signatures. Several studies highlight the potential of synthetic data as an alternative to measured training data, for example by utilizing physics-based modeling of scenes and objects of interest.

Recent advances in generative AI (GenAI), particularly in diffusion-based models, offer opportunities to synthesize data with variations beyond what was previously possible, improving performance in various nonmilitary image analysis tasks. Despite this, the lack of military-relevant data used for GenAI model development suggests that non-specialized models may produce military scenes with limited quality and variation. In this review, we explore the opportunities of state-of-the-art GenAI methods for creating high-quality training data for military AI systems. We identify three key strategies: (1) full-image generation by fine-tuning with application-specific data; (2) inpainting, where objects of interest can be placed in existing image data; and (3) image-to-image translation which is used to augment image conditions or translate between image modalities. Visual results of each of these methods are promising. Some studies have already shown benefits of these data synthesis methods as data augmentation to improve downstream AI models. Further research shall determine the value for operationalization in a wide set of use-cases.