High fidelity medical image-to-image translation

Abstract

Despite much recent progress in image-to-image translation, it remains challenging to apply such techniques to medical images. We develop a novel parameterization of conditional generative adversarial networks that achieves high image fidelity when trained to transform magnetic resonance images (MRIs) conditioned on a patient's age and disease severity. The spatial-intensity transform generative adversarial network (SIT-GAN) constrains the generator to a smooth spatial transform composed with sparse intensity changes. This technique improves image quality and robustness to artifacts, and generalizes to different scanners. Our model achieves state of the art predictions of longitudinal brain MRIs without supervised training on paired scans. We also demonstrate SIT-GAN on a large clinical image dataset of stroke patients, where it captures associations between ventricle expansion and aging, as well as between white matter hyper intensities and stroke severity. Additionally, SIT-GAN provides a disentangled view of anatomical and textural changes with each transformation, making it easier to interpret the model's predictions in terms of physiological phenomena. As conditional generative models become increasingly versatile tools for data exploration, visualization and forecasting, such techniques for improving robustness are critical for their translation to clinical settings.