Image Synthesis with Transformer

Posted on Edited on Views:

Projects Title

Image Synthesis with Transformer: Brain MR Image to PET Image

Location

University of Wisconsin-Madison, USA

Time

2022.9 - present

Advisor

Vikas Singh, PH. D, Professor in the Department of Biostatistics

Yong Jae Lee, PH. D, Associate Professor in the Department of Computer Sciences

Description

To reduce the financial burden on the patient and to speed up the diagnostic process, we propose a novel unsupervised GAN framework to achieve the goal of using magnetic resonance imaging (MRI), an inexpensive and easy-to-use technique, to generate certain types of images of positron emission to mography (PET), which is an expensive and complex time-consuming technique.

Implementation

Our key idea is to use CycleGAN to map the input image to a specified one in the output domain and use a combination of MobileNetV2 and self-attention in the process to achieve global and local feature capture of the input photo. We present experiments on the Alzheimers Disease Neuroimaging Initiative (ADNI) dataset with 726 training and 80 testing subjects and obtain acceptable per- formance in PET image synthesis. We also use various metrics to evaluate the generated images.

GitHub LINK: https://github.com/GeofrreyLi/U-TransCyGan

Paper draft(on going):

**Related study:

Swapping Auto-Encoder presentation: Slides