Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
Author(s)
Chauhan, Geeticka; Liao, Ruizhi; Wells, William; Andreas, Jacob; Wang, X; Berkowitz, S; Horng, S; Szolovits, Peter; Golland, Polina; ... Show more Show less
DownloadSubmitted version (17.89Mb)
Open Access Policy
Open Access Policy
Creative Commons Attribution-Noncommercial-Share Alike
Terms of use
Metadata
Show full item recordAbstract
We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiographs at inference time. Our experimental results suggest that the joint image-text representation learning improves the performance of pulmonary edema assessment compared to a supervised model trained on images only. We also show the use of the text for explaining the image classification by the joint model. To the best of our knowledge, our approach is the first to leverage free-text radiology reports for improving the image model performance in this application. Our code is available at: https://github.com/RayRuizhiLiao/joint_chestxray.
Date issued
2020-09Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer ScienceJournal
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Citation
Chauhan, Geeticka et al. "Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment."
MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention,
Lecture Notes in Computer Science, 12262, Springer International Publishing, 2020, 529-539. © 2020 Springer Nature
Version: Original manuscript
ISBN
978-3-030-59712-2
978-3-030-59713-9
ISSN
0302-9743
1611-3349