Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment
Name
2008.09884.pdf
Description
Submitted version
Size
17.89 MB
Format
Adobe PDF
Checksum (MD5)
6fbc76a00957497d2d3fa25c014b3c65
Author(s)
Chauhan, Geeticka
Liao, Ruizhi
Wells, William
Andreas, Jacob
Wang, X
Berkowitz, S
Horng, S
Szolovits, Peter
Golland, Polina
Date Issued
September 2020
Journal
Lecture Notes in Computer Science
Publisher
Springer International Publishing
Citation
Chauhan, Geeticka et al. "Joint Modeling of Chest Radiographs and Radiology Reports for Pulmonary Edema Assessment."
MICCAI: International Conference on Medical Image Computing and Computer-Assisted Intervention,
Lecture Notes in Computer Science, 12262, Springer International Publishing, 2020, 529-539. © 2020 Springer Nature
Version
Original manuscript
Abstract
We propose and demonstrate a novel machine learning algorithm that assesses pulmonary edema severity from chest radiographs. While large publicly available datasets of chest radiographs and free-text radiology reports exist, only limited numerical edema severity labels can be extracted from radiology reports. This is a significant challenge in learning such models for image classification. To take advantage of the rich information present in the radiology reports, we develop a neural network model that is trained on both images and free-text to assess pulmonary edema severity from chest radiographs at inference time. Our experimental results suggest that the joint image-text representation learning improves the performance of pulmonary edema assessment compared to a supervised model trained on images only. We also show the use of the text for explaining the image classification by the joint model. To the best of our knowledge, our approach is the first to leverage free-text radiology reports for improving the image model performance in this application. Our code is available at: https://github.com/RayRuizhiLiao/joint_chestxray.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/978-3-030-59713-9_51
