Attention-based neural networks for clinical prediction modelling on electronic health records
Name
12874_2023_Article_2112.pdf
Size
1.24 MB
Format
Adobe PDF
Checksum (MD5)
1df1447220033a11c247c197c890324a
Author(s)
Fridgeirsson, Egill A.
Sontag, David
Rijnbeek, Peter
Date Issued
December 7, 2023
Publisher
BioMed Central
Citation
BMC Medical Research Methodology. 2023 Dec 07;23(1):285
Version
Final published version
Abstract
Background
Deep learning models have had a lot of success in various fields. However, on structured data they have struggled. Here we apply four state-of-the-art supervised deep learning models using the attention mechanism and compare against logistic regression and XGBoost using discrimination, calibration and clinical utility.
Methods
We develop the models using a general practitioners database. We implement a recurrent neural network, a transformer with and without reverse distillation and a graph neural network. We measure discrimination using the area under the receiver operating characteristic curve (AUC) and the area under the precision recall curve (AUPRC). We assess smooth calibration using restricted cubic splines and clinical utility with decision curve analysis.
Results
Our results show that deep learning approaches can improve discrimination up to 2.5% points AUC and 7.4% points AUPRC. However, on average the baselines are competitive. Most models are similarly calibrated as the baselines except for the graph neural network. The transformer using reverse distillation shows the best performance in clinical utility on two out of three prediction problems over most of the prediction thresholds.
Conclusion
In this study, we evaluated various approaches in supervised learning using neural networks and attention. Here we do a rigorous comparison, not only looking at discrimination but also calibration and clinical utility. There is value in using deep learning models on electronic health record data since it can improve discrimination and clinical utility while providing good calibration. However, good baseline methods are still competitive.
MIT Department
Massachusetts Institute of Technology. Institute for Medical Engineering & Science
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1186/s12874-023-02112-2
