An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models
Name
applsci-13-09735-v3.pdf
Size
4.41 MB
Format
Adobe PDF
Checksum (MD5)
caf6d8c081b8458dfcff2e2dd247e968
Author(s)
Al-Hadhrami, Suheer
Menai, Mohamed El Bachir
Al-Ahmadi, Saad
Alnafessah, Ahmad
Date Issued
August 28, 2023
Journal
Applied Sciences
Publisher
MDPI
Citation
Al-Hadhrami, S.; Menai, M.E.B.; Al-Ahmadi, S.; Alnafessah, A. An Effective Med-VQA Method Using a Transformer with Weights Fusion of Multiple Fine-Tuned Models. Appl. Sci. 2023, 13.
Version
Final published version
Abstract
Visual question answering (VQA) is a task that generates or predicts an answer to a question in human language about visual images. VQA is an active field combining two AI branches: NLP and computer vision. VQA in the medical field is still at an early stage, and it needs vast efforts and exploration to reach practical usage. This paper proposes two models that are utilized in the latest vision and NLP transformers that outperform the SOTA and have not yet been utilized in medical VQA. The ELECTRA-base transformer is used for textual feature extraction, whereas SWIN is used for visual feature extraction. In the SOTA medical VQA, selecting the model is based on the model that achieves the highest validation accuracy or the last model in training. The first proposed model, the best-value-based model, is selected based on the highest validation accuracy. The second model, the greedy-soup-based model, uses a greedy soup technique based on the fusion of multiple fine-tuned models to set the model parameters. The greedy soup selects the model parameters by fusing the model parameters that have significant performance on the validation accuracy in training. The greedy-soup-based model outperforms the best-value-based model, and both proposed models outperform the SOTA, which has an accuracy of 83.49%. The greedy-soup-based model is optimized with batch size and learning rate. During the optimization, seven extra models exceed the SOTA accuracy. The best model trained with a learning rate of 1.0×10−4 and batch size 16 achieves an accuracy of 87.41%.
Subjects
Fluid Flow and Transfer Processes
Computer Science Applications
Process Chemistry and Technology
General Engineering
Instrumentation
General Materials Science
MIT Department
Massachusetts Institute of Technology. Sociotechnical Systems Research Center
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.3390/app13179735
