Pre-trained Language Models for Clinical Systematic Literature Reviews

Abstract

Although systematic literature reviews play a critical role in clinical-based decision making, manual methods for information extraction can sometimes take prohibitively long. In this work, we first describe the construction of datasets in two distinct clinical domains containing randomized trials and observational studies. We then utilize these two datasets to benchmark the performance of Pretrained Language Model (PLM) based entity and relation extraction models as well as the effect of domain specific pre-training prior to their fine-tuning. Our results show evidence to the effectiveness of pre-training using masked language modeling (MLM), a sentence-level proxy task, on boosting the performance of fine-tuned models on both inter- and intra-sentence level information extraction tasks.