Structure, Function, and Interaction in Protein Language Models

Abstract

In recent years, transformer architectures have shown remarkable capabilities in learning meaningful representations from text and images. This approach has been extended to the realm of protein sequences through pretrained protein language models, which have excelled in various protein engineering tasks. In this thesis, we investigate a pre-trained protein language model’s ability to predict protein structure and the effects of mutations. For many advanced protein understanding tasks, such as predicting protein function and protein-protein interactions, fine-tuning of the model is essential. We explore methods to fine-tune the Evolutionary Scale Modeling (ESM2) model, a pretrained protein language model, for predicting protein functions structured as Gene Ontology terms and predicting protein-protein interactions. Notably, we develop a novel method of modeling the hierarchy constraint in GO term prediction that improves training convergence and test performance while making the model hierarchically consistent with GO. This research aims to enhance our understanding of protein language models in decoding complex biological information, thereby contributing to advancements in computational biology.