Efficient Uncertainty Quantification of Large Language Models

Abstract

Large Language Models (LLMs) have demonstrated remarkable success in many applications; however, their reliability remains a critical concern, especially in high-stakes domains such as healthcare, finance, and law. Uncertainty Quantification (UQ) is essential for assessing LLM outputs and ensuring trust. However, existing UQ methods for LLMs face challenges: high computational costs, difficulties in handling unstructured outputs, and limited generalizability. This thesis addresses these challenges by proposing a systematic investigation into robust and efficient UQ methodologies tailored for LLMs. Specifically, this work focuses on: (1) analyzing probing methods to determine how hidden layers encode information relevant to uncertainty and accuracy, (2) developing novel UQ metrics that strongly correlate with actual model performance, and (3) designing computationally efficient pipelines to make UQ practical for real-world applications. By bridging these gaps, this research aims to establish UQ as a reliable tool for evaluating and improving the trustworthiness of LLM outputs, facilitating their safe and effective deployment in critical domains.