Generative Modeling with Guarantees

Abstract

Language models have become ubiquitous in natural language processing, leveraging large amounts of unlabeled data and fine-tuning for downstream tasks. However, concerns have been raised regarding the accuracy and trustworthiness of the text generated by these models. In parallel, differential privacy has emerged as a framework to protect sensitive information while allowing machine learning algorithms to learn from it. Nevertheless, the trade-off between statistical guarantees and utility poses challenges for many applications. Therefore, this thesis aims to develop techniques that balance guarantees and utility, focusing on improving the reliability of generative models while preserving their flexibility.

First, we propose a framework that enables the generation of text conditionally using hard constraints, allowing users to specify certain elements in advance while leaving others open for the model’s prediction. By facilitating interactive editing and rewriting, this framework provides users with precise control over the generated text.

Next, we introduce conformal prediction methods for generating predictions under soft constraints, ensuring statistical correctness. These methods produce valid confidence sets for text generation while maintaining high empirical precision.

Finally, we explore the balance between privacy and utility in data release by relaxing the notion of guarantees from differential privacy to a definition based on guesswork. We present a learning-based approach to de-identification, addressing the challenges of privacy preservation while still enabling effective data utilization.

The effectiveness of our proposed methods is demonstrated through a range of tasks, including text infilling, radiology report generation, and X-ray classification. These tasks showcase the utility of our techniques in various practical scenarios.