Representation Learning Through the Lens of Science: Symmetry, Language and Symbolic Inductive Biases

Abstract

In this thesis, we explore representation learning, a key technique in machine learning and artificial intelligence that has led to remarkable advancements in fields such as speech, vision, language perception and generation, as well as solving complex scientific problems like protein folding. Despite its success, the prevailing method of end-to-end supervised learning faces challenges, including the need for large datasets, non-interpretable classifications, and difficulties in transferring representations.

To address these limitations, we adopt a scientific perspective, focusing on machine learning tasks that are particularly affected by these issues, and developing benchmarks inspired by scientific principles. Our approach centers on the identification and development of novel inductive biases (assumptions made by the learning algorithm to improve generalization) based on symmetry, language, and symbolic properties. These inductive biases prove beneficial for both solving scientific problems using machine learning and enhancing representation learning methods.

We term this methodology “Representation Learning through the Lens of Science” and demonstrate its effectiveness in various applications. Finally, we discuss the limitations of our approach and propose directions for future research to further refine and expand upon the concepts introduced in this thesis.