Meta-learning and Enforcing Useful Conservation Laws in Sequential Prediction Problems

Abstract

In recent years, deep learning techniques have enjoyed storied success in a wide array of problem domains, including computer vision, natural language processing, and robotics. While much of this success can be attributed to increasing availability of both data and computing resources, the inductive biases induced by various training methods, model components, and architectures has helped enable efficient generalization as well. Useful biases often exploit symmetries in the prediction problem, such as convolutional neural networks relying on translation equivariance. Automatically discovering such useful symmetries is a promising path to greatly improving the performance of ML systems, but it still remains a challenge. In this work, we focus on sequential prediction problems in real and simulated physical domains and take inspiration from Noether’s theorem to reduce the problem of finding inductive biases to that of meta-learning useful conserved quantities. We propose Noether Networks: a class of models where an unsupervised, meta-learned conservation loss is optimized inside the prediction function. This adapts the model weights to the particular input and imposes the approximate meta-learned conservation law in the predictions. We show, theoretically and experimentally, that Noether Networks improve prediction quality, providing a general framework for discovering inductive biases in sequential prediction problems.