Algorithms & Systems for Differentiable Graphics Programming

Abstract

Differentiable graphics representations are now a centerpiece for learning-based approaches in inverse rendering, novel-view synthesis, data & compute efficient rendering, and even 3D generative models. We see great advances in performance & fidelity when mixing classical wisdom from existing primitives like meshes and textures, and novel primitives like tiny neural networks. This cross-pollination of ML & graphics is key to these advances, but is held back due to complications: existing frameworks like PyTorch are ill-suited to graphics programming both due to algorithmic problems, like discontinuities, and system-design problems that lead to poor performance & expressive power. This thesis discusses several original approaches that were developed with generalizability in mind, to allow these approaches to apply broadly to different domains that struggle with the same problems. The first half of this thesis tackles discontinuities in a wide variety of applications, both (i) through a compiler that takes the problem-specific boundary sampling idea and automates it through compiler passes, and (ii) the warped-area reparameterization method that can be used to handle discontinuities by entirely removing the requirement of such problem-specific boundary samplers. We show how this enables light-weight integration with existing renderers by reparameterizing a mesh-based path tracer and a neural SDF renderer to make them fully differentiable. The second half will present the SLANG.D compiler, an industry collaboration that resulted in a high-performance compiler for next-generation differentiable & neural graphics systems. We discuss how the user-centric focus of SLANG.D’s automatic differentiation system enables users to write large-scale differentiable graphics pipelines and re-use 1000s of lines of existing rendering infrastructure without sacrificing its performance.