Procedural Material Generation with Reinforcement Learning
Author(s)
Li, Beichen
Hu, Yiwei
Guerrero, Paul
Hasan, Milos
Shi, Liang
Deschaintre, Valentin
Matusik, Wojciech
Date Issued
December 19, 2024
Journal
ACM Transactions on Graphics
Publisher
ACM
Citation
Li, Beichen, Hu, Yiwei, Guerrero, Paul, Hasan, Milos, Shi, Liang et al. 2024. "Procedural Material Generation with Reinforcement Learning." ACM Transactions on Graphics, 43 (6).
Version
Final published version
Abstract
Modern 3D content creation heavily relies on procedural assets. In particular, procedural materials are ubiquitous in the industry, but their manipulation remains challenging. Previous work conditionally generates procedural graphs that match a given input image. However, the parameter generation step limits how accurately the generated graph matches the input image, due to a reliance on supervision with scarcely available procedural data. We propose to improve parameter prediction accuracy for image-conditioned procedural material generation by leveraging reinforcement learning (RL) and present the first RL approach for procedural materials. RL circumvents the limited availability of procedural data, the domain gap between real and synthetic materials, and the need for end-to-end differentiable loss functions. Given a target image, we retrieve a procedural material and use an RL-trained transformer model to predict a set of parameters that reconstruct the target image as closely as possible. We show that using RL significantly improves parameter prediction to match a given target image compared to supervised methods on both synthetic and real target images.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3687979
