Parametric PAINTOVER: Generating Design Models via Image Encoders and Latent Trajectories

Abstract

Design is an iterative process where physical or virtual prototypes are created, rendered, evaluated and modified repeatedly. Sketches and direct manipulations are made on the rendered or fabricated mediums to create and communicate intended changes. Parametric design is a prominent paradigm in design and architecture where hand crafted functions map input parameters to a design space to rapidly generate samples. Direct modifications often lead to novel states outside the design space of a parametric model. Moreover, Parametric models are not cyclic, their input and output spaces are not interchangeable without human intervention. Models must be reconfigured to accommodate out-of-domain changes, preventing parametric design tools from being integrated into early phases of design where changes are commonplace. We propose latent spaces of large pre-trained auto-encoders as shared, design spaces for translating states of design among mediums and dimensions. We implement rendering and image encoding to use images as an interface among the outputs and inputs of the model, enabling users with direct modification via painting over. We use sketches, renderings, and 3d models for sampling latent spaces. We share experiment results acquired through linear interpolation and a custom spline implementation in latent spaces. We present samples from found latent trajectories matching to samples from ground truth parametric design models. We find that trajectories exist in latent spaces that approximate axes in parameter spaces. Using images and 3d models as input and output, we provide a cyclic, software agnostic tool for design generation with parameter approximation capabilities that generalize. We provide findings from experiments and present a software repository for parametric paintover including our sketch augmentation model Inverse Drawings and many-dimensional latent spline implementation L-NURBS.