3D reconstruction of human body via machine learning

Abstract

Three-dimensional (3D) reconstruction and modeling of the human body and garments from images is a central open problem in computer vision, yet remains a challenge using machine learning techniques. We proposed a framework to generate the realistic 3D human from a single RGB image via machine learning. The framework is composed of an end-to-end 3D reconstruction neural net with a skinned multi-person linear model (SMPL) model by the generative adversarial networks (GANs). The 3D facial reconstruction used the morphable facial model by principal component analysis (PCA) and the LS3D-W database. The 3D garments are reconstructed by the multi-garment net (MGN) to generate UV-mapping and remapped into the human model with motion transferred by archive of motion capture as surface shapes (AMASS) dataset. The clothes simulated by the extended position based dynamics (XPBD) algorithm realized fast and realistic modeling.