Learning to solve problems in computer vision with synthetic data

Abstract

Deep neural networks (DNN) have become the tool of choice for many researchers due to their superior performance. However, for DNNs to reach their full potential, a large enough dataset must be available. This poses severe limitation over problems that DNN can be applied to. Fortunately, many problems in computer vision have well-understood physical models, and can be simulated readily. This thesis considers the use of synthetic data to allow the use of DNN to solve problems in computer vision. First, we consider using synthetic data for problems where collection of real data is not feasible. We focus on the problem of magnifying small motion in videos. Using synthetic data allows us to train DNN models that magnify motion with reduced artifacts and better noise handling compared to traditional signal-processing based algorithm. Then, we discuss the importance of realism of the generated data. We focus on realistic camera pipeline simulation, and use it to study blind denoising in real images. We show that our noise simulation based on realistic camera pipeline significantly outperforms simplified noise models commonly used in the literature. Finally, we show that synthetic data can also be useful for a more general computer vision research. We use synthetic data to study the effect of label quality to the semantic segmentation task. Synthetic data provides us with large enough datasets that we can study the trade-off between quality and quantity of the data. We find that the accuracy of prediction depends largely on the estimated time required for human to annotate data, and that fine-tuning prediction after training on low-quality labels offers the best trade-off between effort to annotate and the accuracy.