GIM: Guidance as Initialization Method

Abstract

This work makes two contributions: the evaluation of early stop guidance for deep Fully Connected Networks (FCNs) and the introduction of guidance as an initialization method (GIM). Network initialization has been a meaningful and challenging topic in the field of machine learning (ML) for a long time. Many initialization methods exist, ranging from data-independent to data-dependent approaches. Initializations allow for a better understanding of model behavior and improvements in model performance. The novel guidance tool enabled us to propose GIM, a new technique that initializes a model by leveraging representational similarity with respect to models of different architectures. A model with an architecture that performs poorly in a specific task can be initialized with guidance from a model with an architecture that performs well in the respective task. We focus on the case of FCNs in the task of image classification and provide experimental results to validate our approach.