The lottery ticket hypothesis in an adversarial setting

Abstract

Deep neural networks are vulnerable to adversarial examples, inputs which appear natural to humans but are misclassified by deep models with a high degree of confidence. The best known defenses against adversarial examples are network capacity and adversarial training. These defenses are very expensive, greatly increasing storage, computation, and energy costs. The Lottery Ticket Hypothesis ("LTH") may help ameliorate this problem. LTH proposes that deep neural networks contain "matching subnetworks", sparse subnetworks to which the network can be pruned early in training, without losing accuracy. In this thesis, we study whether LTH applies in the setting of adversarial training for image classification networks. We find that sparse matching subnetworks indeed exist, and can reduce model sizes as much as 96% early in training. We also find that the size of an architecture's smallest matching subnetworks is always roughly the same, whether or not adversarial training is used.