Robust machine learning models and their applications

Abstract

Recent studies have demonstrated that machine learning models are vulnerable to adversarial perturbations - a small and human-imperceptible input perturbation can easily change the model output completely. This has created serious security threats to many real applications, so it becomes important to formally verify the robustness of machine learning models. This thesis studies the robustness of deep neural networks as well as tree-based models, and considers the applications of robust machine learning models in deep reinforcement learning. We first develop a novel algorithm to learn robust trees. Our method aims to optimize the performance under the worst case perturbation of input features, which leads to a max-min saddle point problem when splitting nodes in trees.

We propose efficient tree building algorithms by approximating the inner minimizer in this saddle point problem, and present efficient implementations for classical information gain based trees as well as state-of-the-art tree boosting models such as XGBoost. Experiments show that our method improve the model robustness significantly. We also propose an efficient method to verify the robustness of tree ensembles. We cast the tree ensembles verification problem as a max-clique problem on a multipartite graph. We develop an efficient multi-level verification algorithm that can give tight lower bounds on robustness of decision tree ensembles, while allowing iterative improvement and termination at any-time.

On random forest or gradient boosted decision trees models trained on various datasets, our algorithm is up to hundreds of times faster than the previous approach that requires solving a mixed integer linear programming, and is able to give tight robustness verification bounds on large ensembles with hundreds of deep trees. For neural networks, we contribute a number of empirical studies on the practicality and the hardness of adversarial training. We show that even with adversarial defense, a model's robustness on a test example has a strong correlation with the distance between that example and the manifold of training data embedded by the network. Test examples that are relatively far away from this manifold are more likely to be vulnerable to adversarial attacks.

Consequentially, we demonstrate that an adversarial training based defense is vulnerable to a new class of attacks, the "blind-spot attack," where the input examples reside in low density regions ("blind-spots") of the empirical distribution of training data but are still on the valid ground-truth data manifold. Finally, we apply neural network robust training methods to deep reinforcement learning (DRL) to train agents that are robust against perturbations on state observations. We propose the state-adversarial Markov decision process (SA-MDP) to study the fundamental properties of this problem, and propose a theoretically principled regularization which can be applied to different DRL algorithms, including deep Q networks (DQN) and proximal policy optimization (PPO). We significantly improve the robustness of agents under strong white box adversarial attacks, including new attacks of our own.