Optimization of neural network feedback control systems using automatic differentiation

Abstract

Optimal control problems can be challenging to solve, whether using analytic or numerical methods. This thesis examines the application of an adjoint method for optimal feedback control, which combines various algorithmic techniques into an original numerical method. In the method investigated here, a neural network defines the control input in both trajectory and feedback control optimization problems. The weights of the neural network that minimize a cost function are determined by an unconstrained optimization routine. By using automatic differentiation on the code that evaluates the cost function, the gradient of the cost with respect to the weights is obtained for the gradient search phase of the optimization process. Automatic differentiation is more efficient than hand-differentiating code for the user and provides exact gradients, allowing the optimization of the neural network weights to proceed more rapidly. Another benefit of this method comes from its use of neural networks, which are able to represent complex feedback control policies, because they are general nonlinear function approximators. Neural networks also have the potential to be generalizable, meaning that a control policy found using a sufficiently rich training set will often work well for other initial conditions outside of the training set. Finally, the software implementation is modular, which means that the user only needs to adjust a few codes in order to set up the method for a specific problem. The application of the adjoint method to three control problems with known solutions demonstrates the ability of the method to determine neural networks that produce near-optimal trajectories and control policies.