Traversing Rugged Domains: Explorations in Non-convex Optimization Theory and Software

Abstract

This thesis introduces theoretical and computational frameworks for nonlinear, nonconvex optimization problems in statistics, machine learning, and optimal control. Disciplined Geodesically Convex Programming (DGCP) extends convexity verification to Riemannian manifolds, enabling optimization on curved spaces with global optimality guarantees. We develop rules and atoms for Cartan-Hadamard manifolds, particularly symmetric positive definite matrices, transforming non-convex problems into tractable ones through Riemannian geometry. We also present Optimization.jl, a unified interface for diverse optimization methods that supports specialized implementations for specific problem classes. Its modular architecture integrates automatic differentiation with an extensible plugin system. The framework’s capabilities are demonstrated through a GPU-accelerated hybrid method combining Particle Swarm Optimization with L-BFGS, and an augmented Lagrangian approach with stochastic inner optimizers that connects constrained optimization with machine learning techniques. Our work combines theoretical foundations with practical implementation, providing researchers tools to use advanced optimization methods without specialized mathematical knowledge.