Deep MinMax Networks
Name
2306.09253v1.pdf
Description
Accepted version
Size
1.13 MB
Format
Adobe PDF
Checksum (MD5)
a6b57ee12f977b16f1f7667dbd4140ff
Author(s)
Lohmiller, Winfried
Gassert, Philipp
Slotine, Jean-Jacques
Date Issued
December 14, 2021
Publisher
IEEE|2021 60th IEEE Conference on Decision and Control (CDC)
Citation
Lohmiller, Winfried, Gassert, Philipp and Slotine, Jean-Jacques. 2021. "Deep MinMax Networks."
Version
Author's final manuscript
Abstract
While much progress has been achieved over the last decades in neuro-inspired
machine learning, there are still fundamental theoretical problems in gradient-based
learning using combinations of neurons. These problems, such as saddle points and
suboptimal plateaus of the cost function, can lead in theory and practice to failures
of learning. In addition, the discrete step size selection of the gradient is problematic
since too large steps can lead to instability and too small steps slow down the learning.
This paper describes an alternative discrete MinMax learning approach for continuous piece-wise linear functions. Global exponential convergence of the algorithm
is established using Contraction Theory with Inequality Constraints [6], which is extended from the continuous to the discrete case in this paper:
• The parametrization of each linear function piece is, in contrast to deep learning, linear in the proposed MinMax network. This allows a linear regression
stability proof as long as measurements do not transit from one linear region to
its neighbouring linear region.
• The step size of the discrete gradient descent is Lagrangian limited orthogonal
to the edge of two neighbouring linear functions. It will be shown that this Lagrangian step limitation does not decrease the convergence of the unconstrained
system dynamics in contrast to a step size limitation in the direction of the gradient.
We show that the convergence rate of a constrained piece-wise linear function learning is equivalent to the exponential convergence rates of the individual local linear
regions.
Description
2021 60th IEEE Conference on Decision and Control (CDC) December 13-15, 2021. Austin, Texas
MIT Department
Massachusetts Institute of Technology. Department of Mechanical Engineering
Massachusetts Institute of Technology. Nonlinear Systems Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-ShareAlike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/cdc45484.2021.9683271
