A unified framework for consistency of regularized loss minimizers
Author(s)Honorio, Jean; Jaakkola, Tommi S.
MetadataShow full item record
We characterize a family of regularized loss minimization problems that satisfy three properties: scaled uniform convergence, super-norm regularization, and norm-loss monotonicity. We show several theoretical guarantees within this framework, including loss consistency, norm consistency, sparsistency (i.e. support recovery) as well as sign consistency. A number of regularization problems can be shown to fall within our framework and we provide several examples. Our results can be seen as a concise summary of existing guarantees but we also extend them to new settings. Our formulation enables us to assume very little about the hypothesis class, data distribution, the loss, or the regularization. In particular, many of our results do not require a bounded hypothesis class, or identically distributed samples. Similarly, we do not assume boundedness, convexity or smoothness of the loss nor the regularizer. We only assume approximate optimality of the empirical minimizer. In terms of recovery, in contrast to existing results, our sparsistency and sign consistency results do not require knowledge of the sub-differential of the objective function.
DepartmentMassachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Journal of Machine Learning Research: Workshop and Conference Proceedings
Association for Computing Machinery (ACM)
Honorio, Jean, and Tommi Jaakkola. "A unified framework for consistency of regularized loss minimizers." Journal of Machine Learning Research: Workshop and Conference Proceedings, Proceedings of The 31st International Conference on Machine Learning, Volume 32 (2014), 136-144.
Final published version