GDSVD: Scalable k-SVD via Gradient Descent
Name
gan-emilygan-meng-eecs-2025-thesis.pdf
Description
Thesis PDF
Size
6.11 MB
Format
Adobe PDF
Checksum (MD5)
2430717f5437a4337ebced769e1edc53
Author(s)
Gan, Emily
Advisor(s)
Shah, Devavrat
Date Issued
May 2025
Publisher
Massachusetts Institute of Technology
Abstract
We show that a gradient-descent with a simple, universal rule for step-size selection provably finds k-SVD, i.e., the k ≥ 1 largest singular values and corresponding vectors, of any matrix, despite nonconvexity. There has been substantial progress towards this in the past few years where existing results are able to establish such guarantees for the exact-parameterized and over-parameterized settings, with choice of oracle-provided step size. But guarantees for generic setting with a step size selection that does not require oracle-provided information has remained a challenge. We overcome this challenge and establish that gradient descent with an appealingly simple adaptive step size (akin to preconditioning) and random initialization enjoys global linear convergence for generic setting. Our convergence analysis reveals that the gradient method has an attracting region, and within this attracting region, the method behaves like Heron’s method (a.k.a. the Babylonian method). Empirically, we validate the theoretical results. The emergence of a modern compute infrastructure for iterative optimization coupled with this work is likely to provide a means of solving k-SVD for very large matrices.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Attribution 4.0 International (CC BY 4.0)
Copyright retained by author(s)
Persistent DSpace Link
