2026-09-18T20:55:48Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1626882025-09-19T04:48:51Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Shah, Devavrat
Gan, Emily
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2025-09-18T14:27:24Z
2025-09-18T14:27:24Z
2025-05
2025-06-23T14:02:01.204Z
https://hdl.handle.net/1721.1/162688
We show that a gradient-descent with a simple, universal rule for step-size selection provably finds k-SVD, i.e., the k ≥ 1 largest singular values and corresponding vectors, of any matrix, despite nonconvexity. There has been substantial progress towards this in the past few years where existing results are able to establish such guarantees for the exact-parameterized and over-parameterized settings, with choice of oracle-provided step size. But guarantees for generic setting with a step size selection that does not require oracle-provided information has remained a challenge. We overcome this challenge and establish that gradient descent with an appealingly simple adaptive step size (akin to preconditioning) and random initialization enjoys global linear convergence for generic setting. Our convergence analysis reveals that the gradient method has an attracting region, and within this attracting region, the method behaves like Heron’s method (a.k.a. the Babylonian method). Empirically, we validate the theoretical results. The emergence of a modern compute infrastructure for iterative optimization coupled with this work is likely to provide a means of solving k-SVD for very large matrices.
M.Eng.
Massachusetts Institute of Technology
Attribution 4.0 International (CC BY 4.0)
Copyright retained by author(s)
https://creativecommons.org/licenses/by/4.0/
GDSVD: Scalable k-SVD via Gradient Descent
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="2da02bee-24d6-4b93-bf58-deb5334a7578">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>GDSVD: Scalable k-SVD via Gradient Descent</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2025-05</PublicationDate>
<Authors>
<Author>
<DisplayName>Gan, Emily</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://creativecommons.org/licenses/by/4.0/</License>
<Abstract>We show that a gradient-descent with a simple, universal rule for step-size selection provably finds k-SVD, i.e., the k ≥ 1 largest singular values and corresponding vectors, of any matrix, despite nonconvexity. There has been substantial progress towards this in the past few years where existing results are able to establish such guarantees for the exact-parameterized and over-parameterized settings, with choice of oracle-provided step size. But guarantees for generic setting with a step size selection that does not require oracle-provided information has remained a challenge. We overcome this challenge and establish that gradient descent with an appealingly simple adaptive step size (akin to preconditioning) and random initialization enjoys global linear convergence for generic setting. Our convergence analysis reveals that the gradient method has an attracting region, and within this attracting region, the method behaves like Heron’s method (a.k.a. the Babylonian method). Empirically, we validate the theoretical results. The emergence of a modern compute infrastructure for iterative optimization coupled with this work is likely to provide a means of solving k-SVD for very large matrices.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>