Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. MIT Open Access Articles
  3. MIT Open Access Articles
  4. Data-dependent compression of random features for large-scale kernel approximation

Data-dependent compression of random features for large-scale kernel approximation

Thumbnail Image
Download
Name

1810.04249.pdf

Description
Submitted version
Size

1.01 MB

Format

Adobe PDF

Checksum (MD5)

26a130c1bd13c8da31f7f4c40cc66f19

sword-2020-12-03T18:42:04.original.xml (130 B)
Original SWORD entry document
Author(s)
Agrawal, Raj
•
Campbell, Trevor David
•
Huggins, Jonathan H.
•
Broderick, Tamara A
Date Issued
February 2019
Journal
Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS)
Citation
Agrawal, Raj et al. “Data-dependent compression of random features for large-scale kernel approximation.” Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), 89, (April 2019) © 2019 The Author(s)
Version
Original manuscript
Abstract
Kernel methods offer the flexibility to learn complex relationships in modern, large data sets while enjoying strong theoretical guarantees on quality. Unfortunately, these methods typically require cubic running time in the data set size, a prohibitive cost in the large-data setting. Random feature maps (RFMs) and the Nyström method both consider low-rank approximations to the kernel matrix as a potential solution. But, in order to achieve desirable theoretical guarantees, the former may require a prohibitively large number of features J+, and the latter may be prohibitively expensive for high-dimensional problems. We propose to combine the simplicity and generality of RFMs with a data-dependent feature selection scheme to achieve desirable theoretical approximation properties of Nyström with just O(log J+) features. Our key insight is to begin with a large set of random features, then reduce them to a small number of weighted features in a data-dependent, computationally efficient way, while preserving the statistical guarantees of using the original large set of features. We demonstrate the efficacy of our method with theory and experiments-including on a data set with over 50 million observations. In particular, we show that our method achieves small kernel matrix approximation error and better test set accuracy with provably fewer random features than state-of-the-art methods.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
http://creativecommons.org/licenses/by-nc-sa/4.0/
Persistent DSpace Link
https://hdl.handle.net/1721.1/128772
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.