High-dimensional similarity search and sketching : algorithms and hardness

Abstract

We study two fundamental problems that involve massive high-dimensional datasets: approximate near neighbor search (ANN) and sketching. We obtain a number of new results including: ' An algorithm for the ANN problem over the ℓ₁ and ℓ₂ distances that, for the first time, improves upon the Locality-Sensitive Hashing (LSH) framework. The key new insight is to use random space partitions that depend on the dataset. ' An implementation of the core component of the above algorithm, which is released as FALCONN: a new C++ library for high-dimensional similarity search. ' An efficient algorithm for the ANN problem over any distance that can be expressed as a symmetric norm. ' For norms, we establish the equivalence between the existence of short and accurate sketches and good embeddings into ℓp spaces for 0 < p </- 2. We use this equivalence to show the first sketching lower bound for the Earth Mover's Distance (EMD).