Show simple item record

dc.contributor.authorCastro Fernandez, Raul
dc.contributor.authorMin, Jisoo
dc.contributor.authorNava, Demitri
dc.contributor.authorMadden, Samuel
dc.date.accessioned2022-07-15T15:54:09Z
dc.date.available2022-07-15T15:54:09Z
dc.date.issued2019
dc.identifier.urihttps://hdl.handle.net/1721.1/143769
dc.description.abstract© 2019 IEEE. Data analysts often need to find datasets that are similar (i.e., have high overlap) or that are subsets of one another (i.e., one contains the other). Exactly computing such relationships is expensive because it entails an all-pairs comparison between all values in all datasets, an O(n2) operation. Fortunately, it is possible to obtain approximate solutions much faster, using locality sensitive hashing (LSH). Unfortunately, LSH does not lend itself naturally to compute containment, and only returns results with a similarity beyond a pre-defined threshold; we want to know the specific similarity and containment score. The main contribution of this paper is LAZO, a method to simultaneously estimate both the similarity and containment of datasets, based on a redefinition of Jaccard similarity which takes into account the cardinality of each set. In addition, we show how to use the method to improve the quality of the original JS and JC estimates. Last, we implement LAZO as a new indexing structure that has these additional properties: i) it returns numerical scores to indicate the degree of similarity and containment between each candidate and the query - instead of only returning the candidate set; ii) it permits to query for a specific threshold on-the-fly, as opposed to LSH indexes that need to be configured with a pre-defined threshold a priori; iii) it works in a data-oblivious way, so it can be incrementally maintained. We evaluate LAZO on real-world datasets and show its ability to estimate containment and similarity better and faster than existing methods.en_US
dc.language.isoen
dc.publisherInstitute of Electrical and Electronics Engineers (IEEE)en_US
dc.relation.isversionof10.1109/ICDE.2019.00109en_US
dc.rightsCreative Commons Attribution-Noncommercial-Share Alikeen_US
dc.rights.urihttp://creativecommons.org/licenses/by-nc-sa/4.0/en_US
dc.sourceMIT web domainen_US
dc.titleLazo: A Cardinality-Based Method for Coupled Estimation of Jaccard Similarity and Containmenten_US
dc.typeArticleen_US
dc.identifier.citationCastro Fernandez, Raul, Min, Jisoo, Nava, Demitri and Madden, Samuel. 2019. "Lazo: A Cardinality-Based Method for Coupled Estimation of Jaccard Similarity and Containment." Proceedings - International Conference on Data Engineering, 2019-April.
dc.contributor.departmentMassachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
dc.relation.journalProceedings - International Conference on Data Engineeringen_US
dc.eprint.versionAuthor's final manuscripten_US
dc.type.urihttp://purl.org/eprint/type/ConferencePaperen_US
eprint.statushttp://purl.org/eprint/status/NonPeerRevieweden_US
dc.date.updated2022-07-15T15:27:26Z
dspace.orderedauthorsCastro Fernandez, R; Min, J; Nava, D; Madden, Sen_US
dspace.date.submission2022-07-15T15:27:27Z
mit.journal.volume2019-Aprilen_US
mit.licenseOPEN_ACCESS_POLICY
mit.metadata.statusAuthority Work and Publication Information Neededen_US


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record