2026-09-19T09:27:16Zhttps://dspace.mit.edu/server/oai/request

oai:dspace.mit.edu:1721.1/663082022-01-13T07:54:29Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023

Robert T. Morris and Jiri Schindler. Goldstein, Matthew (Matthew S.) Massachusetts Institute of Technology. Dept. of Electrical Engineering and Computer Science. Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science 2011-10-17T19:49:00Z 2011-10-17T19:49:00Z 2011 2011 http://hdl.handle.net/1721.1/66308 755241682 Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2011. This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections. Cataloged from student submitted PDF version of thesis. Includes bibliographical references (p. 61-62). As storage capacity requirements grow, storage systems are becoming distributed, and that distribution poses a challenge for space savings processes. In this thesis, I design and implement a mechanism for storing only a single instance of duplicated data within a distributed storage system which selectively performs deduplication across each of the independent computers, known as nodes, used for storage. This involves analyzing the contents of each node for objects with characteristics more likely to have duplicates elsewhere, particularly using duplication within a node as the indicative property - an object duplicated many times in a dataset will likely be duplicated at least once in some node. An inter-node system is responsible for efficiently collecting and distributing the information of these potential duplicates. A test implementation was built and run on several data sets characteristic of common storage workloads where deduplication is important, while distributing across 128 nodes. The efficiency of this implementation was analyzed and compared against the savings when distributed across 128 nodes with deduplication performed only between duplicate objects within each node, without inter-node deduplication. The inter-node deduplication was found to increase the space saved by a factor of four to six times. This represented in one case, a file storage server, only a quarter of potential savings due to the majority of potential savings being in files with only a few duplicates. This left a low probability of locating duplication within a single node. However in another case, a collection of over 100 virtual machine images, nearly all potential duplicates were found due to the high number of duplicates for each object, providing an example of where this inter-node mechanism can be most useful. by Matthew Goldstein. M.Eng. 62 p. eng Massachusetts Institute of Technology M.I.T. theses are protected by copyright. They may be viewed from this source for any purpose, but reproduction or distribution in any format is prohibited without written permission. See provided URL for inquiries about permission. http://dspace.mit.edu/handle/1721.1/7582 Electrical Engineering and Computer Science. Harnessing metadata characteristics for efficient deduplication in distributed storage Systems Thesis application/pdf false Publication unknown unknown <Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="757ec9d0-b19f-474a-8c8e-18d7761e4eaa"> <Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type> <Language>eng</Language> <Title>Harnessing metadata characteristics for efficient deduplication in distributed storage Systems</Title> <PublishedIn> <Publication> </Publication> </PublishedIn> <PublicationDate>2011</PublicationDate> <Authors> <Author> <DisplayName>Goldstein, Matthew (Matthew S.)</DisplayName> <Affiliation> <OrgUnit> </OrgUnit> </Affiliation> </Author> </Authors> <Editors> </Editors> <Publishers> <Publisher> <DisplayName>Massachusetts Institute of Technology</DisplayName> <OrgUnit /> </Publisher> </Publishers> <License>http://dspace.mit.edu/handle/1721.1/7582</License> <Keyword>Electrical Engineering and Computer Science.</Keyword> <Abstract>As storage capacity requirements grow, storage systems are becoming distributed, and that distribution poses a challenge for space savings processes. In this thesis, I design and implement a mechanism for storing only a single instance of duplicated data within a distributed storage system which selectively performs deduplication across each of the independent computers, known as nodes, used for storage. This involves analyzing the contents of each node for objects with characteristics more likely to have duplicates elsewhere, particularly using duplication within a node as the indicative property - an object duplicated many times in a dataset will likely be duplicated at least once in some node. An inter-node system is responsible for efficiently collecting and distributing the information of these potential duplicates. A test implementation was built and run on several data sets characteristic of common storage workloads where deduplication is important, while distributing across 128 nodes. The efficiency of this implementation was analyzed and compared against the savings when distributed across 128 nodes with deduplication performed only between duplicate objects within each node, without inter-node deduplication. The inter-node deduplication was found to increase the space saved by a factor of four to six times. This represented in one case, a file storage server, only a quarter of potential savings due to the majority of potential savings being in files with only a few duplicates. This left a low probability of locating duplication within a single node. However in another case, a collection of over 100 virtual machine images, nearly all potential duplicates were found due to the high number of duplicates for each object, providing an example of where this inter-node mechanism can be most useful.</Abstract> <Access xmlns="http://purl.org/coar/access_right" > </Access> </Publication>