Harnessing metadata characteristics for efficient deduplication in distributed storage Systems
Name
755241682-MIT.pdf
Description
Full printable version
Size
220.37 KB
Format
Adobe PDF
Checksum (MD5)
37bf501b49696b7080670b0f7d40fd7b
Author(s)
Goldstein, Matthew (Matthew S.)
Advisor(s)
Robert T. Morris and Jiri Schindler.
Date Issued
2011
Publisher
Massachusetts Institute of Technology
Abstract
As storage capacity requirements grow, storage systems are becoming distributed, and that distribution poses a challenge for space savings processes. In this thesis, I design and implement a mechanism for storing only a single instance of duplicated data within a distributed storage system which selectively performs deduplication across each of the independent computers, known as nodes, used for storage. This involves analyzing the contents of each node for objects with characteristics more likely to have duplicates elsewhere, particularly using duplication within a node as the indicative property - an object duplicated many times in a dataset will likely be duplicated at least once in some node. An inter-node system is responsible for efficiently collecting and distributing the information of these potential duplicates. A test implementation was built and run on several data sets characteristic of common storage workloads where deduplication is important, while distributing across 128 nodes. The efficiency of this implementation was analyzed and compared against the savings when distributed across 128 nodes with deduplication performed only between duplicate objects within each node, without inter-node deduplication. The inter-node deduplication was found to increase the space saved by a factor of four to six times. This represented in one case, a file storage server, only a quarter of potential savings due to the majority of potential savings being in files with only a few duplicates. This left a low probability of locating duplication within a single node. However in another case, a collection of over 100 virtual machine images, nearly all potential duplicates were found due to the high number of duplicates for each object, providing an example of where this inter-node mechanism can be most useful.
Description
Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2011.
This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.
Cataloged from student submitted PDF version of thesis.
Includes bibliographical references (p. 61-62).
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
M.I.T. theses are protected by
copyright. They may be viewed from this source for any purpose, but
reproduction or distribution in any format is prohibited without written
permission. See provided URL for inquiries about permission.
copyright. They may be viewed from this source for any purpose, but
reproduction or distribution in any format is prohibited without written
permission. See provided URL for inquiries about permission.
Persistent DSpace Link
