Compression of Structured High-Throughput Sequencing Data
Author(s)
Campagne, Fabien; Dorff, Kevin C.; Chambwe, Nyasha; Robinson, James T.; Mesirov, Jill P.
DownloadCampagne-2013-Compression of Struc.pdf (563.7Kb)
PUBLISHER_CC
Publisher with Creative Commons License
Creative Commons Attribution
Terms of use
Metadata
Show full item recordAbstract
Large biological datasets are being produced at a rapid pace and create substantial storage challenges, particularly in the domain of high-throughput sequencing (HTS). Most approaches currently used to store HTS data are either unable to quickly adapt to the requirements of new sequencing or analysis methods (because they do not support schema evolution), or fail to provide state of the art compression of the datasets. We have devised new approaches to store HTS data that support seamless data schema evolution and compress datasets substantially better than existing approaches. Building on these new approaches, we discuss and demonstrate how a multi-tier data organization can dramatically reduce the storage, computational and network burden of collecting, analyzing, and archiving large sequencing datasets. For instance, we show that spliced RNA-Seq alignments can be stored in less than 4% the size of a BAM file with perfect data fidelity. Compared to the previous compression state of the art, these methods reduce dataset size more than 40% when storing exome, gene expression or DNA methylation datasets. The approaches have been integrated in a comprehensive suite of software tools (http://goby.campagnelab.org) that support common analyses for a range of high-throughput sequencing assays.
Date issued
2013-11Department
Koch Institute for Integrative Cancer Research at MITJournal
PLoS ONE
Publisher
Public Library of Science
Citation
Campagne, Fabien, Kevin C. Dorff, Nyasha Chambwe, James T. Robinson, and Jill P. Mesirov. “Compression of Structured High-Throughput Sequencing Data.” Edited by Frederique Lisacek. PLoS ONE 8, no. 11 (November 18, 2013): e79871.
Version: Final published version
ISSN
1932-6203