The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Authors: Sian Jin and Dingwen Tao (Indiana University); Houjun Tang (Lawrence Berkeley National Laboratory (LBNL)); Sheng Di (Argonne National Laboratory (ANL)); Suren Byna and Zarija Lukić (Lawrence Berkeley National Laboratory (LBNL)); and Franck Cappello (Argonne National Laboratory (ANL), University of Illinois)
Abstract: Lossy compression is one of the most efficient solutions to reduce storage overhead and improve I/O performance for HPC applications. However, existing parallel I/O libraries cannot fully utilize lossy compression to accelerate parallel write due to the lack of deep understanding on compression-write performance. To this end, we propose to deeply integrate predictive lossy compression with HDF5 to significantly improve parallel-write performance. Specifically, we propose analytical models to predict the time of compression and parallel write before the actual compression to enable compression-write overlapping. We also introduce an extra space to handle the prediction uncertainty. Moreover, we propose an optimization to reorder the compression tasks to increase the overlapping efficiency. Experiments with up to 4,096 cores show that our solution improves the write performance by up to 4.5x and 2.9x over the non-compression and lossy compression solutions, respectively, with only 1.5% storage overhead (to original data) on two real-world applications.
Presentation: file
Back to Technical Papers Archive Listing