DENKV: Addressing Design Trade-Offs of Key-Value Stores for Scientific Applications
DescriptionHPC facilities have employed flash-based storage tier near to compute nodes to absorb high I/O demand by HPC applications during periodic system-level checkpoints. To accelerate these checkpoints, proxy-based distributed key-value stores (PD-KVS) gained particular attention for their flexibility to support multiple backends and network configurations. PD-KVS rely internally on monolithic KVS, such as RocksDB, to exploit the KV interface and query support. However, PD-KVS are unaware of high redundancy factor in checkpoint data, which can be up to GBs to TBs, and therefore, tend to generate high write and space amplification on these storage layers. We propose DENKV which is deduplication-extended node-local LSM-tree-based KVS. DENKV employs asynchronous partially inline dedup (APID) and aims to maintain the performance characteristics of LSM-tree-based KVS while reducing the write and space amplification problems. We implemented DENKV atop BlobDB and showed that our solution maintains performance while reducing write and space amplification.
Event Type
Workshop
TimeMonday, 14 November 20224:15pm - 4:35pm CST
LocationC148
W
Recorded
Author/Presenters
