The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Authors: Yingjin Qian (DataDirect Networks (DDN)); Wen Cheng (Huazhong University of Science and Technology (HUST)); Lingfang Zeng (Zhejiang Lab); Marc-André Vef (Johannes Gutenberg University Mainz); Oleg Drokin and Andreas Dilger (Whamcloud Inc); Shuichi Ihara (DataDirect Networks (DDN)); Wusheng Zhang (Tsinghua University, China); Yang Wang (Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences); and André Brinkmann (Johannes Gutenberg University Mainz)
Abstract: In parallel and distributed file systems, caching can improve data performance and metadata operations. Currently, most distributed file systems adopt a write-back data cache for performance and a write-through metadata cache for simplifying consistency. However, with modern file systems scales and workloads, write-through metadata caching can impact overall file system performance, e.g., through lock contention and heavy RPC loads required for namespace synchronization and transaction serialization.
This paper proposes a novel metadata writeback caching (MetaWBC) mechanism to improve the performance of metadata operations in distributed environments. To achieve extreme metadata performance, we developed a fast, lightweight, and POSIX-compatible memory file system as a metadata cache. Further, we designed a file caching state machine and included other performance optimizations. We coupled MetaWBC with Lustre and evaluated that MetaWBC can outperform the native parallel file system by up to 8x for metadata-intensive benchmarks, and up to 7x for realistic workloads in throughput.
Presentation: file
Back to Technical Papers Archive Listing