The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: 4th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)
Authors: Daniel J. Milroy (Lawrence Livermore National Laboratory), Claudia Misale (IBM TJ Watson Research Center), Giorgis Georgakoudis (Lawrence Livermore National Laboratory), Tonia Elengikal (IBM TJ Watson Research Center), Abhik Sarkar (Lawrence Livermore National Laboratory), Maurizio Drocco (IBM TJ Watson Research Center), Tapasya Patki and Jae-Seung Yeom (Lawrence Livermore National Laboratory), Carlos Eduardo Arango Gutierrez (Red Hat Inc), Dong H. Ahn (NVIDIA Corporation), and Yoonho Park (IBM TJ Watson Research Center)
Abstract: As High Performance Computing (HPC) workflows increase in complexity, their designers seek to enable automation and flexibility offered by cloud technologies. Container orchestration through Kubernetes enables highly desirable capabilities but does not satisfy the performance demands of HPC. Kubernetes tools that automate the lifecycle of Message Passing Interface (MPI)-based applications do not scale, and the Kubernetes scheduler does not provide crucial scheduling capabilities. In this work, we detail our efforts to port CORAL-2 benchmark codes to Kubernetes on IBM Cloud and AWS EKS. We describe contributions to the MPI Operator to achieve 3,000-rank scale, a two-orders-of-magnitude improvement to state of the art. We discuss enhancements to KubeFlux, our scheduler plugin for Kubernetes based on the next-generation, cloud-ready Flux framework. Finally, we compare the placement decisions of KubeFlux with those of the Kubernetes scheduler and demonstrate that KubeFlux allows simulated scientific workflows to achieve up to 3x higher performance.