RADICAL-Pilot and Parsl: Executing Heterogeneous Workflows on HPC Platforms
DescriptionWorkflow applications are becoming increasingly important to support scientific discovery. That is leading to a proliferation of workflow management systems and, thus, to a fragmented software ecosystem. Integration among existing workflow tools is a way to improve development efficiency and, ultimately, support the sustainability of the scientific workflow community. We describe our experience with integrating RADICAL-Pilot (RP) and Parsl as a way to enable users to develop and execute workflow applications with heterogeneous tasks on heterogeneous high-performance computing resources. We describe our approach to the integration of the two systems and detail the development of RPEX, a Parsl executor which uses RP as its workload manager. We develop an RP executor that enables executing heterogeneous MPI Python functions on CPUs and GPUs, and we measure the weak and strong scaling of RPEX, RP, and Parsl when providing new capabilities to two paradigmatic use cases: Colmena and Ice Wedge Polygons.
Author/Presenters
Event Type
Workshop
TimeMonday, 14 November 20229am - 9:12am CST
LocationD222
Recorded
Cloud and Distributed Computing
In Situ Processing
Scientific Computing
Workflows
W