Birds of a Feather Archive

Workflows Community: An Update on the Community Roadmap for HPC and AI Scientific Workflows Research and Development

Authors: Rafael Ferreira da Silva (Oak Ridge National Laboratory (ORNL)), Rosa Badia (Barcelona Supercomputing Center (BSC)), Kyle Chard (University of Chicago), Olivier Terzo (LINKS Foundation Inc)

Abstract: The interplay of workflow technologies and HPC has been challenged by the fast rise of AI and ML technologies. Workflows empowered with ML techniques largely differ from traditional workflows running on HPC machines. In this BoF, we will bring together researchers from the workflows (https://workflows.community), HPC, and AI/ML communities that work on scientific research questions that require large-scale, distributed, and AI-heavy computing. The session will present an update on challenges, opportunities, new research directions, and future pathways, and will seek input for updating a community roadmap on HPC and AI workflows research and development.

Long Description: Modern scientific data analysis builds on three pillars: (i) Workflow technologies to express and steer the analysis, (ii) machine learning (ML) and artificial intelligence (AI) as fundamental components of the analysis, and (iii) HPC for efficient execution of analytic pipelines over large data sets. These three pillars are researched by communities which traditionally have been separated from each other. However, coping with the current and upcoming large-scale scientific challenges that are interdisciplinary in essence, for instance, Earth Science, Biology, or Energy Science, require, mutual close interactions of these fields for the benefit of the society: ML needs to be integrated within workflows to ease their development and portability; scientific workflows must benefit from ML to scale and increase efficiency of executions; HPC must embrace scientific workflows to democratize access to its resources, as workflow systems should better exploit the power of HPC systems; ML must adapt to HCP architectures to scale to real-world large settings, and HCP must adjust to ML requirements to provide the required computational resources to scale up to extremely large data sets.

In this first edition of this BoF session, we will bring together members of the Workflows Community Initiative (https://workflows.community) to provide an update on challenges, opportunities, new research directions, and future pathways on HPC and AI workflows. Furthermore, to adequately reflect the perspective of the users of workflow systems (i.e., scientists), we will also invite users to provide a brief overview of their science domain challenges developing HPC and AI workflows. Topics of discussion will include, but are not limited to: - AI/ML Workflows and HPC: - Optimization and scheduling of AI-heavy workflows - Workflows integrating HPC and ML frameworks and Big Data - HPC approaches to ML training and application - Use cases for AI heavy workflows in large-scale scientific data analysis - Provenance management and long-term storage - Formal approaches to workflow languages for AI tasks - HPC Workflows and AI: - Usage of AI in task scheduling on HPC systems - Architectures for workflow systems designed as learning systems - Seamless integration of GPUs in HPC management - Workload forecasting for data centers - HPC management in the age of data-parallel and AI heavy workloads - Easing the life-cycle of workflows using ML technologies

This BoF will produce tangible outputs both in terms of a package of materials (lightning talks, live-poll questions and responses) and a draft of an updated community roadmap (based on https://doi.org/10.1109/WORKS54523.2021.00016). During live-polling, we will encourage participants to join the Workflows Community Initiative to facilitate networking after the conference and potentially new international collaborations.

Supercomputing is an ideal venue for this BoF as it brings major HPC resource providers together with scientific computing users, providing an opportunity to discuss existing workflow capabilities supported by several international HPC centers, stimulate conversation on implementation and adoption, and steer future efforts.

URL: https://workflows.community/bof/sc22


Back to Birds of a Feather Archive Listing