The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: 12th International Workshop on Runtime and Operating Systems for Supercomputers (ROSS)
Authors: Ian Foster (Argonne National Laboratory (ANL), University of Chicago)
Abstract: I report on experiences developing and deploying the funcX distributed function as a service (FaaS) platform and in employing this platform to support distributed computing pipelines that link instruments, computers (e.g., for analysis, simulation, AI model training), edge computing (e.g., for analysis), data stores, metadata catalogs, and high-speed networks. Both funcX and the Globus Flows system used to implement these pipelines combine cloud-hosted management, for reliability, with edge-hosted execution, for flexible and scalable execution. I discuss, in particular, the funcX and Globus Flows architectures; the container management strategies used in funcX to execute functions with high performance and efficiency on diverse funcX endpoints; and funcX’s integration with an in-memory data store and Globus for managing data that spans endpoints.