Dask-Enabled External Tasks for In Transit Analytics
DescriptionIn situ models represent a relevant alternative to classical post hoc workflows as they allow bypassing disk accesses, thus reducing the IO bottleneck. However, as most in situ data analytics tools are based on MPI, they are complicated to use, especially to parallelize irregular algorithms. Deisa, a task-based in situ analytics tool, couples MPI with Dask, providing a higher level and easier way to write in situ analytics. In this work, we improve Deisa's design by introducing three main concepts: deisa virtual arrays, contracts, and external tasks in Dask distributed. Those refinements reduce the load in the centralized scheduler of Dask and integrate selected simulation data in Dask task graphs transparently, improving Deisa's performance and productivity.
Event Type
Workshop
TimeMonday, 14 November 20224:10pm - 4:15pm CST
LocationC148
Recorded
W