Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler Augmentation
DescriptionDerivatives are key to numerous science, engineering, and machine learning applications. While existing tools generate derivatives of programs in a single language, modern parallel applications combine a set of frameworks and languages to leverage available performance and function in an evolving hardware landscape.
We propose a scheme for differentiating arbitrary DAG-based parallelism that preserves scalability and efficiency, implemented into the LLVM-based Enzyme automatic differentiation framework. By integrating with a full-fledged compiler backend, Enzyme can differentiate numerous parallel frameworks and directly control code generation. This flexibility permits Enzyme to leverage parallel and differentiation-specific optimizations far beyond existing tools.
We differentiate nine distinct versions of the LULESH and miniBUDE applications, written in different programming languages (C++, Julia) and parallel frameworks (OpenMP, MPI, RAJA, Julia tasks, MPI.jl), demonstrating similar scalability to the original program and with a differentiation overhead of 1.0-11.7x on 64 threads/nodes.
We propose a scheme for differentiating arbitrary DAG-based parallelism that preserves scalability and efficiency, implemented into the LLVM-based Enzyme automatic differentiation framework. By integrating with a full-fledged compiler backend, Enzyme can differentiate numerous parallel frameworks and directly control code generation. This flexibility permits Enzyme to leverage parallel and differentiation-specific optimizations far beyond existing tools.
We differentiate nine distinct versions of the LULESH and miniBUDE applications, written in different programming languages (C++, Julia) and parallel frameworks (OpenMP, MPI, RAJA, Julia tasks, MPI.jl), demonstrating similar scalability to the original program and with a differentiation overhead of 1.0-11.7x on 64 threads/nodes.
Event Type
Paper
TimeWednesday, 16 November 20224:30pm - 5pm CST
LocationC141-143-149
TP
System Software
Best Paper Finalist
Best Student Paper Finalists
Recorded
Archive
view
Authors
