Birds of a Feather Archive

Mixed Feelings About Mixed Precisions

Authors: Hatem Ltaief (King Abdullah University of Science and Technology (KAUST)), Piotr Luszczek (University of Tennessee)

Abstract: Come and share your experiences with the state-of-the-art of mixed-precision techniques! By wisely trading off accuracy, we can mitigate data movement overheads and increase performance of applications. No free lunch however: these optimizations require support from the software/hardware ecosystem and strong numerical validation. This BoF invites the HPC community at large interested in applying mixed precisions into their workflows. Experts from scientific applications/software libraries/hardware architectures will briefly provide the context on this timely topic, share their own perspectives, engage with the audience via a set of questions, and eventually gather feedback to define a roadmap moving forward.

Long Description: Motivated by the quickly changing hardware landscape, which nowadays is prominently equipped in low-precision arithmetic support, mixing floating-point precision formats has become a mainstream algorithmic optimization. The trade-offs are also important to consider and are not immediately obvious. These optimizations require support from the usual software stack (libraries, middleware, and application codes), strong numerical validation procedures, and ultimately scientific evidence that the applications’ key results survived the intermediate precision loss. The emphasis on alternatives to the IEEE 754 floating-point standard, emerged in recent years due to machine learning’s insatiable need for computational power to train deep models composed of convolutional neural networks or ever growing transformer models based on attention mechanism for natural language processing or human-level models that play open-ended games such as Go, DOTA 2, and Poker or predict structural properties of folded proteins (recall AlphaGo, AlphaZero, or AlphaFold). A successful training session for these models requires in excess of Peta-FLOPS per day and costs millions of dollars due their hardware and energy requirements. To reduce the power draw, drastic reduction in floating-point storage bits is required to reap energy-saving benefits in addition to other tricks-of-the-trade such as sparsification and related approaches based on lottery ticket hypothesis. These were the driving factors towards lower-precision representations and specialized tensor FP units that enable Peta-FLOPS level performance from a single compute node. Unlike machine learning models, scientific applications are much more demanding in terms of accuracy but there are early success stories about exploiting a mix of precisions in HPC. In this BoF, we aim at getting together developers and users alike to provide a forum for exchange of information on established and upcoming techniques in mixed-precision applications, software libraries, and hardware platforms of particular interest to HPC community. To that end, we limited the expert participation to few main thrusts including hardware/vendor perspective, theoretical foundations, and software ecosystem. We gathered an inclusive set of moderators to represent a variety of community outlooks and geographical locales. In our opinion, this will assure rapid and iterative process of exploring the topic of mixed-precision and enhance the interaction with the audience which we plan to engage with questions and prompts in an interactive session that leads to rich experience and encourages follow up discussions.

URL: https://cemse.kaust.edu.sa/hicma/news/mixed-feelings-about-mixed-precisions-bird-feather-sc22


Back to Birds of a Feather Archive Listing