Adapting floating-point precision to accelerate deep neural network training
(English) Deep Neural Networks (DNNs) have become ubiquitous in a wide range of application domains. Despite their success, training DNNs is an expensive task which has motivated the use of reduced numerical precision formats to improve performance and reduce power consumption. Emulation techniques...
| Autor: | |
|---|---|
| Formato: | tesis doctoral |
| Fecha de publicación: | 2023 |
| País: | España |
| Recursos: | Universitat Politècnica de Catalunya (UPC) |
| Repositorio: | UPCommons. Portal del coneixement obert de la UPC |
| Idioma: | inglés |
| OAI Identifier: | oai:upcommons.upc.edu:2117/404063 |
| Acesso em linha: | https://hdl.handle.net/2117/404063 https://dx.doi.org/10.5821/dissertation-2117-404063 |
| Access Level: | acceso abierto |
| Palavra-chave: | Àrees temàtiques de la UPC::Informàtica |
| Resumo: | (English) Deep Neural Networks (DNNs) have become ubiquitous in a wide range of application domains. Despite their success, training DNNs is an expensive task which has motivated the use of reduced numerical precision formats to improve performance and reduce power consumption. Emulation techniques are a good fit for understanding the properties of new numerical formats on a particular workload. However, current state-of-the-art techniques cannot perform these tasks quickly and accurately on a wide variety of workloads. The usage of Mixed Precision (MP) arithmetic with floating-point 32-bit (FP32) and 16-bit half-precision aims at improving memory and floating-point operations throughput, allowing faster training of bigger models. This is one of the most used techniques, and has been successfully applied to train DNNs. Despite its advantages in terms of reducing the need for key resources like memory bandwidth or register file size, it has a limited capacity for diminishing further computing costs, as it requires 32-bits to represent its output. On the other hand, full half-precision arithmetic fails to deliver state-of-the-art training accuracy. Several hardware companies are proposing native Brain Float 16-bit (BF16) support for neural network training. Fused Multiply- Add (FMA) functional units constitute a fundamental hardware component to train DNNs. Its silicon area grows quadratically with the mantissa bit count of the computer number format, which has motivated the adoption of the BF16. BF16 features 1 sign, 8 exponent and 7 explicit mantissa bits. Some approaches to train DNNs achieve significant performance benefits by using the BF16 format. However, these approaches must combine BF16 with the standard IEEE 754 FP32 format to achieve state-of-the-art training accuracy, which limits the impact of adopting BF16. To address all of the previous concerns with respect to different numerical formats, specific training techniques, and how to increase the use of reduced precision approaches, this Thesis proposes FASE, a Fast, Accurate, and Seamless Emulator that leverages dynamic binary translation to enable emulation of custom numerical formats. FASE is fast; allowing emulation of large unmodified workloads, accurate; emulating at instruction operand level, and seamless; as it does not require any code modifications and works on any application or DNN framework without any language, compiler or source code access restrictions. We evaluate FASE using a wide variety of DNN frameworks and large-scale workloads. Our evaluation demonstrates that FASE achieves better accuracy than coarser-grain state-of-the-art approaches, and shows that it is able to evaluate the fidelity of multiple numerical formats and extract conclusions on their applicability. To show the advantages of FASE we test it in object classification, natural language processing and generative networks workloads. We use FASE to analyze BF16 usage in the training phase of a 3D Generative Adversarial Network (3DGAN) simulating High Energy Physics detectors. We use FASE to characterize and analyze computer arithmetic to propose a seamless approach to dynamically adapt floating point arithmetic. Our dynamically adaptive methodology enables the use of full half-precision arithmetic for up to 96.4% of the computations when training state-of-the-art neural networks; while delivering comparable accuracy to 32-bit floating point arithmetic. Finally, we propose an approach able to train complex DNNs entirely using the BF16 format. Using FASE we introduce a new class of FMA operators, FMABF16_nm, that entirely rely on BF16 FMA hardware instructions and deliver the same accuracy as FP32. FMABF16_nm operators achieve performance improvements within the 1.28x-1.35x range on ResNet101 with respect to FP32. FMABF16_nm enables training complex DNNs on simple low-end hardware devices without requiring expensive FP32 FMA functional units. |
|---|