Latency-Critical Quantized Inference With Transformer Decoders on ARM and RISC-V CPUs
[EN] Large language models are transforming industries but face challenges due to their high computational and energy demands. Model compression via quantization mitigates these barriers by reducing the bit precision of parameters and arithmetic operations, enabling deployment on resource-constraine...
| Autores: | , , , , |
|---|---|
| Tipo de recurso: | artículo |
| Fecha de publicación: | 2025 |
| País: | España |
| Institución: | Universitat Politècnica de València (UPV) |
| Repositorio: | RiuNet. Repositorio Institucional de la Universitat Politécnica de Valéncia |
| Idioma: | inglés |
| OAI Identifier: | oai:riunet.upv.es:10251/226087 |
| Acceso en línea: | https://riunet.upv.es/handle/10251/226087 |
| Access Level: | acceso abierto |
| Palabra clave: | Decoding Transformers Quantization (signal) Program processors Kernel Optimization Computational modeling Arithmetic Costs Vectors Deep learning (DL) Edge platforms Inference Multicore CPUs Quantization Transformer decoders |
| Sumario: | [EN] Large language models are transforming industries but face challenges due to their high computational and energy demands. Model compression via quantization mitigates these barriers by reducing the bit precision of parameters and arithmetic operations, enabling deployment on resource-constrained devices like smartphones and edge platforms. This article focuses on quantization applied to transformer decoders, which are critical for tasks, such as text generation and conversational artificial intelligence. Unlike encoders, decoders are constrained by memory due to their sequential processing nature and low arithmetic intensity. We propose optimizations targeting inference on low-power CPUs, emphasizing efficient linear layers with quantized data/arithmetic and cache optimization. Using two representative ARM and RISC-V platforms, we present optimized mixed-precision implementations of the matrix multiplication that outperform the instance of that computational kernel in popular libraries, such as basic linear algebra subprograms infrastructure software, XNNPACK and ARMCL. This work thus advances the understanding of the impact of quantization on transformer decoder efficiency, energy consumption and precision in edge environments. |
|---|