In plain words: They tested Apple M4's matrix instructions to learn their speed and data-transfer limits, then used that to write small matrix-multiplication routines on the fly. These beat the chip maker's optimized library in almost every size tried, reaching over 2.3 trillion 32-bit operations per second.
Abstract · Hello SME! Generating Fast Matrix Multiplication Kernels Using the Scalable Matrix Extension
Modern central processing units (CPUs) feature single-instruction, multiple-data pipelines to accelerate compute-intensive floating-point and fixed-point workloads. Traditionally, these pipelines and corresponding instruction set architectures (ISAs) were designed for vector parallelism. In recent years, major hardware vendors have further increased the throughput of their CPUs by introducing matrix units with corresponding ISA extensions. The Scalable Matrix Extension (SME) has been announced for the Arm architecture in 2021 and Apple's M4 chip is the first to support SME. This paper presents an in-depth study of SME on M4. Our microbenchmarks determine the maximum floating-point and fixed-point throughput of M4's SME acceleration and study the achievable bandwidth for transfers to and from the matrix registers. Furthermore, we used the insights gained to design a just-in-time code generator for SME-based small matrix multiplications. The results presented show that M4's SME support is FP32-centric, with an achievable throughput of over 2.3 FP32 TFLOPS. To maximize read and write bandwidth, loading and storing to and from the matrix registers must be done in two steps. Our just-in-time generated small matrix multiplication kernels outperform the vendor-optimized BLAS implementation in almost all tested configurations.
Stefan Remke, Alexander Breuer
arXiv:2409.18779 · cs.DC · submitted Sep 27, 2024
abstract · pdf · html