In plain words: They tested three tricks for speeding AI kernels on small chips: wide chunks, work split across cores, and loading next data while computing. Wide chunks won on memory-heavy kernels, cores paid off on big jobs, and pre-loading helped when data moves and math overlap.
Abstract · Analyzing Latency Hiding and Parallelism in an MLIR-based AI Kernel Compiler
AI kernel compilation for edge devices depends on the compiler's ability to exploit parallelism and hide memory latency in the presence of hierarchical memory and explicit data movement. This paper reports a benchmark methodology and corresponding results for three compiler-controlled mechanisms in an MLIR-based compilation pipeline: vectorization (Vec), multi-threading (MT) across hardware contexts, and double buffering (DB) using ping--pong scratchpad buffers to overlap DMA transfers with compute. Using Triton/Inductor-generated kernels, we present an ablation ladder that separates the contribution of Vec, MT, and DB, and we quantify how MT speedup scales with problem size using GELU as a representative activation kernel. The results show that vectorization provides the primary gain for bandwidth-sensitive kernels, MT delivers substantial improvements once scheduling overhead is amortized, and DB provides additional benefit when transfers and compute can be overlapped (i.e., outside the extremes of purely memory-bound or purely compute-bound behavior).
Javed Absar, Samarth Narang, Muthu Baskaran
arXiv:2602.20204 · cs.PL, cs.AI · submitted Feb 22, 2026
abstract · pdf · html · Accepted at MLBench workshop as part of ASPLOS'26