In plain words: It runs chatbot-style AI on Intel graphics chips by merging memory-shuffling steps and splitting saved conversation memory into two parts, so more requests fit at once. Versus the standard open-source code, it cut per-token delay up to 7x and raised throughput 27x.
Abstract
Transformer based Large Language Models (LLMs) have been widely used in many fields, and the efficiency of LLM inference becomes hot topic in real applications. However, LLMs are usually complicatedly designed in model structure with massive operations and perform inference in the auto-regressive mode, making it a challenging task to design a system with high efficiency. In this paper, we propose an efficient LLM inference solution with low latency and high throughput. Firstly, we simplify the LLM decoder layer by fusing data movement and element-wise operations to reduce the memory access frequency and lower system latency. We also propose a segment KV cache policy to keep key/value of the request and response tokens in separate physical memory for effective device memory management, helping enlarge the runtime batch size and improve system throughput. A customized Scaled-Dot-Product-Attention kernel is designed to match our fusion policy based on the segment KV cache solution. We implement our LLM inference solution on Intel GPU and publish it publicly. Compared with the standard HuggingFace implementation, the proposed solution achieves up to 7x lower token latency and 27x higher throughput for some popular LLMs on Intel GPU.
Hui Wu, Yi Gan, Feng Yuan, Jing Ma, Wei Zhu, Yutao Xu, Hong Zhu, Yuhua Zhu, Xiaoli Liu, Jinghui Gu, Peng Zhao
arXiv:2401.05391 · cs.AR, cs.AI · submitted Dec 19, 2023 · updated Jun 23, 2024
abstract · pdf
Is this paper the same work as goes in to Intel's BigDL-LLM code? That's been out for a few months now but I haven't seen it in use yet. https://medium.com/intel-tech/bigdl-llm-easily-optimize-your...