about
Power Aware Dynamic Reallocation for Inference (arxiv.org)
3 points by PaulHoule 243 days ago | hide | past | pdf | discuss on HN

In plain words: AI systems split generation into two stages on separate chips, but power is the limit. This system shifts power and chips between the stages to stay under a cap. At peak load it met response-time targets up to twice as often as a fixed setup.

Abstract · Power Aware Dynamic Reallocation For Inference

Disaggregation has emerged as a powerful strategy for optimizing large language model (LLM) inference by separating compute-intensive prefill and memory-bound decode phases across specialized GPUs. This separation improves utilization and throughput under fixed hardware capacity. However, as model and cluster scales grow, power, rather than compute, has become the dominant limiter of overall performance and cost efficiency. In this paper, we propose RAPID, a power-aware disaggregated inference framework that jointly manages GPU roles and power budgets to sustain goodput within strict power caps. RAPID utilizes static and dynamic power reallocation in addition to GPU reallocation to improve performance under fixed power bounds. RAPID improves overall performance and application consistency beyond what is achievable in current disaggregation solutions, resulting in up to a 2x improvement in SLO attainment at peak load when compared to a static assignment without an increase in complexity or cost.

Yiwei Jiang, Sangeeta Chowdhary, Nathaniel Morris, Rutwik Jain, Srilatha Manne, Sam Bayliss
arXiv:2601.12241 · cs.DC · submitted Jan 18, 2026
abstract · pdf · html

add comment on HN