In plain words: JustFit is a laptop runtime that squeezes the model's short-term memory to four bits and loads only the pieces each step needs, so execution and chat history share limited RAM. It kept 327,680 text positions alive versus the best tested setup's 30,720.
Abstract · JustFit: Just-in-Time State Management for Local LLM Serving
Local agents need memory for model execution and working history. We present JustFit, an MLX runtime that coordinates their overlapping allocations: KVExec executes and checkpoints four-bit KV with bounded workspace, PhaseSwap loads phase-dependent components, and StateTrans preserves history across execution modes. With Qwen3.8-27B MXFP4 on a 24 GiB M4 Pro MacBook, JustFit reaches 327,680 retained positions across two requests, 10.67 times the evaluated baseline's 30,720-position single-request record. One request completes the full 262,144-position native window at median 5.986 tokens/s. Each shape completes 16,384 outputs per request in three fresh processes: B1 uses one cold build and two prefix extensions; B2 uses three ordered prefix extensions (Section 4). Image encoding can proceed while preserving a live 196,608-input text request. A controlled, repetitive 32K+6K workload reaches median 18.284 tokens/s at 15,626 MiB; a separate AIME 2026 evaluation scores 29/30. Coordinating execution and state lifetimes makes longer histories feasible on personal hardware.
Yuhua Chen
arXiv:2609.17475 · cs.AI, cs.PF · submitted Sep 15, 2026 · updated Sep 30, 2026
abstract · pdf · html · 20 pages, 6 figures, 17 tables. Code, Quick Start, and reproduction materials are publicly available