about
Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decoding (arxiv.org)
4 points by chrsw 127 days ago | hide | past | pdf | discuss on HN

In plain words: It speeds up reasoning on small devices by trimming the model's numbers to key parts to guess the next word, then checking guesses with the full model, keeping answers exact. Without retraining, it ran up to 2.41 times faster than the usual full-precision run.

Abstract

Speculative decoding has emerged as a promising lossless approach for accelerating Large Language Models (LLMs). As reasoning LLMs increasingly suffer from decode-stage overhead and approximation-based methods degrade accuracy, lossless speculative decoding has become essential for efficient inference. However, existing methods still struggle to deliver strong low-batch performance without additional training, limiting practical deployment on consumer devices. To address this challenge, we propose Cassandra, an algorithm-hardware co-designed self-speculative decoding framework optimized for low-batch scenarios. Cassandra constructs a high-performance, training-free draft model through fine-grained data selection. Using optimized pruning and mantissa truncation, it identifies the most salient values in both model weights and the Key-Value (KV) cache, enabling rapid candidate token generation before full-precision parallel verification. Unlike prior self-speculative decoding methods based on layer skipping or structured KV compression, Cassandra achieves significantly higher efficiency. To further reduce the overhead of format conversion between Cassandra representations and standard floating-point formats, we also introduce a lightweight encoder-decoder hardware module designed for seamless integration with commercial GPUs and NPUs. Experimental results show that Cassandra achieves up to 2.41x speedup over the BF16 baseline without additional training. Furthermore, on Llama 3 8B running on an NVIDIA GeForce RTX 4090, Cassandra generates 1.81x more tokens under the same memory budget compared to Eagle-3, a state-of-the-art speculative decoding method.

Soongyu Choi, Yuntae Kim, Muyoung Son, Joo-Young Kim
arXiv:2605.26558 · cs.AR · submitted May 26, 2026
abstract · pdf · html · To appear at ISCA 2026

add comment on HN