In plain words: A custom tool registered through the normal API makes closed models write out the reasoning steps they usually hide. That reasoning matched their hidden reasoning and beat answering with no reasoning, with the strongest model picking the right path early and skipping easy steps.
Abstract · Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models
The rapid capability gains of frontier language models are widely attributed to improved reasoning abilities, yet this cannot be verified as raw CoT traces in closed-source systems are hidden. By registering a simple custom tool through a standard API feature, we induce frontier models to externalize intermediate reasoning. Because these traces may reflect post-hoc rationalization rather than genuine reasoning, we first evaluate against native CoT on open-source models and extend to closed-source frontier models including GPT-6 Astra. We find that the extracted reasoning matches native reasoning performance and substantially outperforms no-reasoning baselines, across competition mathematics, science, and code generation. We then characterize how frontier models structure their intermediate reasoning. Across token efficiency, reasoning-step types, and induced reasoning trees, we identify systematic differences in how models externalize, compress, and organize reasoning. We find that Astra exhibits token-efficient directed reasoning, selecting a correct trajectory earlier, while resolving elementary steps internally and externalizing only crucial reasoning. These findings provide a behavioral lens on frontier-model reasoning beyond benchmark scores.
Xiaoyu Luo, Tao Ren, Wenrui Yu, Xiao Li, Qiongxiu Li, Johannes Bjerva
arXiv:2609.26637 · cs.CL, cs.AI, cs.CR · submitted Sep 22, 2026
abstract · pdf · html · 33 pages,14 figures