In plain words: A new checker scores an AI's reasoning steps by measuring the overall shape of the path they form, not just which steps connect. That shape-based score predicted reasoning quality much better than connection-only graph measures, and a small set of shape signals stayed reliable.
Abstract · The Shape of Reasoning: Topological Analysis of Reasoning Traces in Large Language Models
Evaluating the quality of reasoning traces from large language models remains understudied, labor-intensive, and unreliable: current practice relies on expert rubrics, manual annotation, and slow pairwise judgments. Automated efforts are dominated by graph-based proxies that quantify structural connectivity but do not clarify what constitutes high-quality reasoning; such abstractions can be overly simplistic for inherently complex processes. We introduce a topological data analysis (TDA)-based evaluation framework that captures the geometry of reasoning traces and enables label-efficient, automated assessment. In our empirical study, topological features yield substantially higher predictive power for assessing reasoning quality than standard graph metrics, suggesting that effective reasoning is better captured by higher-dimensional geometric structures rather than purely relational graphs. We further show that a compact, stable set of topological features reliably indicates trace quality, offering a practical signal for future reinforcement learning algorithms.
Xue Wen Tan, Nathaniel Tan, Galen Lee, Stanley Kok
arXiv:2510.20665 · cs.AI · submitted Oct 23, 2025 · updated May 27, 2026
abstract · pdf · html · Accepted in ICML 2026 Workshop: Epistemic Intelligence in Machine Learning