about
Comparing Inferential Strategies of Humans and LLMs in Deductive Reasoning (arxiv.org)
2 points by PaulHoule on Mar 3, 2024 | hide | past | pdf | discuss on HN

In plain words: They studied how large language models solve logic puzzles, sorting their answers into the reasoning strategies psychologists use to describe human thinking. The models showed human-like strategies, with stronger ones using them more, and a right answer did not mean valid reasoning.

Abstract · Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning

Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like $\textit{supposition following}$ or $\textit{chain construction}$. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model's accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.

Philipp Mondorf, Barbara Plank
arXiv:2402.14856 · cs.CL, cs.AI · submitted Feb 20, 2024 · updated Jun 3, 2024
abstract · pdf · html · ACL 2024 main, 31 pages, 19 figures

add comment on HN