about
LLMs can answer multiple choice questions by only seeing the answers (arxiv.org)
3 points by rbanffy 138 days ago | hide | past | pdf | 1 comment on HN

In plain words: Reasoning AIs were given multiple-choice questions with only the answer choices, and their step-by-step thinking was checked for cheap tricks. Reasoning helped even without the question about half the time, and the traces showed they were inferring the missing question, not cheating.

Abstract · Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers

Large language models (LLMs) now give reasoning before answering, excelling in tasks like multiple-choice question answering (MCQA). Yet, a concern is that LLMs do not solve MCQs as intended, as work finds LLMs sans reasoning succeed in MCQA without using the question, i.e., choices-only. Such partial-input success is often linked to trivial shortcuts, but reasoning traces could reveal if choices-only strategies are truly shallow. To examine these strategies, we have reasoning LLMs solve MCQs in full and choices-only inputs; test-time reasoning often boosts accuracy in full and in choices-only, half the time. While possibly due to shallow shortcuts, choices-only success is barely affected by the length of reasoning traces, and after finding traces pass faithfulness tests, we show they use less problematic strategies like inferring missing questions. In all, we challenge claims that partial-input success is always a flaw, so we propose how reasoning traces could separate problematic data from less problematic reasoning.

Nishant Balepur, Atrey Desai, Rachel Rudinger
arXiv:2510.07761 · cs.CL · submitted Oct 9, 2025 · updated Apr 20, 2026
abstract · pdf · html · ACL 2026

add comment on HN

We have the inverse problem making the options for math test. One usual method is to put the correct answers and perturbations, something like

  a) 4    b) 8   c) 16   d) -8
So we tried to use something evil like

  a) 4    b) 8   c) -4   d) -8
---

In some courses I took, there was 1 long positive answer. Like "What is ecology?"

a) An evil trick of communist

b) A disaster for jobs

c) An invention of lunatics

d) The best thing that even happen for animal, plants and humans, that will protect us forever and make us happy.