In plain words: They tested multimodal AI by asking questions with no images, to see how much it leans on text alone. The systems still described images confidently and scored high — one ranked first on a chest X-ray question test with no images at all.
Abstract · MIRAGE: The Illusion of Visual Understanding
Multimodal AI systems have achieved remarkable performance across a broad range of real-world tasks, yet the mechanisms underlying visual-language reasoning remain surprisingly poorly understood. We report three findings that challenge prevailing assumptions about how these systems process and integrate visual information. First, Frontier models readily generate detailed image descriptions and elaborate reasoning traces, including pathology-biased clinical findings, for images never provided; we term this phenomenon mirage reasoning. Second, without any image input, models also attain strikingly high scores across general and medical multimodal benchmarks, bringing into question their utility and design. In the most extreme case, our model achieved the top rank on a standard chest X-ray question-answering benchmark without access to any images. Third, when models were explicitly instructed to guess answers without image access, rather than being implicitly prompted to assume images were present, performance declined markedly. Explicit guessing appears to engage a more conservative response regime, in contrast to the mirage regime in which models behave as though images have been provided. These findings expose fundamental vulnerabilities in how visual-language models reason and are evaluated, pointing to an urgent need for private benchmarks that eliminate textual cues enabling non-visual inference, particularly in medical contexts where miscalibrated AI carries the greatest consequence. We introduce B-Clean as a principled solution for fair, vision-grounded evaluation of multimodal AI systems.
Mohammad Asadi, Jack W. O'Sullivan, Fang Cao, Tahoura Nedaee, Kamyar Rajabalifardi, Fei-Fei Li, Ehsan Adeli, Euan Ashley
arXiv:2603.21687 · cs.AI · submitted Mar 23, 2026 · updated Apr 2, 2026
abstract · pdf · html
"We hypothesize that this phenomenon emerges predominantly from a misassumption about how these systems are trained. Modern multimodal models are developed on web-scale corpora and are commonly built on top of pretrained large language models, which makes them extraordinarily strong at language modeling, retrieval of statistical regularities, and reconstruction of likely contexts from sparse cues.[48, 25, 24] During the multimodal training, the models are presented with the image, a textual question, and are expected to reconstruct the correct answer. Lacking access to an entire text corpora, a human would intuitively answer the question based on the image in that setup; but we should not infer that this would be the default approach for an AI model. Incentivized to generate the correct next tokens, models might learn to easily ignore the visual information and rely only on their vast prior knowledge, taking the shortest route to the correct answer.[36, 5, 48]"
The crazy thing is that based just on the text in the questions a model was able to "guess" answers:
"When fine-tuned on the public training set of this dataset with images removed (i.e., trained in mirage-mode), our 3-billion-parameter, text-only super-guesser outperformed all frontier multimodal models, including those exceeding hundreds of billions of parameters, on the held-out test benchmark (Figure 3c). It also surpassed human radiologists by more than 10% on average, relying entirely on hidden textual cues in the questions and the structural patterns of the benchmark. In addition, our super-guesser was able to create reasoning traces comparable to, and in some cases indistinguishable from, those of the ground-truth or those generated by frontier multi-modal AI models."