In plain words: A visual reasoning test built by throwing out any question today's image-reading AI already gets right, so it stays brutally hard. The best models answered none of it at release and just 6% a year later.
Abstract · ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
Large Multimodal Models (LMMs) exhibit shortfalls when interpreting images and, by some measures, have poorer spatial cognition than young children or animals. Despite this, they attain high scores on many popular visual benchmarks, with headroom rapidly eroded by model progress. This creates a need for difficult benchmarks that remain relevant for longer. We introduce ZeroBench - a lightweight visual reasoning benchmark curated using adversarial filtering to be "impossible" for frontier LMMs at its original release, with initial SotA scores of 0% pass@1 and pass^5. We track progress on ZeroBench over the subsequent year, observing SotA reaching 6% pass^5 and 19% pass@5, indicating the potential longevity of the benchmark. We evaluate 46 LMMs on ZeroBench, compare performance to a human baseline, analyse strengths and weaknesses, chart a year of progress in visual capabilities, and publicly release ZeroBench at https://zerobench.github.io.
Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, Vatsal Raina, Hanyi Xiong, et al.
arXiv:2502.09696 · cs.CV · submitted Feb 13, 2025 · updated Jul 7, 2026
abstract · pdf · html · Accepted at ICML 2026
#4 sticks out as pretty poorly designed because as a human I can't get their answer. Like how many cats, one is just the bowtie, does that count as a cat? I could construe it's there but it's not actually in the picture. how many leaves? if it's hard to tell them apart are they actually "distinct. same with the window panes, based on the answer number they think they're being tricky counting the window in the background but semantically you would probably not want a model to pick up on that and even then I don't think it's actually possible to tell if that's 4 panes (2 double paned sides with inner lattice) or like 12.
Another random sample #64, I see 5 pens 2 of which are "clicky" and one that I can't tell if it's clicky or twisty. 3/5 do not have lids so %60.00 but they got 21.43 which means they counted the markers even though they only asked for PENS, they failed to semantically parse their own question.