about
VisPhyWorld: Probing Physical Reasoning via Code-Driven Video Reconstruction (arxiv.org)
1 point by PaulHoule 222 days ago | hide | past | pdf | discuss on HN

In plain words: Instead of answering questions about a video, models must write runnable simulator code that recreates it, so their guesses can be tested. The pipeline produced valid videos in 97.7% of runs, yet models matched the appearance but botched physical parameters and motion.

Abstract

Evaluating whether Multimodal Large Language Models (MLLMs) genuinely reason about physical dynamics remains challenging. Most existing benchmarks rely on recognition-style protocols such as Visual Question Answering (VQA) and Violation of Expectation (VoE), which can often be answered without committing to an explicit, testable physical hypothesis. We propose VisPhyWorld, an execution-based framework that evaluates physical reasoning by requiring models to generate executable simulator code from visual observations. By producing runnable code, the inferred world representation is directly inspectable, editable, and falsifiable. This separates physical reasoning from rendering. Building on this framework, we introduce VisPhyBench, comprising 209 evaluation scenes derived from 108 physical templates and a systematic protocol that evaluates how well models reconstruct appearance and reproduce physically plausible motion. Our pipeline produces valid reconstructed videos in 97.7% of benchmark runs before fallback. Experiments show that while state-of-the-art MLLMs achieve strong semantic scene understanding, they struggle to accurately infer physical parameters and to simulate consistent physical dynamics. Our code is available https://github.com/TIGER-AI-Lab/VisPhyWorld

Jiarong Liang, Max Ku, Ka-Hei Hui, Ping Nie, Wenhu Chen
arXiv:2602.13294 · cs.CV, cs.AI · submitted Feb 9, 2026 · updated May 21, 2026
abstract · pdf · html

add comment on HN