The S-Lab team at Nanyang Technological University has introduced Apple-π, the first benchmark for evaluating video models against physical laws, designed to determine whether video generation models truly understand the physical world rather than merely producing visually plausible results.

Article author and source: Academic Headline (ID: SciTouTiao)

In recent years, world models based on video generation models have become one of the most prominent directions in artificial intelligence. Video generation models such as Sora, Genie, Cosmos, and V-JEPA all point to a common hypothesis: if they are exposed to enough video data, they may gradually develop a "physical intuition" and come to understand the physical world.

However, the problem is that existing benchmarks make it difficult to verify whether this assumption holds true.

Do video generation models truly follow the laws of physics, or do they merely produce results that appear correct? Can they think in terms of physical laws, like Newton, rather than relying solely on "intuition," like Aristotle?

To address these issues, the S-Lab team at Nanyang Technological University and its collaborators introduced Apple-π, the first physics-law-based evaluation benchmark for video models. The results show that current video generation models are still far from being reliable general-purpose world simulators, with even the state-of-the-art models scoring only 0.473.

Paper link: https://arxiv.org/abs/2607.16401

The research team stated that even if existing models can generate plausible motion based on annotated first frames, it does not mean they truly understand object relationships, have learned physical laws, or can consistently generate results according to coherent motion principles.

This work aims to help future video generation models truly learn to understand and simulate the world according to the laws of physics, but the "ChatGPT moment" for world models remains far off; to achieve this, researchers still need to equip video world models with stronger scene understanding, richer physical reasoning data, and post-training specifically designed for reasoning.

Apple-π, a benchmark anchored in physical laws

Apple-π is the first video model evaluation benchmark grounded in physical laws. It requires models to perceive relevant physical quantities, articulate the governing laws, deduce subsequent motions consistent with those laws, and identify at which step the reasoning fails.

Figure | Apple-π Overview.

The specific design is as follows:

1. Video dataset: Orchard contains 400 classical mechanics videos, generated or selected after defining the physical laws and initial conditions. Tasks are divided into two categories—single law and combined multiple laws—covering gravity, collisions, inertia, and sequential motion transitions, evaluating whether the model can continuously update position and velocity.

Figure | Orchard Overview.

2. Benchmark Protocol: Apple-π splits the Orchard scenario into three categories of tasks—perception, representation, and reasoning—comprising five sub-tracks to evaluate models’ physical reasoning capabilities. Perception-text reproduces numbers and annotations in the scene; perception-graph locates and retains experimental objects. Representation-text selects the governing equation from candidate physical formulas and substitutes variables; representation-graph predicts the object’s position at the target time step and annotates velocity. Reasoning generates a complete motion video based on initial conditions from the first frame.

Figure | Apple-π Benchmark Protocol.

3. Evaluation Suite: Apple-π employs a hybrid scoring mechanism. Multimodal large model scoring assesses answer reasonableness, format compliance, and image clarity; in tracks with object masks and ground-truth trajectories, physics-based metrics quantify errors in object localization, trajectory, and velocity to determine whether motion adheres to physical laws. Scores across all tracks are normalized to the [0,1] range.

For any type of model, physical derivation has limited performance.

The research team evaluated 11 representative models on the full Apple-π benchmark. Overall results show that video pre-training, inference supervision, and unified understanding all enhance the models' perceptual and expressive capabilities; however, physical reasoning performance remains limited across all model types.

Specifically, the proprietary model results overall outperform the baseline open-source video models, as large-scale, high-quality video training can distill useful physical priors; VBVR-Wan2.2, fine-tuned with video inference, also demonstrates stronger competitiveness among video models; the significantly superior results of GPT Image 2 and Nano Banana 2 indicate that explicit understanding and controllable visual generation enhance text reading, object grounding, and final frame generation.

Even the best models achieve only a score of 0.473; physical motion reasoning remains the primary bottleneck today.

Figure | Key Results.

In addition, to further identify issues in the model across different reasoning stages, Apple-π breaks down the evaluation process into three phases: perception, law modeling, and dynamic reasoning, assessing the model’s capabilities in perception, alignment, reasoning, and generation.

The results show that the model's errors are not limited to the final generation stage but also occur during the intermediate physical reasoning phase: even when some models successfully perform annotation reading and object localization, they still significantly struggle with selecting physical rules, updating states, and predicting subsequent motion. They rely more on superficial visual pattern matching and have not yet developed stable physical state modeling capabilities.

Limitations

Although Apple-π can more finely evaluate the physical reasoning capabilities of video models, it still has some limitations.

For example, Apple-π’s scope remains limited. It primarily focuses on classical rigid body mechanics, including gravity-driven motion, momentum-conserving collisions, Newton’s first law, and simple combinations of these principles, but does not evaluate fluids, thermodynamics, electromagnetism, fracture, granular media, biological motion, or quantum phenomena.

At the same time, the objects and scenes covered by Apple-π are also incomplete. The dynamic objects in the data consist only of spheres, cubes, cylinders, and cones. Due to the use of a single fixed camera, it does not directly evaluate multi-view consistency or novel view synthesis capabilities.

The research team noted that Apple-π's input setup still relies on annotations from the first frame to establish the correspondence between the scene and physical quantities. Therefore, the benchmark primarily evaluates the model's ability to perform physical reasoning within existing visual scenes, rather than testing whether the model can construct a complete physical scene from text alone.

In addition, the current evaluation method has limitations: MLLM judges cannot fully determine whether physical processes are correct, so Apple-π also needs to incorporate metrics such as masking, pixel accuracy, and velocity, which are influenced by segmentation quality, frame differences, and annotation noise.

Finally, Apple-π also has limitations in time normalization. The evaluation assumes that the fully decoded output corresponds to the physical duration requested in the prompt, but video generation services may return fixed-length video containers that do not vary with the requested duration, thereby affecting the evaluation results.