The field of physical AI has long struggled with a fundamental disconnect: video models can predict how pixels move, but they lack any grounding in the mechanics of the actual world. Pixels, after all, are merely visual evidence, not an ontological representation of physics. MirroS has introduced Code-as-World, a revolutionary paradigm that moves beyond pixel-prediction to create executable world representations. By rewriting real video footage into functional MuJoCo physics programs, this system allows an agent to verify scenes against physical laws rather than visual artifacts.
The Architectural Shift: From Pixels to Programs
At its core, Code-as-World transforms a standard video file into an Executable World Representation (EWR). This representation is categorized into three distinct, structured components, encoded in a scene.json format that can be directly executed by the MuJoCo physics engine:
- Composition: A definition of the environment’s entities, including object geometry, material properties like mass and friction, and physical constants such as gravity. Static entities like walls and floors are defined explicitly to facilitate collision detection.
- Evolution: The dynamic unfolding of the scene. This encompasses initial velocities, forces, active contact events, and termination states. Executing this code produces a precise physical trajectory.
- Appearance: The visual configuration of the scene, covering camera parameters, lighting, and textures. Crucially, modifying appearance parameters has zero impact on the underlying physical simulation.
The Agentic Discovery Loop
Rather than relying on one-shot neural prediction, Code-as-World employs an agentic search process to solve the inverse problem of scene recovery. The agent iterates through a five-round Propose-Instantiate-Execute-Render-Verify loop:
- Propose: Based on input evidence (masks, tracks, depth), the agent proposes a hypothesis EWR.
- Instantiate: The EWR is compiled into parameters for the MuJoCo simulator.
- Execute: The physics engine simulates the scene, calculating forces and collisions.
- Render: The simulation is projected into RGB, depth, and mask formats to match the input video.
- Verify: The model compares simulated keyframes against the real footage. If the error is too high, it generates specific feedback (Δ) to guide the next iteration.
Benchmark Performance: Setting a New Standard
The team behind Code-as-World fine-tuned models from the Qwen3.5 family, specifically targeting physical reasoning. When tested on the QuantiPhy benchmark, the results demonstrate that world-space supervision significantly outperforms standard training methods.
| Model Variant | Mean Relative Accuracy (MRA) |
|---|---|
| Code-as-World-VL-27B (Reasoning) | 58.6 |
| Code-as-World-VL-9B | 55.4 |
| Gemini-3.1 Flash | 54.8 |
| Code-as-World-VL-4B | 50.6 |
| ChatGPT-5.1 | 48.4 |
| Qwen3-VL-32B-Instruct | 40.2 |
Deployability and Research Access
MirroS has moved this research into the public domain by releasing the source code and two primary model weights under the Apache 2.0 license. These models, Code-as-World-VL-4B and Code-as-World-VL-9B, are currently accessible via vLLM with an OpenAI-compatible interface, making them immediately deployable for developers looking to integrate physical grounding into their AI agents.
By treating the physical world as code, this approach provides the training data necessary to teach models about mass, contact, and gravity—concepts that current video-generation models simply cannot learn from pixels alone.
