Paper Detail

When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?

Anton Hristov, Aleksandar Slavov, Vasil Vasilev

huggingface Score 7.0

Published 2026-11-01 · First seen 2026-11-03

General AI

Abstract

Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked pixel reconstruction, and simple visual statistics. Appearance remains easy to decode, while V-JEPA2 exposes modest information about whether events are possible or impossible. Looking at the latent geometry, appearance changes become smaller compared with differences between videos, their directions line up more consistently across scenes, and their overlap with physical changes decreases in deeper layers. Across action-conditioned models, incorrect actions raise prediction error, while appearance changes also produce substantial responses. In Physion++ video clips, repeating a single frame can produce higher error than the original clip, while changing color throughout the clip can lower it. Using Kubric, a simulation and rendering framework, we find that impossible events can produce lower error than possible events. A fixed error threshold also incorrectly flags many possible events as impossible. A model can therefore represent physical information and respond to incorrect actions without its native prediction error reliably measuring physical surprise.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{hristov2026when,
  title = {When Does Latent Prediction Error Support Physical Surprise Claims in Video World Models?},
  author = {Anton Hristov and Aleksandar Slavov and Vasil Vasilev},
  year = {2026},
  abstract = {Prediction errors from video world models are often treated as physical surprise, but they can also reflect appearance, static content, motion, or ordinary prediction difficulty. We present an evaluation framework that separates what frozen representations contain, how they organize appearance and physical differences, and what changes a model's own prediction error. On IntPhys2, a benchmark of physically possible and impossible events, we compare representations from latent prediction, masked p},
  url = {https://huggingface.co/papers/2611.01815},
  keywords = {video world models, frozen representations, V-JEPA2, IntPhys2, latent geometry, action-conditioned models, Physion++, Kubric, code available, huggingface daily},
  eprint = {2611.01815},
  archiveprefix = {arXiv},
}

Metadata

{}