Paper Detail

Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments

Xusen Hei, Jiali Chen, Jinyu Yang, Yi Cai

huggingface Score 11.0

Published 2026-11-02 · First seen 2026-11-03

General AI

Abstract

Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{hei2026lab2sim,
  title = {Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments},
  author = {Xusen Hei and Jiali Chen and Jinyu Yang and Yi Cai},
  year = {2026},
  abstract = {Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental proced},
  url = {https://huggingface.co/papers/2611.02928},
  keywords = {multimodal models, state machines, execution-based verification, apparatus fidelity, code available, huggingface daily},
  eprint = {2611.02928},
  archiveprefix = {arXiv},
}

Metadata

{}