Paper Detail

Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation

Zirui Wu, Chen Guo, Tianjian Jiang, Zeren Jiang, Zeming Li, Lingchen Yang, Longteng Duan, Ying Xue, Handi Yin, Shenyuan Gao, Martin R. Oswald, Jun Zhang, Ping Tan, Jie Song

arxiv Score 9.8

Published 2026-10-15 · First seen 2026-10-16

General AI

Abstract

We introduce P$^{3}$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a language instruction, and an optional visual goal. Our model integrates a compact, structured 3D action representation into a pretrained video diffusion transformer through a mixture-of-transformers architecture with shared base weights, modality-specific adaptation, and bidirectional visual-action attention. Randomized modality visibility lets video and 3D actions serve as either conditioning context or prediction targets across the three modes. Experiments show that unified training improves all three capabilities over single-mode training, including gains in planning quality under a vision-language model evaluation. Through unified training, our model outperforms specialized baselines on 3D hand reconstruction and action-conditioned video prediction across four datasets, while producing goal-directed manipulation videos with stronger instruction adherence and action progression than video-generation baselines. Project Page: https://p3-wam.pages.dev/

Workflow Status

Review status
pending
Role
unreviewed
Read priority
now
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@article{wu2026perceive,
  title = {Perceive, Predict, and Plan: a Unified World Action Model for Egocentric Dexterous Manipulation},
  author = {Zirui Wu and Chen Guo and Tianjian Jiang and Zeren Jiang and Zeming Li and Lingchen Yang and Longteng Duan and Ying Xue and Handi Yin and Shenyuan Gao and Martin R. Oswald and Jun Zhang and Ping Tan and Jie Song},
  year = {2026},
  abstract = {We introduce P\$\textasciicircum{}\{3\}\$-WAM, a unified world action model for egocentric dexterous manipulation that learns 3D perception, action-conditioned prediction, and goal-directed planning through three complementary modes: Perceive, which reconstructs articulated hand motion and camera trajectories from video; Predict, which generates visual futures under prescribed actions; and Plan, which jointly generates action trajectories and their anticipated visual consequences conditioned on an initial image, a l},
  url = {https://arxiv.org/abs/2610.22363},
  keywords = {cs.CV, cs.RO},
  eprint = {2610.22363},
  archiveprefix = {arXiv},
}

Metadata

{}