: Benchmarking Bimanual Manipulation across Simulation and Reality

Chak-Wing Mak1,6,*, Zewen Yang1,6,*, Xiaojie Zhang2,*,†, Guanyu Zhu3,6, Yuhan Rui4, Yu-Kai Wang1, Zijie Nie6, Kai Tang1, Yinyu Wu1, Chuyao Fu6, Cheng-Ching Tseng6, Kuangzhi Ge6, Shengjie Gao5, Kevin Zhang1,6, Xiaowei Chi2,6, Yike Guo2, Sirui Han2,‡, Shanghang Zhang1,‡

  • 1State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University
  • 2The Hong Kong University of Science and Technology
  • 3Fudan University
  • 4Shanghai Jiao Tong University
  • 5University of Electronic Science and Technology of China
  • 6Muka Robotics

*Equal contribution†Project leader‡Corresponding authors

Peking University
The Hong Kong University of Science and Technology
Fudan University
Shanghai Jiao Tong University
University of Electronic Science and Technology of China
Muka Robotics

TOWER project trailer

TOWER · Project trailer · 2 minutes

Download video

Six task families across simulation and reality, the Real2Sim pipeline, and benchmark findings. On-screen captions and music.

Explore each task and its real-world recordings

Abstract

Every move changes what comes next.

Complex robotic manipulation requires interpreting visual scenes and anticipating how actions change object contacts and structural support. Evaluating these capabilities calls for diverse, long-horizon tasks in which the physical consequences of earlier actions shape subsequent execution. To this end, we introduce TOWER, a novel Jenga-based benchmark for sustained contact and structural change, comprising six configurable task families involving bimanual coordination and tool use. Jenga provides repeatable structures in which local manipulations alter the support and stability needed for subsequent actions. A Real2Sim pipeline builds corresponding simulated environments for scalable evaluation. We evaluate four policy architectures across twelve configurations using separately trained real and simulated policies. Progress metrics reveal substantial partial completion hidden by binary task success, while taller configurations generally exhibit lower progress. Real–sim evaluation shows strong progress correlation (Pearson r = 0.880) and a normalized MMRV of 0.1217. TOWER provides a unified testbed for studying how policies respond to interaction-induced scene changes and supports systematic policy evaluation across reality and simulation.

Task families
6
Configurations
12
Training demonstrations
12,000
Hours of demonstrations
455.7
Evaluation episodes
960
Figure 1 of the paper. (A) Six atomic actions, four key moments of a Task F episode, and how taller towers extend the plan from 8 to 16 operations. (B) Real2Sim stages: scene reconstruction, camera alignment, dynamics alignment, friction calibration. (C) Paired evaluation: SIM versus REAL progress for four policies, Pearson r 0.880, MMRV 0.1217, and matching REAL and SIM failure sequences.
Figure 1. Overview of TOWER. (A) Dataset scale and task design. (B) Scene reconstruction, camera alignment, dynamics alignment, and friction calibration. (C) Cross-domain progress correspondence, model comparisons, and representative REAL/SIM sequences.

Task suite

Six task families, two tower heights.

Easy tasks relocate blocks, medium tasks extract or insert, and hard tasks add tool use and reconstruction. Choose a cinematic simulation demo or a real-world recording. Cinematic clips show three layers; real recordings cover both heights.

Simulation · 3 layers · scripted demonstration · edited recorded-state replay; speed labeled in each shot.

Task A · Easy

Cross Transfer

Blocks are transferred one by one to rebuild the cross-oriented tower on a new base.

Planned operations
6 (3 G + 3 P)
Operation plan
Per source layer: grasp → place
Clip length
0:15

Real2Sim

A simulator measured from the physical station.

Scene reconstruction, camera alignment, robot dynamics alignment, and physics alignment build the simulated counterpart used for the shared task suite, with held-out checks for camera and robot-response alignment.

Figure 3 of the paper, Real2Sim construction: (A) station and block reconstruction into an assembled USD scene, (B) fixed and wrist camera alignment, (C) two-stage CMA-ES robot dynamics alignment with simulated versus recorded joint response, (D) force-gauge friction measurement for Jenga–Jenga and Jenga–PLA contact.
Figure 3. Real2Sim construction. Rendered from the vector figure in the paper source.
  1. A

    Scene reconstruction

    Station CAD, photographs, and measured block dimensions and mass define the scene; an agent-assisted CAD-to-SimReady workflow builds the assets.

  2. B

    Camera alignment

    Point-to-plane ICP and silhouette, contour, and depth losses align the static camera view; wrist cameras are calibrated to the gripper from multi-pose silhouettes.

  3. C

    Dynamics alignment

    Two-stage CMA-ES fits response parameters, command delay, and PD gains. Held-out position RMSE falls by 28.0% and 18.5% (left and back arms); velocity RMSE by 45.2% and 51.1%.

  4. D

    Physics alignment

    A robot-held force gauge measures breakaway and sliding friction. Block coefficients of 0.551 static and 0.263 dynamic are applied in SIM.

Evaluation

Look beyond the final outcome.

Two VLAs (π0.5, StarVLA-α) and two WAMs (Fast-WAM, OpenWAM) are trained separately on REAL and SIM demonstrations and evaluated in their own domain, with matched architectures, tasks, and scoring. Each setting runs ten episodes per domain: 960 episodes in total.

TSR ↑
Task success rate: all prescribed operations completed with a stable final arrangement.
LHPR ↑
Long-horizon progress rate: successful planned operations divided by planned operations.
ODR ↓
Operation disturbance rate: disturbed non-target blocks per counted step; can exceed one.
GSR · PSR ↑
Success rates of the planned G (grasp or interaction) and P (place or terminal) operations.

Finding 1

Progress reveals partial completion.

The 31 settings with no successful episode still span 11.9–77.5% LHPR. Fast-WAM completes 3.1 of four operations on average on REAL Task D (3 layers) while failing all ten episodes.

Finding 2

Taller towers lower completion.

Mean LHPR falls from 79.62% to 63.19% from two to three layers, with decreases in five of six tasks. B, E, and F drop by 27.50, 33.54, and 28.59 points.

Finding 3

SIM and REAL progress correspond.

Across 48 paired settings, Pearson r = 0.880 with a mean absolute gap of 0.994 successful operations per episode; normalized MMRV is 0.1217.

Results explorer

Every policy on every task. Values from Appendix Tables 17–18; each mark is a ten-episode result.

Metric
Compare
Height

Show the data table

SIM–REAL correspondence

Mean successful operations per episode in each domain for the 48 paired settings: the coordinates of Figure 4, from Appendix Tables 19–20.

Policies
Height

Show the data table

Resources

Build on TOWER.

The public dataset provides 600 raw simulation demonstrations, a subset of the paper’s 12,000 SIM/REAL demonstrations. Model-specific preprocessing is left to users. Simulator assets are available separately; real-world demonstrations are not included in this dataset release.

Citation

@misc{tower2026,
  title = {TOWER: Benchmarking Bimanual Manipulation across Simulation and Reality},
  author = {Mak, Chak-Wing and Yang, Zewen and Zhang, Xiaojie and Zhu, Guanyu and Rui, Yuhan and Wang, Yu-Kai and Nie, Zijie and Tang, Kai and Wu, Yinyu and Fu, Chuyao and Tseng, Cheng-Ching and Ge, Kuangzhi and Gao, Shengjie and Zhang, Kevin and Chi, Xiaowei and Guo, Yike and Han, Sirui and Zhang, Shanghang},
  year = {2026},
  url = {https://tower-benchmark.com}
}