IEEE Robotics and Automation Letters (RA-L), 2026 (Accepted)
Ruopeng Huang completed this work during an internship at the Trustworthy Autonomous Systems Laboratory.
Accurate and safe robotic manipulation under dynamic and visually occluded conditions remains a core challenge in real-world deployment. We introduce SyncTwin, a novel digital twin framework that unifies fast 3D scene reconstruction and real-to-sim synchronization for robust and safety-aware robotic manipulation in such environments. In the offline stage, we employ VGGT to rapidly reconstruct object-level 3D assets from RGB images, forming a reusable geometry library. During execution, SyncTwin continuously synchronizes the digital twin by tracking real-world object states and uses stored assets to complete partially observed geometry. The synchronized twin enables motion planners to compute collision-aware robot trajectories in simulation, with execution on the real robot closing the real-to-sim-to-real loop. Experiments in dynamic and occluded scenes show that SyncTwin improves manipulation performance and motion safety, demonstrating the effectiveness of digital twin synchronization for real-world robotic execution.
A single camera often sees only fragments of an object, while moving obstacles quickly make a static scene model outdated. SyncTwin combines object-level geometry memories with continuous pose updates, giving the planner a more complete and current representation of the workspace.
The framework connects RGB-only asset construction, live RGB-D perception, and model-based planning. It uses robot models rather than robot-specific learned policies, so the design supports different manipulators without policy retraining.
SyncTwin consists of two stages: fast RGB-only construction of reusable object assets, followed by online tracking and geometry completion for continuous real-to-sim-to-real synchronization.
VGGT reconstructs a scene-level point cloud from a small set of RGB images. SAM2 generates masks from manually clicked points or detector-provided bounding boxes, and expanded multi-view masks compensate for camera-pose inaccuracies during projection. A geometric sphere-expansion procedure detects opening boundaries and removes support-plane noise. A marker of known physical dimensions provides metric scale alignment, and feature-aware mesh simplification produces lightweight, simulation-ready assets for the memory bank.
Streaming RGB-D observations drive temporally consistent segmentation and object-state updates. The tracking backend can use real-time point-cloud tracking with colored-ICP (RT-ICP), or FoundationPose when reference meshes and sufficient GPU memory are available. Known objects reuse complete memory-bank geometry; unseen objects retain a point-cloud branch with online collision-geometry generation.
Updated geometry and poses are streamed into Isaac Sim. GraspGen predicts grasp candidates, while cuRobo MPC replans collision-aware motions. The real robot executes the first action of each short-horizon plan, closing the real-to-sim-to-real loop.
Setup: Franka Panda · Intel RealSense D455 · iPhone 12 RGB capture · Isaac Sim 4.0 · cuRobo MPC · one RTX 4090. Scene synchronization runs asynchronously to the 10 Hz planning loop.
SyncTwin generates usable object meshes from only 5–10 RGB images in approximately 1–2 minutes. Photogrammetry, 3D Gaussian Splatting, and Nerfstudio require more images and typically 4–8 minutes of processing. In the reported setup, the full workflow—including capture, upload, prompt initialization, and script launch—takes less than 5 minutes.
We evaluate in-place obstacle rotation (SelfRot) and obstacles moving into the robot trajectory (EnterTraj), using four boxes and 20 trials per box per condition. SyncTwin uses RT-ICP. Stored geometry further improves results when objects are partially occluded.
For unseen objects, SyncTwin achieves a weighted avoidance score of 85.5% for SelfRot (0 m/s translational speed), versus 50.3% with NVBlox. For EnterTraj at 0.2 m/s, the scores are 71.5% and 37.0%, respectively. Memory-bank geometry raises SyncTwin’s scores to 93.5% for SelfRot and 78.8% for EnterTraj at 0.2 m/s for seen objects. At 0.5 m/s, SyncTwin’s EnterTraj scores fall to 35.8% and 44.5% for unseen and seen objects, respectively.
| Method | Motion | Translation speed | Unseen objects | Seen objects |
|---|---|---|---|---|
| NVBlox | SelfRot | 0 m/s | 50.3 ± 15.1 | — |
| NVBlox | EnterTraj | 0.2 m/s | 37.0 ± 20.4 | — |
| SyncTwin | SelfRot | 0 m/s | 85.5 ± 4.2 | 93.5 ± 3.1 |
| SyncTwin | EnterTraj | 0.2 m/s | 71.5 ± 5.4 | 78.8 ± 6.9 |
| SyncTwin | EnterTraj | 0.5 m/s | 35.8 ± 20.1 | 44.5 ± 13.9 |
Weighted score = (full avoidances + 0.8 × edge avoidances) / trials × 100%. Edge avoidance allows slight contact without object displacement. Seen objects have memory-bank assets; unseen objects do not. SelfRot has zero translational speed while the obstacle rotates.
Single-view point clouds can miss critical geometry, such as a cup handle, and produce colliding grasp candidates. Aligning the complete stored asset supplies that missing geometry and yields denser, more geometrically feasible grasps.
We compare RT-ICP and FoundationPose on a box, cup, bottle, and ketchup bottle, using the same Stage I assets and RGB-D observations. Each object is translated by 15 cm along each Cartesian axis and rotated by 90° around roll, pitch, and yaw; each motion is repeated 50 times.
| Backend | Condition | Translation | Rotation |
|---|---|---|---|
| RT-ICP | Unoccluded | 95.23 ± 0.74 | 84.85 ± 5.68 |
| RT-ICP | Occluded | 88.68 ± 2.81 | 76.88 ± 5.06 |
| FoundationPose | Unoccluded | 97.65 ± 0.71 | 94.35 ± 2.10 |
| FoundationPose | Occluded | 94.28 ± 2.04 | 91.60 ± 1.94 |
Normalized accuracy is 1 − translation error / 15 cm or 1 − angular error / 90°, expressed as a percentage.
| Metric / configuration | RT-ICP | FoundationPose |
|---|---|---|
| Four-object tracking, unoccluded (FPS) | 3.43 | 11.57 |
| Four-object GPU memory (GiB, mean ± SD / peak) | 5.70 ± 0.50 / 6.10 | 7.70 ± 0.38 / 16.03 |
| One seen object: observation-to-update (ms) | 73.93 ± 9.74 | 25.09 ± 2.87 |
| One seen object: scene update rate (Hz) | 13.53 | 39.86 |
| One seen + one unseen object: update time / rate (ms / Hz) | 185.34 ± 21.67 / 5.40 | Not applicable |
| Initialization / recovery time (ms) | 214.12 ± 46.08 | 819.63 ± 46.35 |
| One seen object: GPU memory (GiB, mean ± SD / peak) | 5.59 ± 0.51 / 5.99 | 7.56 ± 0.37 / 15.86 |
Observation-to-update latency excludes asynchronous motion planning. The unseen-object RT-ICP configuration includes online V-HACD collision-geometry generation. Initialization/recovery also includes detection and segmentation; FoundationPose additionally re-registers the mesh.
FoundationPose offers higher pose accuracy and faster tracking when reference assets and GPU memory are available. RT-ICP uses less memory, recovers more quickly after an object returns to view, and retains support for completely unseen objects without stored assets.
Without mask expansion, inconsistent multi-view projections remove valid object geometry. Without support-plane denoising, table points contaminate the reconstructed mesh. Both components are needed to extract complete, clean object assets.
Faster obstacle motion remains challenging: at 0.5 m/s, perception latency, the planner’s lack of obstacle-motion prediction, and execution lag reduce avoidance performance. Abrupt accelerations and direction changes were not explicitly evaluated and would further challenge the reactive loop. Support-plane denoising can also fail for tall or weakly supported objects such as wine glasses.
Future work includes integrating asset construction into online synchronization, appearance-level consistency for visuomotor learning, distributed or multi-GPU processing, and probabilistic obstacle-state modeling for uncertainty-aware planning.
@article{huang2026synctwin,
title={{SyncTwin}: Fast Digital Twin Construction and Synchronization for Safe Robotic Manipulation},
author={Huang, Ruopeng and Yang, Boyu and Gui, Wenlong and Morgan, Jeremy and B{\i}y{\i}k, Erdem and Li, Jiachen},
journal={IEEE Robotics and Automation Letters},
year={2026}
}