This paper was just submitted to RSS 2026, so more details (and videos) will be coming soon. As co-author I helped develop a latent dynamics model for whole-arm robot manipulation tasks, training it on various sensor modalities through a contact-centric representation. These included whole-arm tactile sensors, an RGB-D camera, and proprioception (joint states and velocities over time). This latent model allows us to predict contacts and plan complex motion, while our kinematics model ensures potential actions are both feasible and within safety limits. When used as part of a Model Predictive Control formulation, these two models enable better completion of manipulation tasks with whole-arm contact than baselines such as diffusion policies. As part of this project I determined which scenarios would be best for demonstrating the method, coded up ROS nodes, set up and ran experiments, wrote parts of the paper, and created key figures.

TACTIC: Tactile and Vision Conditioned Contact Centric Control for Whole-Arm Manipulation

Paper Abstract: Whole-arm physical human-robot interaction (pHRI) involves direct, sustained contact with a person. The robot must complete the task while distributing contact across multiple links to maintain safe interaction forces as contacts form, slide, and break. This setting breaks common implicit assumptions in many learning-based manipulation pipelines: arm configuration tightly couples motion and forces, contact state is partially observed under occlusion, and purely learned rollouts can become physically inconsistent under distribution shift because many whole-arm contact configurations are sparsely represented in the data. To address this, we propose TACTIC (Tactile and Vision Conditioned Contact-Centric Control), a receding-horizon controller for whole-arm pHRI. TACTIC uses a contact-centric hybrid predictive model that couples RGB-D, distributed tactile sensing, and a compact 2D proximity representation. The model couples a learned, action-conditioned latent dynamics model with analytical kinematics through contact Jacobians, enabling rollouts of future contact configurations and interaction forces. TACTIC integrates these rollouts into a sampling-based MPC planner with contact-aware action sampling. Contact Jacobian-based projections steer sampled action sequences toward force-modulating directions, and objectives defined over predicted proximity and interaction forces trade task progress against whole-arm force regulation. We evaluate TACTIC in simulation against state-of-the-art model-based and model-free methods, and perform ablations that isolate the contribution of each design choice. In all of these experiments, TACTIC consistently outperforms alternative methods. We further demonstrate real-world performance on a robot with distributed tactile sensing in two caregiving tasks, turning over and limb repositioning, as well as goal reaching in a dynamic 3D maze that requires multi-contact trajectories.