Non-Prehensile Model-Based Manipulation & Estimation Pipeline

I created a model-based manipulation, perception, and estimation pipeline for pick and pack operations using a singular non-prehensile end-effector. More specifically I successfully adapted a simplified complementary-free contact model and MPC from Jin et al. so as to quickly reorient a given object from one pose to another using another shape (simulating a robot end-effector) in MuJoCo. I then implemented an object 6D pose estimation module using FoundationPose so as to detect the object’s state in real-time. On top of this, I created and tuned an Unscented Kalman Filter using the FoundationPose estimates for the update step and the contact-model model for a prediction step. This pipeline was tested by rendering the environment in Blender, extracting RGB and depth images from a simulated sensor, and then adding additional noise before attempting estimation.

This project was a combination of two final projects: one for Cornell’s Model-Based Estimation (MAE 6760) and the other serving as my Mechanical Engineering Senior Design Project. The two reports are below.