Latent space planning for multi-object manipulation with enviornment-aware relational classifiers

A useful robot operating in human environments will no doubt encounter scenes where it needs to manage and manipulate multiple objects. The interactions of these objects with each other and the environment are important to reason about in order to predict the outcome of any action. We aim to learn about these interactions from the perspective of the relationships between the objects and how they change. We predict future relationships between objects given an action, using only partial viewpoint clouds. We can use this prediction to plan using a graph search to accomplish goals specified by relations. We find the best model for this is based on using a transformer to learn a latent space dynamics function and predict relations from that latent space. We show that we can adapt to novel objects and environments and transfer from simulation to real robotics with no extra training.