logo Idiap Research Institute        
 [BibTeX] [Marc21]
SoftGPT: Learn Goal-oriented Soft Object Manipulation Skills by Generative Pre-trained Heterogeneous Graph Transformer
Type of publication: Conference paper
Citation: Liu_IROS_2023
Publication status: Published
Booktitle: Proc. IEEE/RSJ Intl Conf. on Intelligent Robots and Systems (IROS)
Year: 2023
Abstract: Soft object manipulation tasks in domestic scenes pose a significant challenge for existing robotic skill learning techniques due to their complex dynamics and variable shape characteristics. Since learning new manipulation skills from human demonstration is an effective way for robot applications, developing prior knowledge of the representation and dynamics of soft objects is necessary. In this regard, we propose a pre-trained soft object manipulation skill learning model, namely SoftGPT, that is trained using large amounts of exploration data, consisting of a three-dimensional heterogeneous graph representation and a GPT-based dynamics model. For each downstream task, a goal-oriented policy agent is trained to predict the subsequent actions, and SoftGPT generates the consequences of these actions. Integrating these two approaches establishes a thinking process in the robot's mind that provides rollout for facilitating policy learning. Our results demonstrate that leveraging prior knowledge through this thinking process can efficiently learn various soft object manipulation skills, with the potential for direct learning from human demonstrations.
Keywords: robot learning
Projects Idiap
Authors Liu, J.
Li, Z.
Calinon, Sylvain
Chen, F.
Added by: [UNK]
Total mark: 0
  • Liu_IROS_2023.pdf