The project, “Robotic Learning for Mobile Manipulation and Social Interaction,” addresses the challenge of autonomous robots meeting complex needs, such as physical and social interactions in real-world environments. Current challenges include adaptability to unforeseen situations, smooth collaboration with humans, and the ability to navigate diverse settings. These issues are particularly critical in domains like domestic services, logistics, and agriculture.
The context
The ISIR context provides unique expertise in the multidisciplinary aspects of robotic learning, including control, machine learning, and social interaction. ISIR also provides access to advanced robotic platforms such as Tiago, Miroki, Pepper, and PR2, specifically designed for mobile manipulation. With these resources and its innovative approach, ISIR plays a pivotal role in developing autonomous robots capable of addressing societal and industrial challenges while strengthening collaborations within the European robotics community.
Objectives
The project aims to achieve a high level of autonomy for robots in complex environments. One of the primary goals is leveraging large language models (LLMs) for robotic planning, affordance identification, and object grasping, enabling better understanding and interaction with the real world. Additionally, the project seeks to develop an integrated system combining state-of-the-art perception models, particularly vision-based ones, and advanced control methods. For example, the QD-grasp (Quality-Diversity grasping) technique, developed at ISIR, is a cornerstone of this approach.
The overarching goal is to create robots that can interact autonomously, efficiently, and reliably with their environments while exploiting synergies between perception, control, and machine learning for applications across various domains.
The results
The project successfully integrated the QD-grasp stack on the Tiago robot, featuring advanced functionalities such as grasp generation, object detection, segmentation, and identification. Additionally, LLM-based planning was implemented, enabling the robot to understand and execute tasks expressed in natural language by human users. These developments significantly enhance human-robot interaction and the robot’s ability to operate in complex environments.
The project also participated in the annual euROBIN competition, showcasing advancements in the modularity and transferability of robotic skills. We continually share our developed components for mobile manipulation with the community, contributing to the collective evolution of robotic technologies and their application to real-world challenges.
Partnerships and collaborations
This project is an internal initiative of ISIR, leveraging expertise from various research and development areas. It particularly benefits from the contributions from:
– the ASIMOV team (on the manipulation and robotic interaction aspect),
– the ACIDE team (on cognition and interaction),
– and the Intelligent Systems Engineering priority.
These internal collaborations enable the mobilization of complementary skills in robotic learning, perception, control, and social interaction, thereby enhancing ISIR’s capacity to address major scientific and technological challenges.
