The Audio-touch project approach combines the results of research into social touch and the sonification of movement, making it possible to extract certain significant features from tactile interactions and convert them into sounds. In the project’s experimental studies, participants listening to skin-to-skin audio-touch stimuli accurately associated the sounds with specific gestures and consistently identified the socio-emotional intentions underlying the touches converted into sounds.
In addition, the same movements were recorded with inanimate objects, and further experiments revealed that participants’ perception was influenced by the surfaces involved in tactile interactions (skin or plastic). This audio-touch approach could be a springboard for providing access to the ineffable experience of social touch at a distance, with human or virtual agents.
The context
Touch is the first sensory modality to develop, and social touch has many beneficial effects on the psychological and physiological well-being of individuals. The lack of social touch accentuates feelings of isolation, anxiety and the need for social contact. Given this situation, there is an urgent need to explore ways of recreating the beneficial effects of emotional touch from a distance.
This project was born of this urgency, but also of a fundamental observation: touch and hearing share a common physical basis, that of vibrations. Sonification studies have already shown that it is possible to translate movements and their properties into sounds. It is therefore conceivable to translate certain characteristics of skin-to-skin interactions into perceptible auditory signals. The convergence of this need for a tactile link at a distance and the physical similarities between touch and sound has given rise to the Audio-touch project, which aims to transmit the information of affective touch via sound.
Objectives
The main objective of this research is to study the possibilities offered by audio-touch: can we convert skin-to-skin touch into sound? More specifically, the team then experimentally determined what social touch information the audio-touch stimuli could communicate to the people listening to them: can we recognise different types of touch gestures in our audio-touch stimuli? can we recognise different socio-affective intentions underlying these sonified touch gestures? does the surface on which the touch takes place (skin versus inanimate object) influence these perceptions?
The results
The four experiments conducted in this study show that social tactile gestures, their socio-emotional intentions and the surface touched (skin vs plastic) can be recognised solely from the audio-touch sounds created. The results show that :
– the participants correctly categorised the gestures (experiment 1),
– participants identified the emotions conveyed (experiment 2),
– their recognition was influenced by the surface on which the touch took place (experiments 3 and 4).
These results open up new perspectives at the intersection of the fields of haptics and acoustics, with the Audio-touch project situated precisely at the crossroads of these two fields, both in terms of its methodology for capturing tactile gestures and its interest in the human perception of sound signals derived from touch. They are also in line with research into human-computer interaction, and suggest that sounds derived from social touch could enrich multisensory experiences, particularly in interactions with virtual social agents.
Partnerships and collaborations
The project is a collaboration between :
– Alexandra de Lagarde, PhD student in the ACIDE team at ISIR,
– Catherine Pelachaud, CNRS research director in the ACIDE team at ISIR,
– Louise P. Kirsch, lecturer at Université Paris Cité, and former post-doctoral student in the Piros team at ISIR,
– Malika Auvray, CNRS research director in the ACIDE team at ISIR.
In addition, for the acoustic aspects of the project, the study benefited from the expertise of Sylvain Argentieri for the recordings and Mohamed Chetouani for the signal processing.
It is also part of the ANR MATCH programme, which explores the perception of touch in interactions with virtual agents, in partnership with HEUDIASYC (UTC) and LISN (Université Paris Saclay).
MAPTICS Project: Developing a Multisensory Framework for Haptics
The MAPTICS project that aims to develop a novel multimodal haptic framework by characterizing how users integrate multisensory cues into a unified perception and which facets of touch are the most essential to reproduce in virtual and digital applications.
To that end, it will investigate the potential of vibro-frictional rendering and multisensory reinforcement to create the illusion of 3D objects on the screen of a novel multimodal haptic interface and explore how haptics can be integrated in the next-generation of multisensory interfaces that combine high-fidelity tactile, auditory and visual feedback. Ultimately, this research it will enable the inception and evaluation of novel multimodal HCI applications for healthy and sensory impaired users.
The context
Current HCI interfaces commonly provide high-resolution visual and auditory feedback. On the opposite, haptic feedback is often absent from displays despite recent commercial products such as the Taptic engine on high-end Apple smartphones. Although multimodal interaction has the potential to enhance the acceptability, improve interaction efficiency, and be more inclusive for sensory impaired people, its development is hampered by the lack of multisensory framework for integrating haptic feedback with audio-visual signals, as well as by the low-fidelity of haptics in comparison to vision and audition in displays.
Consequently, the future of multisensory feedback faces the challenges of the design of high-fidelity haptic feedback and the efficient use of the cognitive principles underlying multisensory perception. In this context, novel rendering strategies and more capable devices integrating rich haptic feedback with the existing high-resolution audio-visual rendering are increasingly considered to optimize the feedback provided to users during their interaction with interfaces.
Objectives
– Develop a large interactive display capable of rendering vibrotactile and force-controlled haptic feedback alongside high-resolution audio-visual capabilities. The challenge will be to create an active touch interface are that provides richer feedback than the current state of the art scientific and commercial devices.
– Developing a realistic framework for multisensory rendering and use it to understand the cognitive processes at play when multisensory input is processed. For this goal, we will use the multisensory platform being developed in MAPTICS to manipulate the congruence between auditory, visual and tactile sensory inputs with the aim to investigate the multimodal reinforcement or disruption of cognitive representations.
– We will leverage the previous two subgoals to build a user interface with high-resolution multisensory reinforced haptic feedback. With this novel interface, we will study interactive scenarios related to the navigation in multisensory interactive maps or to the use of virtual buttons.
The results
Simultaneous vibrotactile stimulation and ultrasonic lubrication of the finger-surface interaction has not been studied yet even though it has the potential to lift serious limitations of current technology such as its inability of ultrasonic vibration to provide compelling feedback when the user gesture does not involve motion of the finger or the difficulty to create virtual shapes with mechanical vibrations.
Understanding better the joint implementation of both types of feedback will empower Haptic and HCI scientists in their research for more efficient and more compelling haptic devices. Also, vibrotactile and frictional haptic rendering are currently separate research topics.
After Maptics, we expect that haptic research will increasingly study these two complementary feedbacks in connection in order to efficiently develop more integrated rendering. In addition, the project will provide a new multisensory experimental tool that will enable a more advanced study and understanding of the underlying mechanisms of sensory integration.
Partnerships and collaborations
The MAPTICS project is an ANR (Agence nationale de la recherche) project, run in partnership with Betty Lemaire-Semail from the L2Ep laboratory at the University of Lille.
The project, “Robotic Learning for Mobile Manipulation and Social Interaction,” addresses the challenge of autonomous robots meeting complex needs, such as physical and social interactions in real-world environments. Current challenges include adaptability to unforeseen situations, smooth collaboration with humans, and the ability to navigate diverse settings. These issues are particularly critical in domains like domestic services, logistics, and agriculture.
The context
The ISIR context provides unique expertise in the multidisciplinary aspects of robotic learning, including control, machine learning, and social interaction. ISIR also provides access to advanced robotic platforms such as Tiago, Miroki, Pepper, and PR2, specifically designed for mobile manipulation. With these resources and its innovative approach, ISIR plays a pivotal role in developing autonomous robots capable of addressing societal and industrial challenges while strengthening collaborations within the European robotics community.
Objectives
The project aims to achieve a high level of autonomy for robots in complex environments. One of the primary goals is leveraging large language models (LLMs) for robotic planning, affordance identification, and object grasping, enabling better understanding and interaction with the real world. Additionally, the project seeks to develop an integrated system combining state-of-the-art perception models, particularly vision-based ones, and advanced control methods. For example, the QD-grasp (Quality-Diversity grasping) technique, developed at ISIR, is a cornerstone of this approach.
The overarching goal is to create robots that can interact autonomously, efficiently, and reliably with their environments while exploiting synergies between perception, control, and machine learning for applications across various domains.
The results
The project successfully integrated the QD-grasp stack on the Tiago robot, featuring advanced functionalities such as grasp generation, object detection, segmentation, and identification. Additionally, LLM-based planning was implemented, enabling the robot to understand and execute tasks expressed in natural language by human users. These developments significantly enhance human-robot interaction and the robot’s ability to operate in complex environments.
The project also participated in the annual euROBIN competition, showcasing advancements in the modularity and transferability of robotic skills. We continually share our developed components for mobile manipulation with the community, contributing to the collective evolution of robotic technologies and their application to real-world challenges.
Partnerships and collaborations
This project is an internal initiative of ISIR, leveraging expertise from various research and development areas. It particularly benefits from the contributions from:
– the ASIMOV team (on the manipulation and robotic interaction aspect),
– the ACIDE team (on cognition and interaction),
– and the Intelligent Systems Engineering priority.
These internal collaborations enable the mobilization of complementary skills in robotic learning, perception, control, and social interaction, thereby enhancing ISIR’s capacity to address major scientific and technological challenges.
Open A-Eye project: Kinesthetic feedback to guide the visually impaired
The originality of the A-Eye team’s device is that it provides kinaesthetic guidance that is more intuitive than the audio and/or vibrating feedback offered by commercial solutions. This reduces the cognitive load required to follow the information. This device, the fine-tuning of the feedback it provides and all the other solutions for meeting the various Cybathlon* challenges were co-designed with the help of our ‘pilot’. The A-Eye team places co-creation at the heart of the process, working with associations, accessibility experts and the blind A-Eye pilot. The aim is to create a solution that is ergonomic, accessible, easy to use and adapted to everyday life.
The A-Eye team’s device incorporates cutting-edge technologies, combining intuitive kinaesthetic feedback with computer vision and artificial intelligence functionalities. The wearable device, in the form of a harness/backpack, offers precise navigation, mimicking the interaction with a human guide and providing an intuitive experience.
*The Cybathlon is an event that takes place every 4 years and is organised by the Swiss Federal Institute of Technology in Zurich. It challenges teams from all over the world in 8 events. The aim is to demonstrate the technological advances that have been made in assisting people with disabilities to carry out everyday tasks
The context
It is well known that the least tiring guidance in a new environment is that provided by a person trained in guiding techniques. This guidance is much more intuitive and requires much less concentration than walking with a conventional assistance device such as a white cane.
A new environment involves obstacles that are difficult to predict (particularly those at head height) and a trajectory that has to be prepared in advance. This means a significant cognitive overload. In this situation, a digital assistance system capable of sensing the environment, calculating a trajectory and providing intuitive information (positive information) on the direction to follow would be much easier to integrate than a solution that only indicates obstacles (negative information), as is the case with white canes, even so-called ‘intelligent’ ones. Intuitive information could mimic the information about forces and movements exchanged between the guide and the guided. We call this type of information kinaesthetic information.
Our expertise in this area takes several forms.
– As a roboticist, we have noted that the technologies used for obstacle detection and trajectory planning for autonomous robots could find a positive echo in the development of an assistance device;
– On the other hand, ISIR and its specific activities are concerned precisely with good practice and the development of new devices for augmenting/substituting sensory information in different application contexts (assistance/rehabilitation/surgery).
Our objectives
Our ambition was to design a device that was as intuitive as possible. The A-Eye team’s device was therefore created with a series of converging objectives:
– to exploit the context offered by the international Cybathlon competition to validate the effectiveness of the sensory feedback developed within our teams,
– and to take advantage of the skills of the students on our courses to highlight their full potential.
The device takes the form of a harness/plastron to which a kinaesthetic feedback system (pantograph) is attached, with a 3D camera. It also has a powerful computer for analysing/mapping the environment before proposing a trajectory for reaching the desired position, updating it as new obstacles appear. The kinaesthetic feedback enables effort to be applied in two directions (left/right and forward/backward), providing an intuitive indication of the direction to follow.
This device represents an innovative solution at the frontier of current technologies and software developed in robotics, image processing, artificial intelligence and haptic/kinesthetic communication.
The results
The device developed at ISIR has reached a sufficient level of maturity, enabling it to be used autonomously by the pilot, with the ability to change mode according to the challenges encountered.
Weekly training sessions with the pilot, Salomé Nashed, who has been visually impaired since birth and is a biology researcher, enable the guidance system to be fine-tuned and personalised to suit the various Cybathlon events. Although she describes the feedback as ‘intuitive’, it is essential to test it with a wider range of users in order to continue to improve the solution and enhance its customisability. At the same time, presenting the system will raise awareness of current technological advances, demonstrating both their potential and their limitations. The partnership with the INJA (Institut National des Jeunes Aveugles) Louis Braille will provide an opportunity to present the system to students and mobility instructors.
The main task will be to make the existing system open-source. We detail here the approach envisaged to achieve this objective:
– Documentation and maintenance of an open-source GIT,
– Assessing and optimising the choice of materials and hardware,
– Drawing up plans for 3D printing and laser cutting,
– Creating video tutorials,
– Throughout the development process, we will be gathering feedback from ‘technical’ users:
(1) Take into account feedback from a group of student users from the Sorbonne University fablab, who will have to reproduce the device from the available documentation,
(2) Take into account feedback from the INJA Louis Braille locomotion instructors, who will be the first to test the device as a newly-formed technical team. Their role will be to get to grips with the device and propose customisations adapted to users, in order to improve its ergonomics and better respond to the specific needs and difficulties of each user.
As well as opening up the current system, the project also aims to evaluate and develop solutions that are more closely connected with the people concerned. Contacts have therefore been made with the INJA Louis Braille to give them the opportunity to work on participative innovations. This project is also being carried out with Sorbonne University’s FabLab. The project will also provide an opportunity to organise discussion groups and brainstorming sessions between researchers, Sorbonne University students and young blind and visually impaired people. These activities will be supported by the project to develop technological building blocks adapted to the people concerned.
Partnerships and collaborations
The A-Eye team was founded by Ludovic Saint-Bauzel and Fabien Vérité, researchers at ISIR and lecturers at Sorbonne University. The engineer in charge of the project is Axel Lansiaux, with the help of Aline Baudry and Samuel Hadjes, engineers at ISIR. A number of students from Sorbonne University’s masters programmes, and from the Main and ROB specialisms at Polytech Sorbonne, took part in the project as part of their end-of-study project. This project also involves other colleagues at ISIR, such as Nicolas Baskiotis and Olivier S. (Machine Learning) and Nizar Ouarti (Perception), who are interested in contributing their expertise to this opensource project.
Link to the A-Eye team website: https://a-eye.isir.upmc.fr
RODEO project – Robust deep learning for surgical robotics
The RODEO project is set to reshape surgical robotics by adapting cutting-edge AI. The practical testbed is robotics-guided spine surgery, based on a surgical platform available at ISIR.
In this setup, a 7 DoFs (degrees of freedom) robotic arm is equipped with various sensors (i.e. position, speed, force, electrical conductivity, vibrations) and is used during surgical intervention, e.g., pedicle screw insertion into the spine. This current robotic platform uses a set of already implemented control laws (e.g., position, speed, force control) to execute surgical tasks or subtasks, e.g., drilling a pretrajectory for pedicle screw placement. Before the operation, i.e., “pre-operative”, the patient undergoes a pre-operative 3D CT scan, such that the surgeon defines the medical procedure to follow during the surgery, i.e., “per-operative”.
Although fully automatic controllers can be deployed in safe sub-tasks, surgeons rather rely on the co-manipulation paradigm for sensitive operations, where surgical robots assist medical procedures. In this case, the robotic assistant is expected to faithfully react to the surgeon’s instructions while guaranteeing the safety of both patient and medical staff, and to adapt to the environment.
The context
While the current co-manipulation system [1] is helpful and meets some surgeons’ needs, they could be substantially improved to enhance the surgical experience. The ISIR surgical platform lacks perception and registration modules, and the current procedure assumes that once the patient is positioned for surgery, they does not move and that the spine is rigid. This can make the transfer of pre-operative information complex, imprecise, and dangerous for the patient. Additionally, current controllers fail to represent complex physical phenomena during co-manipulation, such as robot friction (difficult to model because the force applied by the surgeon is unknown beforehand), vibrations (critical for fine-grained manipulation such as peg in a hole), or gravity compensation (crucial for maintaining the robot in a steady position).
Objectives
RODEO aims to develop the next generation of deep generative AIs, which aims at overcoming the main challenges previously mentioned. The project especially aims at designing AI systems with enhanced robustness in terms of flexibility and reliability while remaining sustainable, and hybrid models able to incorporate physical knowledge of the world.
The central research hypothesis is that these improvements can lead to a major breakthrough in surgical robotics. In general, more reliable AI systems can substantially enhance their acceptance among medical experts and patients, especially by giving to AI systems the opportunity to assess their own confidence or explain their decisions in a human-understandable manner. Hybrid and sustainable models have the potential to substantially improve the level of automation in robot/surgeon co-manipulation. This holds the promise of reducing the cognitive load on surgeons, enabling them to concentrate entirely on medical interventions, ultimately elevating surgical procedures and patient care.
The results
In our spine surgery testbed, we expect major improvement at three main fronts:
– Firstly, we want to design hybrid controllers that can exploit AI to learn the residual components in current controllers that are difficult to model, such as friction, gravitation compensation or vibrations. The goal is to augment the level of automation of some tasks and to further assist in other tasks to release the surgeon’s mental load and enhance the accuracy and the safety of the surgical procedure.
– Secondly, we want to increase the flexibility of the system by using multi-modal deep generative AIs to tackle long-term planning based on language. It will include visual perception by developing AI-based methods for registering pre-operative CT-scans to per-operative depth cameras for enabling robust localization and control, e.g., for breathing compensation.
– Last but not least, we will endow AI systems with the ability to quantify their own confidence, and to explain their decision to the surgical team in an understandable manner, ultimately leading to rich human-robot interactions and dialogs.
Partnership and collaboration
The ANR RODEO project is a single-team project led by ISIR’s MLR (Machine Learning and Robotics) project team, headed by Nicolas Thome, a researcher at ISIR and professor at Sorbonne University.
Project “Multi-sensory integration to maintain balance”
The context
Maintaining balance requires the integration of information from different sensory systems: visual, vestibular, proprioceptive and haptic. These different senses are typically studied one by one, which leaves open the question of their integration. The IRIS team at ISIR brings together experts in postural control, haptics and visual-motor adaptation.
Objectives
The aim is to combine this expertise to study multi-sensory integration during balance perturbations, by combining perturbations that are :
– mechanical (via a perturbation platform),
– visual (in virtual reality)
– and haptic (with a ‘light touch’ device and haptic stimulation devices).
Partnerships and collaboration
The “Multi-sensory integration for balance maintenance” project is a federative project, internal to ISIR, which does not involve any collaboration outside the laboratory.
FHF : A Frontal Human Following technology for mobile robots
Robots following humans is an effective and practical function, particularly in the context of service robotics. However, the majority of existing research has focused on the robot following the human from behind, with relatively little attention given to the robot operating in front of the human. This project of frontal following, where the robot remains within the user’s field of view, is more reassuring and facilitates interaction. New challenges will arise when developing a tracker that is able to estimate the user’s pose from 2D LiDAR at knee height, especially when the legs frequently occlude each other. There is also a need to ensure the safety of the user while addressing how the robot can keep up with them in situations where it may fall behind.
The context
Mobile robots are increasingly ubiquitous in various settings, such as shopping centers, hospitals, warehouses, and factories. Many tasks in these applications are shared between robots and human operators through voice, video, force interaction, etc., either because inherent human expertise or agility is required for certain tasks or because the robot can provide potential assistance to the operator. In this project, we focus on investigating the automatic tracking and following of the user by the mobile robot based on 2D LiDAR. Common rear-following algorithms keep the robot at a certain distance from the user. Some studies have shown that people prefer to see robots in their field of vision, and may feel uncomfortable and unsafe when robots appear behind them. Moreover, some specific services require robots to come in front of users. For example, assistance robots act like guide dogs to provide navigation assistance for the visually impaired. Therefore, frontal following is progressively becoming popular and valuable.
Objectives
The scientific objectives of this project are as follows:
– Build a human pose (orientation and position) tracker based on 2D LiDAR at knee height for mobile robots by studying human gait ;
– Collect LiDAR scans from different volunteers and ground truth data on human orientation to build a data-driven model to improve human orientation estimation ;
– Solve the problem of self-occlusion of legs during scanning by modeling gait or using machine learning techniques ;
– Develop a motion generator to enable the robot to move safely and naturally in front of the user.
The results
Twelve volunteers (three females and nine males) were invited to participate in the frontal following experiment, and the human pose tracker based on human gait performed well. The quantitative analyses showed that the mean absolute error (MAE) of position was around 4 cm, and the MAE of orientation was less than 12 degrees for complex walking.
Data collected over six hours on five volunteers (one female and four males) were used to build model(s) to improve orientation estimation. By solving the problem of delay, the customized model achieved an MAE of between 4 and 7 degrees for all five volunteers.
The frontal following motion generator enables the robot to come naturally in front of the user, always at a safe distance of one meter during the experiment. For more details, see the video.
Publications
– 2D LiDAR-Based Human Pose Tracking for a Mobile Robot, ICINCO 2023 ;
– Human Orientation Estimation from 2D Point Clouds Using Deep Neural Networks in Robotic Following Tasks, submitted to IROS 2024 ;
– Large Workspace Frontal Human Following for Mobile Robots Utilizing 2D LiDAR, submitted to JIRS.
TELIN – The Laughing Infant
The TELIN project focuses on developing a socially interactive robotic agent (SIA) capable of replicating the behavior of an infant during laughter acquisition. Its main challenges lie in modeling the robot’s laughter and in real-time decision-making on when to laugh, taking into account the cognitive state of infants, thus surpassing currently assumed cognitive capacities.
To address these challenges, TELIN compiles a vast corpus of recordings of infants laughing in various contexts and develops methods for manual and automatic annotation. The project then analyzes the production of laughter in infants to create a formal model. Based on this model, TELIN develops and evaluates a computational model that enables the robot to decide and generate laughter in real-time during interactions. This initiative requires interdisciplinary collaboration between formal linguistics, artificial intelligence, and audio signal processing.
The context
Laughter, one of the earliest forms of communication in infants, emerges as early as three months of age, well before language, gestures, or walking. Recent studies have highlighted the close link between the acquisition of laughter and advanced cognitive skills, particularly related to understanding negation, offering an intriguing perspective on the evolution of human communication.
The TELIN project draws on a synthesis of diverse research areas including language acquisition, semantics and pragmatics of laughter, Socially Interactive Agents (SIAs), as well as laughter analysis and synthesis, combined with advancements in machine learning. Its aim is to develop an SIA capable of mimicking an infant during laughter acquisition, and to use this SIA to evaluate multiple learning algorithms. These algorithms take into account different input modalities such as audio, facial expression, and language, as well as various contexts such as playing with toys and family interactions, to generate laughter responses.
The project is supported by the Mission for Transversal and Interdisciplinary Initiatives (MITI) of CNRS, which funds interdisciplinary research projects by financing doctoral allocations for a period of three years, coupled with research budgets for the first two years.
Objectives
The thesis topic of the TELIN project is to develop formal and computational models that calculate when and how a robotic infant (using the Furhat robot with a baby mask) responds to human participant expression and activity. The focus is on infant laughter production. This entails:
– Analysis of a corpus of infant laughter,
– Development of a rigorous theoretical analysis of laughter during interaction between a parent and infants,
– Development of a computational model based on deep learning approaches that simulates when laughter should be triggered.
The models will be objectively evaluated as well as through experimental studies.
The results
The integration of language and non-verbal communication is a crucial goal for AI. TELIN advances this field by studying it in a simpler yet ecologically valid environment, regarding natural language understanding, interaction, and world knowledge.
Modeling when laughter should occur in an interaction between humans and AI is still in its infancy. Research within the TELIN framework will address this question and contribute to refocusing efforts in this direction. Additionally, the development of a computer model of a laughing virtual agent (integrated into the Greta platform) will benefit the AI community by providing a new sequence-to-sequence architecture.
Finally, TELIN will provide a platform for a more ecologically valid study of communication development, given the emphasis on multimodal interaction. It will provide detailed empirical and formal reports on the emergence of laughter, a relatively unexplored domain. The SIA platform stemming from TELIN will be available for conducting human-agent studies.
Partnerships and collaborations
The project is led by Université Paris-Cité and also involves:
– ISIR at Sorbonne University,
– and the Sciences and Technologies of Music and Sound (STMS) laboratory at Sorbonne University.
Projet GUIDANCE – “General pUrpose dIalogue-assisted Digital iNformation aCcEss”
GUIDANCE project aims at federating the Information Retrieval (IR) French Community project, by bringing together experts of the field to advance the development of Dialogue-based Information Access (DbIA) models leveraging LLMs.
The aim of the project is to develop new models and resources for interactive information access, e.g. dialoguing with a computer system in order to access (possibly automatically generated) information, while ensuring, on the one hand, adaptation to domains or languages with low resources (compared with English), and on the other hand, the explicability and veracity of the information generated.
The context
The GUIDANCE project takes place in the context of large language models (LLMs) and conversational systems (e.g. ChatGPT, WebGPT), which have experienced tremendous practical progress in the last few months. The project GUIDANCE aims to conduct research on General Purpose Dialogue-assisted Digital Information Access, specifically how to enable users to access digital information, with the goal of overcoming several limitations of current LLMs:
– LLMs were not designed with Information Access – whether at the level of pre-training tasks or fine-tuning ones
– LLMs have limited generalization abilities to new domains and/or languages;
– The veracity and truthfulness of the output are questionable.
– Potentially state-of-the-art LLMs models are not open access and the scientific methodology and proper evaluation are barely described in the scientific literature.
Objectives
From a research perspective, GUIDANCE addresses four challenges associated with this project:
- How to design new LLMs or re-use LLMs for DbIA;
- How to leverage retrieval-Enhanced Machine Learning (ReML) techniques to improve the accuracy and efficiency of information retrieval systems;
- Adapt LLMs and develop new architectures (for DbIA models) to deal with low resource and domain adaptation – with special attention paid to the low/medium-resource languages (e.g. Occitan, French);
- Design DbIA models that can ensure the veracity and explainability of retrieved and synthesized information, while preserving the user’s subjectivity.
The results
The GUIDANCE project is expected to deliver multiple results, paving the way for significant advances in the field of information access.
Firstly, the development of resources for training information access models (made available to the community). These are learning corpora that can be used to train new, more powerful models.
Secondly, the project aims to develop new modes of interaction with information access systems: a search engine can be pro-active in guiding the user towards relevant results (much more than by proposing related questions, as is currently the case).
Finally, the provision of pre-trained models for information access, which will enable these interactive models to be used freely, whether for research or other purposes.
Partnerships and collaborations
Led by Benjamin Piwowarski, CNRS research fellow at ISIR (MLIA team), the GUIDANCE project also involves:
– the Institute of Research in Computer Science of Toulouse (IRIT) through the two IRIS and SIG research teams,
– the Grenoble Computer Science Laboratory (LIG) through the APTIKAL and MRIM research teams,
– and the Computer Science and Systems Laboratory (LIS) through the R2I research team.
The project, which began in October 2023, brings together 18 researchers from 6 research groups in Information Retrieval (IR) and Natural Language Processing (NLP).
NeuroHCI project – Multi-scale decision making with interactive systems
This multi-disciplinary project uses Computational Neuroscience to develop HMI models of user behavior. The aim is to study the extent to which Computational Neuroscience theories, models and methods can be transposed to HMI.
The NeuroHCI project aims to improve human decision-making in the physical and digital worlds in interactive contexts. The situations in which a human makes a decision with an interactive system are varied:
Do I use my experience or Google Maps to choose my route? Do I reply to this e-mail on my smartphone or PC? Do I use menus or shortcuts to select this frequent command? Do I use the Da Vinci surgical robot to operate on my patient, or traditional laparoscopic instruments? How can I reach this object with my robotic prosthesis?
The decision may concern a complex computer-aided real-world choice (e.g., a medical treatment) or the choice of a method for carrying out a digital task (e.g., editing a photo with the preferred tool).
The context
Neuroscience studies phenomena involving both decision-making and learning in humans, but has received little attention in HCI.
The NeuroHCI project is a human-computer interaction (HCI) project that aims to design interactive systems that develop user expertise by establishing a human-machine partnership. Interaction with these systems can be seen as a multi-scale decision-making problem:
– A task, e.g. choosing the right medical treatment on the basis of AI-based recommendations;
– A method, e.g. choosing between different devices or modalities to perform a task;
– An object, e.g. which physical or virtual object users will interact with;
– A movement, e.g. which trajectory to take to reach the target object.
Objectives
The scientific objective is to understand how users make decisions with interactive systems, and how these decisions evolve over time. Indeed, users gradually develop expertise over the course of repeated use of interactive systems. This expertise influences the way users make decisions. This requires the simultaneous study of the learning and decision-making phenomena underlying the use of interactive systems.
The application goal is to design and implement better interactive and adaptive systems. Human beings adapt and develop their expertise by using an interactive system. The aim here is for the system, for its part, to also evolve to adapt to its users, i.e. to become accustomed to user behavior and in particular to their expertise. The aim is to establish a man-machine partnership in which both actors (human and machine) adapt to each other.
The results
To achieve these objectives, we demonstrate the benefits of our approach through 3 applications, for which platforms already exist and are maintained by the partners, but where scientific challenges remain for their adoption in the real world. These three applications are:
– intelligent graphical interfaces such as AI-based recommendation systems;
– immersive simulation systems offering rich haptic feedback;
– and medical cobotic interfaces, which aim to restore or enhance the ability of humans to interact with objects in the real world.
Our research hypothesis is that it is necessary to develop robust computational models of learning and decision-making in HMIs. Computational models make it possible to explain and predict human behavior by synthesizing complex phenomena in a testable and refutable way. In HMI, they can be used to assess the quality of an interface without the need for costly and time-consuming user studies. When these models are robust, they can be integrated into interactive systems to optimize interaction and adapt the interface to users’ expertise and/or actions.
Partnerships and collaborations
Led by Gilles Bailly, CNRS Research Director at ISIR, the ANR NeuroHCI project is a multi-team project at ISIR, involving several members of the laboratory.
