Opportunities

Job title: Post-doc on haptic and multisensory human-computer interactions

Context:

This post-doc is part of the NeuroHCI ANR project. The overall goal of NeuroHCI is to improve human decision making in the physical and digital worlds in interac,ve contexts. There are various scenarios in which a human makes a decision with an interac,ve system. The decision might be about a complex real-world choice assisted by a computer (e.g. medical treatment), the choice of a method to achieve a digital task (e.g. editng a photo with the preferred tool), or the way we decide the best way to perform a haptic interaction.

Missions:

The envisioned scientific approach will rely on optimizing the haptic feedback delivered to the user with relation to vision and hearing by leveraging computational models of multisensory integration. Thus, the scientific activities of the project will revolve around questions including but not limited to:

– How to ensure that the inconsistencies between what the user sees and what the user feels does not break the illusion and how to mitigate their effects on user experience?

– How visuo-haptic inconsistencies influence users’ strategies (e.g. which objects they will decide to interact with) and high-level decision making?

Required profile:

The ideal candidate must have a PhD degree and a strong background in human-computer interaction and/or cognitive science.

Required skills:

– Experience in haptics is a strong plus ;

– Strong skills in Python, Matlab or equivalent ;

– Good knowledge of experimental design, psychophysics and statistics ;

– Excellent publication record ;

– Willingness to work in a multi-disciplinary team ;

– Good communication skills.

General Information:

• Contract start date: at the latest 06/01/2024
• Contract duration: 24 months
• Working time: 100%
• Desired experience: beginner at 4 years old
• Level of studies required: PhD
• Host laboratory: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

How to apply:

– David Gueorguiev ; david.gueorguiev(at)sorbonne-universite.fr

– Send your application by email, with [name of offer] in the subject line, a CV and a cover letter.

Download this offer

Post-doc : Learning in robotics, with application to grasping

Context:

During the FET Proactive DREAM project (http://dream.isir.upmc.fr/) has been defined an approach for adaptive robotics based on open-ended learning. The main goal is to allow a robot to learn without requiring a careful preparation by an expert. This approach raises many challenges, notably learning with sparse reward, representation learning (for states and actions), model learning and exploitation, transfer learning, meta-learning and generalization. These topics are considered in simulation, but also on real robotics setup, notably in the context of grasping.

Missions:

This position aims at contributing to these topics in the context of several European projects, in particular SoftManBot, Corsmal, INDEX and Learn2Grasp. Calling upon previous works in the research team, the proposed approaches need to be easy to adapt to different robotic platforms and will thus be applied to different robots (Panda arm from Franka-Emika, Baxter, PR2 or TIAGO, for instance).

Required profile:

Candidates for the position must have a PhD degree in machine learning or related field in which robotics applications (either simulated or real) have been considered.

Required skills:

An excellent background is expected in machine learning as well as an experience in robotics. Excellent programming skills in Python are expected.

General Information: 

• Position Type: Post-doctoral researcher
• Contract duration: 24 months
• Level of education required: PhD
• Remuneration : Remuneration according to experience
• Location: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

Contact person: 

• Stephane Doncieux
• stephane.doncieux(at)sorbonne-universite.fr
• Send your application by email, with a CV and a cover letter.

Download this offer

Doctoral, Postdoctoral or research engineer positions for the HCI Sorbonne group (Human Computer Interaction)

Context:

We have multiple postdoctoral and engineering positions in the HCI group (https://hci.isir.upmc.fr) at Sorbonne Université, Paris, France.

Missions:

We are searching for curious minds who want to conduct cutting-edge research at the intersection of HCI with VR, Haptics, Robotics or AI. Possible topics/areas of research are:

• Novel Interaction techniques in VR,
• VR and haptics for gaming or training,
• Computational models for learning, decision making and human performance,
• AI-based recommendation systems.

Some of our previous work in these areas:

• CoVR (UIST21): https://dl.acm.org/doi/10.1145/3379337.3415891
• AI-based Recommendation systems (CSCW 21): https://dl.acm.org/doi/abs/10.1145/3476068?sid=SCITRUS
• Adapting UIs with Reinforcement Learning (CHI 21): https://dl.acm.org/doi/abs/10.1145/3411764.3445497
• Mixed Control of Robotic systems (CHI 20): https://dl.acm.org/doi/10.1145/3313831.3376795

Required profile:

For the postdoctoral position, a Phd degree in Computer science, HCI or other field related to our research areas is required.

Required skills:

• strong programming and analytical skills,
• strong background in at least one of the following areas (HCI, VR, Haptics, Robotics, AI).

More information : 

• Type of position: Postdoctoral or Research Engineer position
• Start date: as soon as possible
• Duration: 1 to 2 years
• Level of study required: Master 2 (for engineer), PhD (for post-doc)
• Location: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

Contacts: 

• Gilles Bailly et Sinan Haliyo
• Email : gilles.bailly(at)sorbonne-universite.fr ; sinan.haliyo(at)sorbonne-universite.fr
• Application: Send your application by email, with a CV and a cover letter.
• Application deadline: None

Download the offer

Thesis title: “Foundation Models for Physics-Aware Deep Learning”

Context:

Physics-aware deep learning is an emerging research field aiming at investigating the potential of AI methods to advance scientific research for the modeling of complex natural phenomena. This is a fast-growing research topic with the potential to boost scientific progress and to change the way we develop research in a whole range of scientific domains. An area where this idea raises high hopes is the modeling of complex dynamics characterizing natural phenomena occurring in domains as diverse as climate science, earth science, biology, fluid dynamics. A diversity of approaches is being developed including data-driven techniques, methods that leverage first principles (physics) prior knowledge coupled with machine learning, neural solvers that directly solve differential equations. Despite significant advances, this remains an emerging topic that raises several open problems in machine learning and application domains. Among all the exploratory research directions, the idea of developing foundation models for learning from multiple physics is emerging as one of the fundamental challenges in this field. This PhD proposal is aimed at exploring different aspects of this new challenging topic.

Research Directions:

Foundation models have become prominent in domains like natural language processing (GPT, Llama, Mistral, etc) or vision (CLIP, DALL-E, Flamingo, etc). Trained with large quantities of data using self-supervision, they may be used or adapted for downstream tasks through pre-training from large amounts of training data. Initial attempts at replicating this framework in scientific domains is currently being investigated in fields as diverse as protein (Jumper et al. 2021), molecule (Zhou 2023), weather forecasting (Pathak 2022, Nguyen 2023, Kochkov 2024). Is the paradigm of foundation models adaptable to more general physics modeling such as the complex behavior of dynamical systems? Large initiatives are emerging on this fundamental topic (https://iaifi.org/generative-ai-workshop). Some preliminary attempts are currently being developed (McCabe 2023, Subramanian 2023, Hao 2024). They suggest that learning from multiple steady-state or time dependent partial differential equations (PDEs) could enhance the prediction performance on individual equations. This high stake, high gain setting might be the next big move in the domain of data-driven PDE modeling. The objective of the PhD is to explore different directions pertaining to the topic of foundation models for physics, focused on the modeling of dynamical systems.

– Solving parametric PDEs:

A first step is to consider solving parametric partial differential equations (PDEs), i.e. PDEs from one family with varying parameters including initial and boundary conditions, forcing functions, or coefficients. It is possible that different parameters values, give rise to very different dynamics. Current neural solvers operate either on fixed conditions or on a small range of parameters with training performed on a sample of the parameters. A first direction will be to analyze the potential of representative NN solvers to interpolate and extrapolate out of distribution to a large range of conditions when learning parametric solutions. A key issue is then the development of training techniques allowing for fast adaptation on new dynamics. We will investigate methods inspired from meta-learning for adaptive strategies (Yin 2021, Kirchmeyer 2022).

– Tackling multiple physics:

The foundation approach is particularly interesting in the case of scarce data, provided physics primitive could be learned from related but different PDE dynamics that are available in large quantities and then transferred to the case of interest. Learning from multiple PDEs raises algorithmic challenges since they operate on domains with different space and time resolutions, shapes and number of channels. We will consider an Encode-Process-Decode framework so that the commonalities between the dynamics are encoded and modeled in a shared latent space and the encoding-decoding process allows to project from and to the observation space for each PDE. As for the temporal variability of the observations, one will consider models that can operate on irregular series in the spirit of (Yin2023). This framework will be evaluated with selected backbones.

– Generalization and few shot capabilities:

Generalization to new dynamics is the core problem motivating the development of foundation models in science. This is a key issue for the adoption of data-driven methods in physics and more generally in any context were the data is scarce. We will consider the general framework of few shot learning aiming at fine tuning pre-trained models for downstream tasks. In this context the objective will be to develop frameworks for the fast adaptation of foundation models to target tasks. Different strategies will be analyzed and developed including parameters sampling, meta-learning for adaptation (Yin 2023) and strategies inspired from the developments in semantics and language applications like in-context learning (Chen 2024).

Position and Working Environment:

The PhD studentship is a three years position starting in October/November 2024. It does not include teaching obligation, but it is possible to engage if desired. The PhD candidate will work at Sorbonne Université (S.U.), Pierre et Marie Campus in the center of Paris. He/She will integrate while benefiting the MLIA team (Machine Learning and Deep Learning for Information Access) at ISIR (Institut des Systèmes Intelligents et de Robotique). MLIA is collaborating with fellow scientists from other disciplines such as climate or fluid mechanics. The PhD candidate will be encouraged to get involved in such collaborations.

Required Profile:

Master degree in computer science or applied mathematics, Engineering school. Background and experience in machine learning. Good technical skills in programming.

General information:

– Supervisor: Patrick Gallinari, patrick.gallinari@sorbonne-universite.fr

– Collaboration for the thesis: Cerfacs Toulouse, Institut d’Alembert, Sorbonne Université, CNAM Paris

– Host laboratory: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

– Start date: October/November 2024

– Note: The research topic is open and depending on the candidate profile could be oriented more on the theory or on the application side

– Keywords: deep learning, physics-aware deep learning, fluid dynamics, AI4Science

Contact person:

– Patrick Gallinari

– Email: patrick.gallinari@sorbonne-universite.fr

– Please send a cv, motivation letter, grades obtained in master, recommendation letters when possible to patrick.gallinari@sorbonne-universite.fr

– Application deadline: 15/12/2024

Donwload this offer

Internship topic: Learning communication in action models for human- robot collaboration

Context:

When humans demonstrate tasks, they integrate instrumental actions (manipulating objects) with ostensive communicative cues (e.g., eye gaze, pauses, and exaggeration). These belief-directed actions are intentional, aimed at conveying information beyond the immediate task. This concept, termed communication in action, has been widely studied in Cognitive Science as a key mechanism for establishing shared understanding between actors and observers [Ho-2022].

In robotics, communication in action (Figure 1) is mirrored by the challenge of generating legible robot motion [Dragan-2013,Wallkötter-2022]. Legible motions combine instrumental actions (achieving goals) with belief- directed cues to make the robot’s intent clear to human observers. This often involves exaggerated movements or gaze cues, enabling humans to infer the robot’s goal accurately [Wallkötter-2021]. Computational approaches in Human-Robot Interaction (HRI) typically rely on mathematical models of human expectations to design such motions. However, most current approaches assume a literal interpretation of human and robot actors, leading to limitations in understanding and responding to each other’s actions and intentions. These limitations highlight the need for forward and inverse models capable of encoding and decoding both instrumental and belief- directed intentions, enabling mutual comprehension in human-robot collaborations.

OSTENSIVE is a recent ANR project starting in 2025, coordinated by ISIR and involving CNRS LAAS and INRIA LARSEN. This project aims to advance HRI by developing human-centric interaction systems that integrate instrumental and belief-directed dimensions by exploiting recent machine learning approaches. A PhD project is planned to be proposed as a follow-up to the internship.

Objectives:

This internship focuses on the development of machine learning models that enable robots to act based on double goals: achieving tasks (instrumental actions) and communicating intent (belief-directed actions). For this purpose, we will exploit behavior cloning approaches based on variational auto-encoders and/or diffusion policies [Lee-2024; Liang-2024; Caselles-2022].

The specific objectives are:

– Develop forward and inverse models to encode and decode instrumental and belief-directed actions.

– Design machine learning algorithms to integrate these dimensions into robot behavior.

– Test these models in tasks where robots are instructed through combined instrumental and belief-based goals.

Profile:

– Level: Master 2 / Engineer school

– Skills: Python, Machine Learning, Robotics, and Cognitive Science

– Duration: 5-6 months

Contact:

– Mohamed Chetouani

– ISIR – CNRS UMR7222 Sorbonne Université

– mohamed.chetouani@sorbonne-universite.fr

Download this offer

Internship topic: Multimodal and Privacy-Preserving Machine Learning for Human-Behavior Analysis

Context:

Understanding human behavior is critical for developing intuitive, personalized AI systems, particularly in healthcare [Driss-2020]. However, analyzing human behavior involves sensitive data (e.g., audio, video, wearable sensors) that pose privacy challenges, especially in mental healthcare applications like stress detection and mood recognition [Aigrain-2018]. Current machine learning approaches often rely on centralized data collection, leading to limited applicability across tasks and raising ethical concerns [Guerra-Manzanares-2023]. Privacy regulations such as GDPR and the proposed AI Act emphasize the need for transparent, privacy-compliant solutions. This calls for innovative methods that balance privacy protection with the utility of predictive models while supporting diverse tasks through transferable multimodal representations [Zhao-2020].

Objectives:

This internship focuses on advancing privacy-preserving multimodal machine learning for human behavior analysis. The research will explore methods such as adversarial training, deep auto-encoders, and multimodal foundation models, aiming to achieve robust privacy-utility trade-offs and ethical AI deployment in healthcare [Guerra-Manzanares-2023]. A possible research direction will consider the generation of synthetic data using machine learning.

The specific objectives include:

– Develop generative AI models to learn intrinsic multimodal representations of human behavior while preserving privacy.

– Generate synthetic, privacy-compliant data that maintains utility for predictive tasks.

– Evaluate models on publicly available datasets (e.g., audio, video, and physiological data) for tasks like mental state detection and diagnosis.

This thesis is conducted within a collaborative effort between ISIR (SU) and TICLab (International University of Rabat, IUR, Morocco), which is a strategic partner of Sorbonne University.

Profile:

– Level: Master 2 / Engineer school

– Skills: Python, Machine Learning, Robotics, and Cognitive Science

– Duration: 5-6 months

Contact:

– Mohamed Chetouani

– ISIR – CNRS UMR7222 Sorbonne Université

– mohamed.chetouani@sorbonne-universite.fr

Download this offer

Internship topic: Engineering internship for assistive technology for the visually impaired

Presentation:

The A-Eye team, who took part in the Cybathlon competition with the attached device, would like to move forward and work on a more open-source (more available to people) and more compact solution to the device.

So there are two avenues to explore to improve the device. Firstly, we would like to develop a guidance solution that would be positioned at shoulder level, rather than one that is attached to a harness with elements all around the body, as shown in the photo.

The candidate’s mission will be to develop a mechatronic solution for effort feedback integrated into a kind of rigid collar that could be positioned on the neck. This sensory feedback will consist of a slight stretching of the skin around the neck. The system built will need to provide space to integrate a computer, a 3d camera, batteries and skin- level feedback. The computer and camera are those already in use and functional. This project will have to work with ROS2 to enable it to interface with software solutions already developed in the lab.

The aim of this internship is to develop a prototype sensory feedback system that integrates the elements of the current device (computer, battery, Camera3D) in a more compact way. The system will need to be able to interface with existing software, so we’ll need to develop the device’s driver layer up to the ros2 node to interface with our codes.

Skills :

–  Python

–  CAD (Solidworks, Fusion360)

–  Embedded system (Raspberry pi, Linux)

–  Electronics (Arduino)

–  ROS2 could be an asset

Additional information:

– Supervisors : Ludovic Saint-Bauzel

– Starting date: February 2025

– Duration of internship: 6 months

– Host laboratory: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

Contact person:

– Ludovic Saint-Bauzel

– Email : saintbauzel@isir.upmc.fr

– Send your application by email, with [internship subject] in the subject line, a CV and a covering letter.

Download this offer

Internship topic: Minimizing expected cumulative cost of actions during interaction by solving the 20 questions game

Subject:

The 20 questions game is a guessing game where a questionner has 20 questions to determine the object an answerer is thinking of. On the other hand, many forms of interaction are about conveying a user’s goal to the computer, e.g., locating the right file, or selecting the right widget. There exists a connection between taking actions to identify a goal and the 20 questions game. In this internship, we try to reinforce this link, with the promise that we can then exploit known algorithms to optimally play the 20 questions game to enhance interaction.

Internship Objectives:

The game of 20 questions has many variants, depending on the potential questions that may be asked (is any question admissible? Are only comparison questions, that partition the ordered set in two allowed? Are only questions of restricted size allowed?), and whether the answerer is considered to make mistakes/lie or not. We have preliminary results that establish the equivalence between weight balancing in alphabetic trees for comparison questions and a known interaction technique known as Bayesian Information Gain (BIG) when the user makes no mistakes.

The tasks of the intern will be to:

• based on a review of existing interactions, list all type of possible question types (comparison, interval, arbitrary etc.), and determine the state of the art optimal strategy for the questionner.

• adapt the existing algorithms to minimize either the expected number of actions or the expected cumulative cost of actions.

• make a link with BIG whenever possible

• consider the case when the user makes errors, which includes work on user modeling

• implement a demonstrator for one question type e.g., comparison questions

• conduct a controlled experiment to show the objective (minimizing actions, minimizing interaction time) has indeed been achieved

Depending on the candidate’s profile, the work may focus more on the theoretical side or on the side of conducting experiments.

Required Profile:

We are looking for an M2 intern in computer science or a related field interested in working on applied and theoretical topics, as this internship requires both. Knowledge of source / channel coding, and theoretical computer science will be an advantage; so will be an interest in computationally modeling human behavior. This topic could be pursued with a PhD, for which we have already secured funding.

Additional information:

– Supervisors : Julien Gori

– Starting date: January -March 2025

– Duration of internship: 5-6 months

– Level of studies required: Master 2

– Host laboratory: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

Contact person: 

– Julien Gori

– Email : gori@isir.upmc.fr

– Send your application by email, with [internship subject] in the subject line, a CV and a covering letter.

Download this offer

Internship topic: TargetFinder – Developing an External Access System for Widget Information in Graphical User Interfaces

Subject:

In most graphical applications, users interact with various elements — like buttons, menus, and title bars — called widgets. Information about these widgets (such as their size, location, and labels) is typically accessible only within the application itself. The objective of this internship is to develop methods to access this information from outside the application, enabling a broader range of uses, such as cross-application automation, user interface analysis, and accessibility enhancements.

Internship Objectives:

This project is part of an ongoing collaboration between Sorbonne Université and Université de Lille. During this internship, the student will work on designing a system that can gather widget information from graphical user interfaces (GUIs) externally. The student will build on an existing demonstrator that can identify a single target’s location and size. The demonstrator builds on a YOLOv8 pre-trained object detection network. The first goal of this project is to improve the demonstrator to identify multiple target locations and sizes at once, as well as determine their labels. To do so, the intern will apply existing image processing techniques to pre-label an existing dataset, which will, after verification, be used to retrain the existing network. The second goal of this project is to implement an existing interaction technique that leverages target information, for example semantic pointing, to showcase the possibilities offered by this project.

Required Profile:

We are looking for a person with a background in computer science, with skills in machine learning / computer vision / image processing. Knowledge of Python is required, knowledge of C++ is not needed but is appreciated.

Additional information:

– Supervisors : Julien Gori

– Starting date: January – March 2025

– Duration of internship: 5-6 months

– Level of studies required: Master 2

– Host laboratory: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

Contact person:

– Julien Gori

– Email : gori@isir.upmc.fr

– Send your application by email, with [internship subject] in the subject line, a CV and a covering letter.

Download this offer

Internship subject: Modeling the contribution of thalamo-cortical circuits to individual differences in behavioral flexibility

The ability to generate flexible behavioural responses is crucial for survival in complex and dynamic environments. Within a population, behavioural output is typically quite variable, leading to individual choices with differential adaptive values. Understanding the neural bases of these specific behavioural traits is currently a growing issue as it may be a key element to better understand the trajectories that may lead to pathological states.

While past research has largely considered the role of highly evolved brain regions such as the prefrontal cortex, the importance of subcortical regions has been increasingly recognized over the past few years. This is especially true for the mediodorsal thalamus (MD) which has extensive and multiple reciprocal connections with prefrontal areas and especially the orbitofrontal cortex (OFC), a well-known key hub for flexible behaviours, making it an important hub for executive functions. Functional dysconnectivity within thalamocortical circuits is associated with many conditions and neuropsychiatric disorders such as Schizophrenia, obsessive-compulsive disorder, ADHD or addiction. But the mechanisms by which these circuits may contribute to behavioral flexibility are still largely unknown.

In this project, we hypothesize that MD-OFC circuits may constitute a key element for understanding the neural underpinnings of variable behavioral output ranging from adaptive to maladaptive decision-making. Our preliminary dataset and model suggest that the MD->OFC functional connection is critical to support efficient flexible behaviour. We thus hypothesize that inter-individual variability in the learning strategy employed depends on the  individual functional endophenotype of this pathway.

In this work, we will first model experimental data collected by our collaborators at the CNRS INCIA in Bordeaux where rats learn to choose between different levers that have different reward probabilities, different uncertainty levels, and subject to abrupt task changes. At a second stage, we will derive theoretical predictions from this modeling work in order to prepare for the new experiments that our collaborators will perform during a new ANR-funded project starting in October 2024. This ANR project also includes PhD funding at ISIR which could start in October 2025 in extension of the present internship.

The current task includes unsignalled abrupt changes to which the animal has to adapt, requiring a constant exploration-exploitation trade-off. The present version of the task is an extension of a task where we have previously shown that dopamine blockade impairs the exploration-exploitation trade-off in rats. Here we will compare an OFC-lesioned group and an MD-lesioned group with a control group, all alternatively facing difficult versus easy task conditions, where the contrast between levers’ reward probability is manipulated to make the best option more or less easy to find.

We predict that OFC lesions will impair rat performance only in the difficult condition, where subcortical structures involved in reward-based learning may be insufficient to learn the task. We will develop alternative computational models which may explain these behavior impairments through the manipulation of different model parameters. We will then simulate these models to verify that they can reproduce rat behavior, and fit them to the experimental data to find the best model. We will then evaluate whether significant model parameters change explain the data. We will finally perform model simulations in novel extensions of the task in order to derive theoretical predictions which could drive future experiments.

Job description:

– Supervisor: Mehdi Khamassi

– Starting date: 01/10/2024

– Duration: 1d/week during S1, full-time during S2 until 30/06/2025

– Level of studies required: M1/engineering

– Host laboratory: ISIR (Institut des Systèmes Intelligents et de Robotique), Campus Pierre et Marie Curie, 4 place Jussieu, 75005 Paris.

Contact person:

– Mehdi Khamassi

– Email : mehdi.khamassi@sorbonne-universite.fr

– Send your application by email, with [internship subject] in the subject line, a CV and a covering letter.

– Application deadline: 30/09/2024

Download this internship offer