In an era where remote interactions are becoming increasingly common, driven by the rise of digital technologies, virtual reality, and periods of social distancing, a key question remains: how can we maintain emotional connection without physical contact? Social touch, which plays a crucial role in psychological and physiological well-being, is still difficult to replicate at a distance. Its absence can intensify feelings of isolation, anxiety, and the fundamental human need for connection.
This is the challenge addressed by the Audio-touch project, a research initiative led by ISIR. It builds on an original idea: to exploit the physical links between touch and hearing to translate affective tactile gestures into sound. In other words, to make touch audible, to help people perceive it remotely.
The project is the result of a collaboration between Alexandra de Lagarde, PhD candidate at ISIR; Catherine Pelachaud, CNRS research director at ISIR; Malika Auvray, CNRS research director at ISIR; and Louise P. Kirsch, associate professor at Université Paris Cité and former postdoctoral researcher at ISIR. The project is also part of the ANR MATCH program, which explores the perception of touch in interactions with virtual agents.
What if you could hear a caress?
The Audio-touch project stems from a simple observation: touch and hearing share a common physical foundation, vibrations. Research in sonification has already shown that it is possible to translate movements and their properties into sounds. The project takes this further by asking whether social touch gestures, such as a caress, a tap, or a press, can be converted into meaningful auditory signals.
The project has two main objectives: to study the technical feasibility of converting skin-to-skin touch into sound, and to experimentally determine which aspects of social touch the audio-touch stimuli can convey to listeners. This work follows an experimental approach at the intersection of human-agent interactions, social cognition, haptics, and acoustics.
Four experiments to test the sound perception of touch
At the heart of the project is a series of four experiments designed to assess how well these gestures and the emotions they carry can be recognized through sound:
– Participants were able to correctly categorize the gestures (Experiment 1);
– Participants successfully identified the emotions conveyed (Experiment 2);
– Their recognition was influenced by the surface on which the touch occurred (Experiments 3 and 4).
Participants who listened to the audio-touch skin-to-skin stimuli were able to accurately associate the sounds with specific gestures and consistently identify the underlying socio-emotional intentions of the tactile interactions. Moreover, the same movements were recorded on inanimate objects, and further experiments revealed that participants’ perception was influenced by the nature of the surfaces involved (skin vs. plastic).
These are significant results: they show that social tactile gestures, their emotional intent, and the touched surface can be recognized from the sounds they produce.
Toward new forms of multisensory communication
This project aligns with ISIR’s research themes and complements the work of the ANR MATCH project, which aims to simulate the illusion of touch in virtual environments.
The Audio-touch approach contributes to our understanding of multisensory strategies by providing empirical data on how touch can be perceived through sound. It informs the design of immersive virtual agents capable of fostering a sense of social connection through a subtle combination of sensory cues.
These findings have been published in the journal Proceedings of the National Academy of Sciences (PNAS) under the title: «Paving the way for social touch at a distance: Sonifying tactile interactions and their underlying emotions ». By showing that affective gestures can be recognized through sound alone, this research opens up promising new avenues for developing multisensory communication methods, especially in remote or virtual agent interactions.
Scientific contacts at ISIR: Malika Auvray, CNRS Research Director, Alexandra de Lagarde, PhD Student, and Catherine Pelachaud, CNRS Research Director.
Published on 23/05/2025.
