Institut des Systèmes Intelligents
et de Robotique

Partenariats

UPMC

CNRS

INSERM

Tremplin CARNOT Interfaces

Labex SMART

Rechercher

Researches

khamassi Mehdi
Title : Researcher
Address : 4 place Jussieu, CC 173, 75252 Paris cedex 05
Phone : +33 (0) 1 44 27 28 85
Email : khamassi(at)isir.upmc.fr
Group : AMAC (AMAC)

 

Research topics

 


Cognitive bio-inspired humanoid robotics

 

 

A neuromimetic model was developped to describe neural mechanisms in the prefrontal cortex for decision-making and reinforcement learning. The model mimicks the way the prefrontal cortex uses reward information to update values associated to different possible actions, and to regulate exploration during decision-making (i.e. sometimes exploiting learned action values, and sometimes exploring by choosing suboptimal actions so as to gather new information).
Here the model is applied to a simple human-robot game. The robot has to find under which cube a star (reward) is hidden. The robot alternates between exploration (searching for the correct cube) and exploitation phases (repeating the same choice). In addition, the model mimicks the way the prefrontal cortex monitores performance and can associate some cues or task events to variations in performance. This enables the robot to learn by itself to recognize that some objects or events are associated with changes in the task and thus shall be followed by a re-exploration.

Work done in collaboration with Peter Ford Dominey (INSERM U846, Lyon), Emmanuel Procyk (INSERM U846, Lyon), Stéphane Lallée (INSERM U846, Lyon) and Pierre Enel (INSERM U846, Lyon).

 

Related publications

  • Khamassi, M. and Lallée, S. and Enel, P. and Procyk, E. and Dominey, P.F. (2011). Robot cognitive control with a neurophysiologically inspired reinforcement learning model.
    Frontiers in Neurorobotics. Vol 5:1 Pages 1-14.
  • Khamassi, M. and Enel, P. and Dominey P.F. and Procyk, E. (in press). Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters.
    Progress in Brain Research. Vol 202 Pages 441-464. Pammi, V.S.C. and Srinivasan, N. (Eds.) Decision Making: Neural and Behavioral Approaches.

 

 


Neurorobotic models of action selection and spatial navigation

 

 

This work aimed at proposing a computational model for the coordination of navigation strategies in rodents and their associated learning mechanisms. The model was funded on experimental evidence showing that mammals are able to alternate between strategies relying on a mental ("cognitive") map of the environment and cue-guided or response strategies. In particular, depending on the current uncertainty, stability and familiarity of the environment, rats are able to choose the most appropriate strategy for a given task.

The proposed computational model comprised a new formal hypothesis for the interaction between brain systems during navigation (Dollé et al., 2008; 2010; submitted). A hippocampal module containing "place cells" projected to prefrontal cortical cortical columns representing nodes of a topological map of the environment. Planning within this map resulted in suggested paths which were sent to a striatal module selecting movements according to the planning strategy. This module was in competition with two other striatal modules: one for a cue-guided learning through reinforcement to associate visual cues with directions of movements; one for an exploration strategy proposing random movements. Finally, a specific cortico-striatal modules called the "gating network" was dedicated to the selection of the strategy which should guide behavior at which moment. This gating network learns with reinforcement learning which strategy is the most efficient in each context. The model was used in simulation to reproduce a series of existing rat behavioral data in navigation mazes.

The second part of the work consisted in testing the robustness of the model on a robotic platform with noisy and unpredictable interactions with the real-world (Caluwaerts et al., 2012a,b). During exploration, the Psikharpax robot builds a mental ("cognitive") map of the environment by observing the configuration of visual cues and integrating odometry measurements. After exploration, the robot can reach any goal location. Efficient trajectories emerge out of the coordination of 2 behavioral strategies :
(1) planning (using the map)
(2) taxon (cue-guided).
 

Related publications

  • Dollé, L. and Khamassi, M. and Girard, B. and Guillot, A. and Chavarriaga, R. (2008). Analyzing interactions between navigation strategies using a computational model of action selection.
    Spatial Cognition VI. Learning, Reasoning, and Talking about Space, Springer, publisher. Pages 71-86.
  • Dollé, L. and Sheynikhovich,D. and Girard,B. and Chavarriaga,R. and Guillot,A. (2010). Path planning versus cue responding: a bioinspired model of switching between navigation strategies.
    Biological Cybernetics. Vol 103 No 4 Pages 299-317.
  • Dollé, L., Chavarriaga, R., Khamassi, M., Guillot, A. (submitted). Interactions of spatial strategies producing generalization gradient and blocking: a computational approach.
  • Caluwaerts, K. and Staffa, M. and N'Guyen, S. and Grand, C. and Dollé, L. and Favre-Felix, A. and Girard, B. and Khamassi, M. (2012). A biologically inspired meta-control navigation system for the Psikharpax rat robot.
    Bioinspiration & Biomimetics. Vol 7(2):025009 Pages 1-29.
  • Caluwaerts, K. and Favre-Felix, A. and Staffa, M. and N'Guyen, S. and Grand, C. and Girard, B. and Khamassi, M. (2012). Neuro-inspired navigation strategies shifting for robots: Integration of a multiple landmark taxon strategy. Living Machines 2012, Lecture Notes in Artificial Intelligence, Prescott, T.J. et al. (Eds.). Vol 7375/2012 Pages 62-73. Barcelona, Spain.

 

 


Reinforcement Learning and Meta-Learning

 

Related publications

  • Khamassi, M. and Lachèze, L. and Girard, B. and Berthoz, A. and Guillot, A. (2005). Actor-Critic Models of Reinforcement Learning in the Basal Ganglia: From Natural to Artificial Rats.
    Adaptive Behavior. Vol 13 No 2 Pages 131-148.
  • Khamassi, M. and Martinet, L.-E. and Guillot, A. (2006). Combining Self-Organizing Maps with Mixture of Experts: Application to an Actor-Critic Model of Reinforcement Learning in the Basal Ganglia.
    From Animals to Animats 9 (SAB 2006), Berlin, Heidelberg: Springer-Verlag, publisher. Pages 394-405.
  • Khamassi, M. and Quilodran, R. and Enel, P. and Procyk, E. and Dominey, P.F. (2010). A model of integration between reinforcement learning and task monitoring in the prefrontal cortex.
    From animals to animats: Proceedings of the Eleventh International Conference on Simulation of Adaptive Behavior (SAB2010), Springer Verlag LNAI 6226, publisher. Pages 424-434.
  • Khamassi, M. and Wilson, C. and Rothé, R. and Quilodran, R. and Dominey, P.F. and Procyk, E. (2011). Meta-learning, cognitive control, and physiological interactions between medial and lateral prefrontal cortex.
    Neural Basis of Motivational and Cognitive Control, Cambridge, MA: MIT Press, publisher. Pages 351-370.
  • Khamassi, M. and Lallée, S. and Enel, P. and Procyk, E. and Dominey, P.F. (2011). Robot cognitive control with a neurophysiologically inspired reinforcement learning model.
    Frontiers in Neurorobotics. Vol 5:1 Pages 1-14.
  • Khamassi, M. and Humphries, M. D. (2012). Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies.
    Frontiers in Behavioral Neuroscience. Vol 6:79 Pages 1-19.

 

 


Neurophysiology of the prefrontal cortex, striatum and dopaminergic neuromodulation in mammals

 

Related publications

  • Khamassi, M. and Lachèze, L. and Girard, B. and Berthoz, A. and Guillot, A. (2005). Actor-Critic Models of Reinforcement Learning in the Basal Ganglia: From Natural to Artificial Rats.
    Adaptive Behavior. Vol 13 No 2 Pages 131-148.
  • Khamassi, M. and Mulder, A.B. and Tabuchi, E. and Douchamps, V. and Wiener S.I. (2008). Anticipatory reward signals in ventral striatal neurons of behaving rats.
    European Journal of Neuroscience. Vol 28 No 9 Pages 1849-1866.
  • Battaglia, F.P. and Peyrache, A. and Khamassi, M. and Wiener S.I. (2008). Spatial decisions and neuronal activity in hippocampal projection zones in prefrontal cortex and striatum.
    Hippocampal place fields: Relevance to learning and memory, Oxford University Press, publisher. Pages 289-309.
  • Peyrache, A. and Khamassi, M. and Benchenane, K. and Wiener, S.I. and Battaglia, F.P. (2009). Replay of rule-learning related neural patterns in the prefrontal cortex during sleep.
    Nature Neuroscience. Vol 12 No 7 Pages 919-926.
  • Benchenane, K. and Peyrache, A. and Khamassi, M. and Tierney, P.I. and Gioanni, Y. and Battaglia, F.P. and Wiener, S.I. (2010). Coherent theta oscillations and reorganization of spike timing in the hippocampal-prefrontal network upon learning.
    Neuron. Vol 66 No 6 Pages 921-936.
  • Peyrache, A. and Benchenane, K. and Khamassi, M. and Wiener, S.I. and Battaglia, F.P. (2010). Principal component analysis of ensemble recordings reveals cell assemblies at high temporal resolution.
    Journal of Computational Neuroscience. Vol 29 No 1-2 Pages 309-325.
  • Peyrache, A. and Benchenane, K. and Khamassi, M. and Wiener, S.I. and Battaglia, F.P. (2010). Sequential reinstatement of neocortical activity during slow oscillations depends on cells' intrinsic excitability.
    Frontiers in Systems Neuroscience. Vol 3:18 Pages 1-7.
  • Khamassi, M. and Wilson, C. and Rothé, R. and Quilodran, R. and Dominey, P.F. and Procyk, E. (2011). Meta-learning, cognitive control, and physiological interactions between medial and lateral prefrontal cortex.
    Neural Basis of Motivational and Cognitive Control, Cambridge, MA: MIT Press, publisher. Pages 351-370.
  • Humphries, M. and Khamassi, M. and Gurney, K. (2012). Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia.
    Frontiers in Neuroscience. Vol 6:9 Pages 1-14.
  • Bellot, J. and Sigaud, O. and Khamassi, M. (2012). Which Temporal Difference Learning algorithm best reproduces dopamine activity in a multi-choice task?.
    From Animals to Animats: Proceedings of the 12th International Conference on Adaptive Behaviour (SAB 2012), Ziemke, T., Balkenius, C., Hallam, J. (Eds), Springer, publisher. Vol 7426/2012 Pages 289-298. Odense, Denmark. BEST PAPER AWARD.
  • Bellot, J. and Sigaud, O. and Roesch, M. R. and Schoenbaum, G. and Girard, B and Khamassi, M. (2012). Dopamine neurons activity in a multi-choice task: reward prediction error or value function?.
    Proceedings of the French Computational Neuroscience NeuroComp/KEOpS'12 workshop. Pages 1-7. Bordeaux, France.
  • Khamassi, M. and Humphries, M. D. (2012). Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies.
    Frontiers in Behavioral Neuroscience, 6:79.
  • Khamassi, M. and Enel, P. and Dominey P.F. and Procyk, E. (2013). Medial prefrontal cortex and the adaptive regulation of reinforcement learning parameters.
    Progress in Brain Research. Vol 202 Pages 441-464. Pammi, V.S.C. and Srinivasan, N. (Eds.) Decision Making: Neural and Behavioral Approaches.
  • Lesaint, F. and Sigaud, O. and Flagel, S.B. and Robinson, T.E. and Khamassi, M. (2014). Modelling individual differences observed in Pavlovian autoshaping in rats using a dual learning systems approach and factored representations. PLoS Computational Biology, 10(2): e1003466.
  • Khamassi, M. and Quilodran, R. and Enel, P. and Dominey P.F. and Procyk, E. (2015). Behavioral regulation and the modulation of information coding in the prefrontal cortex. Cerebral Cortex, 25(9):3197-218.
  • Palminteri, S., Khamassi, M., Joffily, M. and Coricelli, G. (2015).Contextual modulation of value signals in reward and punishment learning. Nature Communications, 6:809.