Tessereau, Charline
(2021)
Reinforcement Learning Approaches to Rapid Hippocampal Place Learning.
PhD thesis, University of Nottingham.
Abstract
The ability to successfully navigate the physical environment is a vital skill for numer- ous species, including humans, to find food and shelter and remember how to return to important locations. As environments are inherently variable, brains have evolved amazing capabilities to adapt to various new situations. In particular, animals and humans have the ability to return to specific locations based on as few as a single experience. The mechanisms underlying behavioural flexibility in spatial navigation is the focus of ongoing research with repercussions in behavioural sciences, neurosciences, and artificial intelligence.
In particular, the field of Reinforcement Learning (RL), which investigates how an or- ganism, virtual or living, learns to generate actions based on the reception of rewards, has been extremely active since the 1970s for the exploration of the mechanisms of flexibil- ity underlying decision making. In parallel, neuroscience has also significantly advanced in uncovering the neural basis underlying spatial navigation mechanisms, for example with the discovery of neurons underlying the computation of cognitive maps [O’Keefe and Dostrovsky, 1971, Hafting et al., 2005], an internal representation of space. Past RL models design relies on representations that do not allow efficient flexibility in spatial navigation. However, models provide a theoretical framework that influences the interpretation of neural recordings. As recent recording technologies enable experimentalists to target an increasing number of neurons, there is a compelling need to develop new RL computational approaches for flexible spatial navigation, in particular to bridge the gap between neural population recordings and the production of behaviours.
In this thesis, I consider RL approaches in which the known coding properties of the cognitive map are used as a basis to perform spatial navigation. Specifically, I investigate computational ideas which enable agents to be more flexible in virtual spatial navigation scenarios. In particular, this thesis focuses on the Morris watermaze, an experimental apparatus in which rodents have to find a hidden platform within a pool of cloudy water. Rapid place learning in the Morris watermaze, demonstrated by rodents requiring only one exposure to a new platform location to subsequently be able to retrieve its position, is an example of flexibility in spatial navigation. I present different RL-based architectures which generate flexible behaviours in a virtual watermaze equivalent, and compare them to behavioural observations. I discuss both the similarity in behavioural performance (i.e., how well they reproduce behavioural measures of rapid place learning) and neuro- biological realism (i.e., how well they map to neurobiological substrates involved in rapid place learning).
I propose distinct biologically realistic computational properties which enable an agent to be more flexible towards changes in goal locations. Behavioural flexibility requires hier- archical and generalisable representations for flexible transfer of knowledge. Hierarchical control is useful to generalise action chains, such as selecting a trajectory, to fulfil differ- ent purposes, such as reaching different goal locations. It also enables the adjustment of ongoing behaviour to unforeseen situations, for example, adapting to misprediction of the goal’s location. Continuous encoding of space, action and time, permits smoother control and generalisation of experience, and removes the constraints caused by the choice of the representation’s granularity. Neural networks in which connections between neurons re- flect predictions about most likely future scenarios enable efficient planning of trajectories to adapt to novel situations. In a nutshell, flexibility requires efficient representations, and this thesis contributes to the investigation of their neural implementations.
Actions (Archive Staff Only)
 |
Edit View |
Tools
Tools
