Exploration in reward-based motor learning

Our movements are variable: no movement is the same, even if you intend to make the same movement. This motor variability may allow you to find the movements that serve your movement goals best. The relation between variability and motor learning is however unclear. This may be the case because motor variability can arise from two sources: inevitable sensorimotor noise (inherent, random variability) and exploration (variability that can be controlled and can be learnt from). In this thesis, I aimed to disentangle sensorimotor noise and exploration to study the role of exploration in reward-based motor learning. We speak of this type of motor learning when learning how to move based on binary success and failure feedback: whether your movement was successful or not. In this thesis, we operationalized exploration as the additional variability following failure as compared to following success. In the first part of my thesis, I report on one experiment and one simulation study aiming to validate a method for quantifying exploration based on this operationalization. In Chapter 2, we proposed to estimate variability based on trial-to-trial changes, since standard measures of variability are sensitive to learning. We consider trial-to-trial changes following successful trials sensorimotor noise, since participants likely aim to repeat their movement in that situation. We proposed a trial-to-trial change (TTC) method for quantifying exploration as the additional trial-to-trial change following failed trials relative to the trial-to-trial change following successful trials. In Chapter 3, we aimed to validate the trial-to-trial change (TTC) method by simulating learning with four reward-based motor learning models and comparing the input exploration of the models to the trial-to-trial change exploration estimates of our method. Since the simulations allowed us to identify two pitfalls in quantifying exploration in reward-based motor learning, we reformulated our method to the additional trial-to-trial change (ATTC) method that is valid for one class of reward-based motor learning models and under specific circumstances. In the last part of my thesis, I report on two experiments in which I studied reward-based motor learning. In Chapter 4, I found that binary success and failure feedback can induce implicit learning, and that this implicit learning cannot be explained by use-dependent learning. In Chapter 5, I found no relation between overall motor variability, sensorimotor noise and exploration as estimated with the ATTC method on the one hand, and reward-based motor learning on the other hand. In conclusion, we operationalized the intuitive concept of exploration as the additional variability following failed movements as compared to successful movements. Even this intuitive operationalization seems to have its snags. Our ATTC method could be used for quantifying exploration under specific circumstances and under the assumption that humans control exploratory variability only based on the success of the most recent movement. A more general lesson is that intuitive concepts like exploration and implicit learning are difficult to capture once one tries to formalize them.