Emergent dynamics of joy, distress, hope and fear in reinforcement learning agents

We report on a study that shows plausible emotion dynamics for joy, distress, hope and fear, emerging in an adaptive agent that uses Reinforcement Learning (RL) to adapt to a task. Joy/distress is a signal that is derived from the RL update signal, while hope/fear is derived from the utility of the current state. Agent-based simulation experiments replicate psychological and behavioral dynamics of emotion including: joy and distress reactions that develop prior to hope and fear; fear extinction; habituation of joy; and, task randomness that increases the intensity of joy and distress. This work distinguishes itself by assessing the dynamics of emotion in an adaptive agent framework - coupling it to the literature on habituation, development, and extinction. Our results support the idea that the function of emotion is to provide a complex feedback signal for an organism to adapt its behavior. We show this feedback signal can be operationalized for RL agents. This is important because (a) RL-based models can help understand the relation between emotion and adaptation in animals, (b) the emotional state might be used to increase adaptive potential, and (c) expression of an emotion to a human observer that it is grounded in the learning mechanism of the agent should help interpret the meaning of the emotion.