Towards empathic deep q-learning

Bart Bussmann*, Jacqueline Heinerman, Joel Lehman

*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review


As reinforcement learning (RL) scales to solve increasingly complex tasks, interest continues to grow in the fields of AI safety and machine ethics. As a contribution to these fields, this paper introduces an extension to Deep Q-Networks (DQNs), called Empathic DQN, that is loosely inspired both by empathy and the golden rule ("Do unto others as you would have them do unto you"). Empathic DQN aims to help mitigate negative side effects to other agents resulting from myopic goal-directed behavior. We assume a setting where a learning agent coexists with other independent agents (who receive unknown rewards), where some types of reward (e.g. negative rewards from physical harm) may generalize across agents. Empathic DQN combines the typical (self-centered) value with the estimated value of other agents, by imagining (by its own standards) the value of it being in the other's situation (by considering constructed states where both agents are swapped). Proof-of-concept results in two gridworld environments highlight the approach's potential to decrease collateral harms. While extending Empathic DQN to complex environments is non-trivial, we believe that this first step highlights the potential of bridge-work between machine ethics and RL to contribute useful priors for norm-abiding RL agents.

Original languageEnglish
Title of host publicationArtificial Intelligence Safety 2019
Subtitle of host publicationProceedings of the Workshop on Artificial Intelligence Safety 2019 co-located with the 28th International Joint Conference on Artificial Intelligence (IJCAI-19), Macao, China, August 11-12, 2019
EditorsHuáscar Espinoza, Han Yu, Xiaowei Huang, Freddy Lecue, Cynthia Chen, José Hernández-Orallo, Seán Ó hÉigeartaigh, Richard Mallah
Number of pages7
Publication statusPublished - 11 Aug 2019
Event2019 Workshop on Artificial Intelligence Safety, AISafety 2019 - Macao, China
Duration: 11 Aug 201912 Aug 2019

Publication series

NameCEUR Workshop Proceedings
PublisherCEUR Workshop Proceedings
ISSN (Print)1613-0073


Conference2019 Workshop on Artificial Intelligence Safety, AISafety 2019


Dive into the research topics of 'Towards empathic deep q-learning'. Together they form a unique fingerprint.

Cite this