Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Denis Steckelmacher; Hélène Plisnier; Diederik M. Roijers; Ann Nowé

doi:10.1007/978-3-030-46133-1_2

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Denis Steckelmacher^*
, Hélène Plisnier
, Diederik M. Roijers
, Ann Nowé

^*Corresponding author for this work

Artificial Intelligence

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi. Appendix: https://arxiv.org/abs/1903.04193.

Original language	English
Title of host publication	Machine Learning and Knowledge Discovery in Databases
Subtitle of host publication	European Conference, ECML PKDD 2019Würzburg, Germany, September 16–20, 2019Proceedings, Part III
Editors	Ulf Brefeld, Elisa Fromont, Andreas Hotho, Arno Knobbe, Marloes Maathuis, Céline Robardet
Publisher	Springer
Pages	19-34
Number of pages	16
Volume	3
ISBN (Electronic)	9783030461331
ISBN (Print)	9783030461324
DOIs	https://doi.org/10.1007/978-3-030-46133-1_2
Publication status	Published - 2020
Event	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019 - Wurzburg, Germany Duration: 16 Sept 2019 → 20 Sept 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11908 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019
Country/Territory	Germany
City	Wurzburg
Period	16/09/19 → 20/09/19

Funding

Acknowledgments. The first and second authors are funded by the Science Foundation of Flanders (FWO, Belgium), respectively as 1129319N Aspirant, and 1SA6619N Applied Researcher.

Keywords

Actor-critic
Reinforcement learning
Value iteration

Access to Document

10.1007/978-3-030-46133-1_2

Persistent URL (handle)

Persistent link

Cite this

Steckelmacher, D., Plisnier, H., Roijers, D. M., & Nowé, A. (2020). Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics. In U. Brefeld, E. Fromont, A. Hotho, A. Knobbe, M. Maathuis, & C. Robardet (Eds.), Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019Würzburg, Germany, September 16–20, 2019Proceedings, Part III (Vol. 3, pp. 19-34). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11908 LNAI). Springer. https://doi.org/10.1007/978-3-030-46133-1_2

Steckelmacher, Denis ; Plisnier, Hélène ; Roijers, Diederik M. et al. / Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019Würzburg, Germany, September 16–20, 2019Proceedings, Part III. editor / Ulf Brefeld ; Elisa Fromont ; Andreas Hotho ; Arno Knobbe ; Marloes Maathuis ; Céline Robardet. Vol. 3 Springer, 2020. pp. 19-34 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{c6221af8ec0b4cc19d3967065b9c980e,

title = "Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics",

abstract = "Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi. Appendix: https://arxiv.org/abs/1903.04193.",

keywords = "Actor-critic, Reinforcement learning, Value iteration",

author = "Denis Steckelmacher and H{\'e}l{\`e}ne Plisnier and Roijers, \{Diederik M.\} and Ann Now{\'e}",

year = "2020",

doi = "10.1007/978-3-030-46133-1\_2",

language = "English",

isbn = "9783030461324",

volume = "3",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "19--34",

editor = "Ulf Brefeld and Elisa Fromont and Andreas Hotho and Arno Knobbe and Marloes Maathuis and C{\'e}line Robardet",

booktitle = "Machine Learning and Knowledge Discovery in Databases",

note = "European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019 ; Conference date: 16-09-2019 Through 20-09-2019",

}

Steckelmacher, D, Plisnier, H, Roijers, DM & Nowé, A 2020, Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics. in U Brefeld, E Fromont, A Hotho, A Knobbe, M Maathuis & C Robardet (eds), Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019Würzburg, Germany, September 16–20, 2019Proceedings, Part III. vol. 3, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11908 LNAI, Springer, pp. 19-34, European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019, Wurzburg, Germany, 16/09/19. https://doi.org/10.1007/978-3-030-46133-1_2

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics. / Steckelmacher, Denis; Plisnier, Hélène; Roijers, Diederik M. et al.
Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019Würzburg, Germany, September 16–20, 2019Proceedings, Part III. ed. / Ulf Brefeld; Elisa Fromont; Andreas Hotho; Arno Knobbe; Marloes Maathuis; Céline Robardet. Vol. 3 Springer, 2020. p. 19-34 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11908 LNAI).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

AU - Steckelmacher, Denis

AU - Plisnier, Hélène

AU - Roijers, Diederik M.

AU - Nowé, Ann

PY - 2020

Y1 - 2020

N2 - Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi. Appendix: https://arxiv.org/abs/1903.04193.

AB - Value-based reinforcement-learning algorithms provide state-of-the-art results in model-free discrete-action settings, and tend to outperform actor-critic algorithms. We argue that actor-critic algorithms are limited by their need for an on-policy critic. We propose Bootstrapped Dual Policy Iteration (BDPI), a novel model-free reinforcement-learning algorithm for continuous states and discrete actions, with an actor and several off-policy critics. Off-policy critics are compatible with experience replay, ensuring high sample-efficiency, without the need for off-policy corrections. The actor, by slowly imitating the average greedy policy of the critics, leads to high-quality and state-specific exploration, which we compare to Thompson sampling. Because the actor and critics are fully decoupled, BDPI is remarkably stable, and unusually robust to its hyper-parameters. BDPI is significantly more sample-efficient than Bootstrapped DQN, PPO, and ACKTR, on discrete, continuous and pixel-based tasks. Source code: https://github.com/vub-ai-lab/bdpi. Appendix: https://arxiv.org/abs/1903.04193.

KW - Actor-critic

KW - Reinforcement learning

KW - Value iteration

UR - https://www.scopus.com/pages/publications/85084826247

UR - https://www.scopus.com/pages/publications/85084826247#tab=citedBy

U2 - 10.1007/978-3-030-46133-1_2

DO - 10.1007/978-3-030-46133-1_2

M3 - Conference contribution

AN - SCOPUS:85084826247

SN - 9783030461324

VL - 3

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 19

EP - 34

BT - Machine Learning and Knowledge Discovery in Databases

A2 - Brefeld, Ulf

A2 - Fromont, Elisa

A2 - Hotho, Andreas

A2 - Knobbe, Arno

A2 - Maathuis, Marloes

A2 - Robardet, Céline

PB - Springer

T2 - European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML PKDD 2019

Y2 - 16 September 2019 through 20 September 2019

ER -

Steckelmacher D, Plisnier H, Roijers DM, Nowé A. Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics. In Brefeld U, Fromont E, Hotho A, Knobbe A, Maathuis M, Robardet C, editors, Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019Würzburg, Germany, September 16–20, 2019Proceedings, Part III. Vol. 3. Springer. 2020. p. 19-34. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). Epub 2020 Apr 30. doi: 10.1007/978-3-030-46133-1_2

Sample-Efficient Model-Free Reinforcement Learning with Off-Policy Critics

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this