Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems

Eugenio Bargiacchi; Timothy Verstraeten; D.M. Roijers; Ann Nowé; Hado Van Hasselt

Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems

Eugenio Bargiacchi, Timothy Verstraeten, D.M. Roijers, Ann Nowé, Hado Van Hasselt

Artificial Intelligence

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

2 Downloads (Pure)

Abstract

Learning to coordinate between multiple agents is an important problem in many reinforcement learning problems. Key to learning to coordinate is exploiting loose couplings, i.e., conditional independences between agents. In this paper we study learning in repeated fully cooperative games, multi-agent multi-armed bandits (MAMABs), in which the expected rewards can be expressed as a coordination graph. We propose multi-agent upper confidence exploration (MAUCE), a new algorithm for MAMABs that exploits loose couplings, which enables us to prove a regret bound that is logarithmic in the number of arm pulls and only linear in the number of agents. We empirically compare MAUCE to sparse cooperative Q-learning, and a state-of-the-art combinatorial bandit approach, and show that it performs much better on a variety of settings, including learning control policies for wind farms.

Original language	English
Title of host publication	International Conference on Machine Learning, 10-15 July 2018, Stockholmsmässan, Stockholm Sweden
Subtitle of host publication	Proceedings of the 35th International Conference on Machine Learning, ICML 2018
Place of Publication	Stockholm
Pages	482-490
Number of pages	9
Publication status	Published - 2018
Event	International Conference on Machine Learning - Stockholmsmässan, Stockholm, Sweden Duration: 9 Jul 2018 → … Conference number: 35 https://icml.cc/

Publication series

Name	Proceedings of Machine Learning Research
Volume	80
ISSN (Electronic)	1938-7228

Conference

Conference	International Conference on Machine Learning
Abbreviated title	ICML
Country/Territory	Sweden
City	Stockholm
Period	9/07/18 → …
Internet address	https://icml.cc/

Keywords

Multi-agent
reinforcement learning
Coordination
Wind Energy

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Persistent URL (handle)

Persistent link

Cite this

Bargiacchi, E., Verstraeten, T., Roijers, D. M., Nowé, A., & Van Hasselt, H. (2018). Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems. In International Conference on Machine Learning, 10-15 July 2018, Stockholmsmässan, Stockholm Sweden : Proceedings of the 35th International Conference on Machine Learning, ICML 2018 (pp. 482-490). (Proceedings of Machine Learning Research; Vol. 80)..

Bargiacchi, Eugenio ; Verstraeten, Timothy ; Roijers, D.M. et al. / Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems. International Conference on Machine Learning, 10-15 July 2018, Stockholmsmässan, Stockholm Sweden : Proceedings of the 35th International Conference on Machine Learning, ICML 2018. Stockholm, 2018. pp. 482-490 (Proceedings of Machine Learning Research).

@inproceedings{c23270d847514b4db08b0e567e60df8e,

title = "Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems",

abstract = "Learning to coordinate between multiple agents is an important problem in many reinforcement learning problems. Key to learning to coordinate is exploiting loose couplings, i.e., conditional independences between agents. In this paper we study learning in repeated fully cooperative games, multi-agent multi-armed bandits (MAMABs), in which the expected rewards can be expressed as a coordination graph. We propose multi-agent upper confidence exploration (MAUCE), a new algorithm for MAMABs that exploits loose couplings, which enables us to prove a regret bound that is logarithmic in the number of arm pulls and only linear in the number of agents. We empirically compare MAUCE to sparse cooperative Q-learning, and a state-of-the-art combinatorial bandit approach, and show that it performs much better on a variety of settings, including learning control policies for wind farms.",

keywords = "Multi-agent, reinforcement learning, Coordination, Wind Energy",

author = "Eugenio Bargiacchi and Timothy Verstraeten and D.M. Roijers and Ann Now{\'e} and {Van Hasselt}, Hado",

year = "2018",

language = "English",

series = "Proceedings of Machine Learning Research",

pages = "482--490",

booktitle = "International Conference on Machine Learning, 10-15 July 2018, Stockholmsm{\"a}ssan, Stockholm Sweden",

note = "International Conference on Machine Learning, ICML ; Conference date: 09-07-2018",

url = "https://icml.cc/",

}

Bargiacchi, E, Verstraeten, T, Roijers, DM, Nowé, A & Van Hasselt, H 2018, Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems. in International Conference on Machine Learning, 10-15 July 2018, Stockholmsmässan, Stockholm Sweden : Proceedings of the 35th International Conference on Machine Learning, ICML 2018. Proceedings of Machine Learning Research, vol. 80, Stockholm, pp. 482-490, International Conference on Machine Learning, Stockholm, Sweden, 9/07/18.

Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems. / Bargiacchi, Eugenio; Verstraeten, Timothy; Roijers, D.M. et al.
International Conference on Machine Learning, 10-15 July 2018, Stockholmsmässan, Stockholm Sweden : Proceedings of the 35th International Conference on Machine Learning, ICML 2018. Stockholm, 2018. p. 482-490 (Proceedings of Machine Learning Research; Vol. 80).

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems

AU - Bargiacchi, Eugenio

AU - Verstraeten, Timothy

AU - Roijers, D.M.

AU - Nowé, Ann

AU - Van Hasselt, Hado

N1 - Conference code: 35

PY - 2018

Y1 - 2018

N2 - Learning to coordinate between multiple agents is an important problem in many reinforcement learning problems. Key to learning to coordinate is exploiting loose couplings, i.e., conditional independences between agents. In this paper we study learning in repeated fully cooperative games, multi-agent multi-armed bandits (MAMABs), in which the expected rewards can be expressed as a coordination graph. We propose multi-agent upper confidence exploration (MAUCE), a new algorithm for MAMABs that exploits loose couplings, which enables us to prove a regret bound that is logarithmic in the number of arm pulls and only linear in the number of agents. We empirically compare MAUCE to sparse cooperative Q-learning, and a state-of-the-art combinatorial bandit approach, and show that it performs much better on a variety of settings, including learning control policies for wind farms.

AB - Learning to coordinate between multiple agents is an important problem in many reinforcement learning problems. Key to learning to coordinate is exploiting loose couplings, i.e., conditional independences between agents. In this paper we study learning in repeated fully cooperative games, multi-agent multi-armed bandits (MAMABs), in which the expected rewards can be expressed as a coordination graph. We propose multi-agent upper confidence exploration (MAUCE), a new algorithm for MAMABs that exploits loose couplings, which enables us to prove a regret bound that is logarithmic in the number of arm pulls and only linear in the number of agents. We empirically compare MAUCE to sparse cooperative Q-learning, and a state-of-the-art combinatorial bandit approach, and show that it performs much better on a variety of settings, including learning control policies for wind farms.

KW - Multi-agent

KW - reinforcement learning

KW - Coordination

KW - Wind Energy

M3 - Conference contribution

T3 - Proceedings of Machine Learning Research

SP - 482

EP - 490

BT - International Conference on Machine Learning, 10-15 July 2018, Stockholmsmässan, Stockholm Sweden

CY - Stockholm

T2 - International Conference on Machine Learning

Y2 - 9 July 2018

ER -

Bargiacchi E, Verstraeten T, Roijers DM, Nowé A, Van Hasselt H. Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems. In International Conference on Machine Learning, 10-15 July 2018, Stockholmsmässan, Stockholm Sweden : Proceedings of the 35th International Conference on Machine Learning, ICML 2018. Stockholm. 2018. p. 482-490. (Proceedings of Machine Learning Research). Epub 2018 Jul.

Learning to Coordinate with Coordination Graphs in Repeated Single-Stage Multi-Agent Decision Problems

Abstract

Publication series

Conference

Keywords

UN SDGs

Persistent URL (handle)

Fingerprint

Cite this