Learning to Communicate Using Counterfactual Reasoning
Abstract
Learning to communicate in order to share state information is an active problem in the area of multi-agent reinforcement learning (MARL). The credit assignment problem, the non-stationarity of the communication environment and the creation of influenceable agents are major challenges within this research field which need to be overcome in order to learn a valid communication protocol. This paper introduces the novel multi-agent counterfactual communication learning (MACC) method which adapts counterfactual reasoning in order to overcome the credit assignment problem for communicating agents. Secondly, the non-stationarity of the communication environment while learning the communication Q-function is overcome by creating the communication Q-function using the action policy of the other agents and the Q-function of the action environment. Additionally, a social loss function is introduced in order to create influenceable agents which is required to learn a valid communication protocol. Our experiments show that MACC is able to outperform the state-of-the-art baselines in four different scenarios in the Particle environment.
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2020
- DOI:
- 10.48550/arXiv.2006.07200
- arXiv:
- arXiv:2006.07200
- Bibcode:
- 2020arXiv200607200V
- Keywords:
-
- Computer Science - Machine Learning;
- Computer Science - Multiagent Systems;
- Statistics - Machine Learning
- E-Print:
- Accepted at Adaptive and Learning Agents Workshop (ALA 2022) https://ala2022.github.io/