Leveraging Communication Topologies Between Learning Agents in Deep Reinforcement Learning
Abstract
A common technique to improve learning performance in deep reinforcement learning (DRL) and many other machine learning algorithms is to run multiple learning agents in parallel. A neglected component in the development of these algorithms has been how best to arrange the learning agents involved to improve distributed search. Here we draw upon results from the networked optimization literatures suggesting that arranging learning agents in communication networks other than fully connected topologies (the implicit way agents are commonly arranged in) can improve learning. We explore the relative performance of four popular families of graphs and observe that one such family (ErdosRenyi random graphs) empirically outperforms the de facto fullyconnected communication topology across several DRL benchmark tasks. Additionally, we observe that 1000 learning agents arranged in an ErdosRenyi graph can perform as well as 3000 agents arranged in the standard fullyconnected topology, showing the large learning improvement possible when carefully designing the topology over which agents communicate. We complement these empirical results with a theoretical investigation of why our alternate topologies perform better. Overall, our work suggests that distributed machine learning algorithms could be made more effective if the communication topology between learning agents was optimized.
 Publication:

arXiv eprints
 Pub Date:
 February 2019
 DOI:
 10.48550/arXiv.1902.06740
 arXiv:
 arXiv:1902.06740
 Bibcode:
 2019arXiv190206740A
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning
 EPrint:
 arXiv admin note: substantial text overlap with arXiv:1811.12556