RL-Scope: Cross-Stack Profiling for Deep Reinforcement Learning Workloads
Abstract
Deep reinforcement learning (RL) has made groundbreaking advancements in robotics, data center management and other applications. Unfortunately, system-level bottlenecks in RL workloads are poorly understood; we observe fundamental structural differences in RL workloads that make them inherently less GPU-bound than supervised learning (SL). To explain where training time is spent in RL workloads, we propose RL-Scope, a cross-stack profiler that scopes low-level CPU/GPU resource usage to high-level algorithmic operations, and provides accurate insights by correcting for profiling overhead. Using RL-Scope, we survey RL workloads across its major dimensions including ML backend, RL algorithm, and simulator. For ML backends, we explain a $2.3\times$ difference in runtime between equivalent PyTorch and TensorFlow algorithm implementations, and identify a bottleneck rooted in overly abstracted algorithm implementations. For RL algorithms and simulators, we show that on-policy algorithms are at least $3.5\times$ more simulation-bound than off-policy algorithms. Finally, we profile a scale-up workload and demonstrate that GPU utilization metrics reported by commonly used tools dramatically inflate GPU usage, whereas RL-Scope reports true GPU-bound time. RL-Scope is an open-source tool available at https://github.com/UofT-EcoSystem/rlscope .
- Publication:
-
arXiv e-prints
- Pub Date:
- February 2021
- DOI:
- 10.48550/arXiv.2102.04285
- arXiv:
- arXiv:2102.04285
- Bibcode:
- 2021arXiv210204285G
- Keywords:
-
- Computer Science - Machine Learning;
- Computer Science - Software Engineering
- E-Print:
- RL-Scope is an open-source tool available at https://github.com/UofT-EcoSystem/rlscope . Proceedings of the 4th MLSys Conference, 2021. Changes: camera ready for MLSys publication -- shorten abstract, add acknowledgements, minor grammar fixes