Timezone: »

 
Oral
RL-Scope: Cross-stack Profiling for Deep Reinforcement Learning Workloads
James Gleeson · Srivatsan Krishnan · Moshe Gabel · Vijay Janapa Reddi · Eyal de Lara · Gennady Pekhimenko

Wed Apr 07 11:10 AM -- 11:30 AM (PDT) @
in Session 6: Benchmarks, Cost models, and Profiling »

Deep reinforcement learning (RL) has made groundbreaking advancements in robotics, data center management and other applications. Unfortunately, system-level bottlenecks in RL workloads are poorly understood; we observe fundamental structural differences in RL workloads that make them inherently less GPU-bound than supervised learning (SL). To explain where training time is spent in RL workloads, we propose RL-Scope, a cross-stack profiler that scopes low-level CPU/GPU resource usage to high-level algorithmic operations, and provides accurate insights by correcting for profiling overhead. Using RL-Scope, we survey RL workloads across its major dimensions including ML backend, RL algorithm, and simulator. For ML backends, we explain a 2.3× difference in runtime between equivalent PyTorch and TensorFlow algorithm implementations, and identify a bottleneck rooted in overly abstracted algorithm implementations. For RL algorithms and simulators, we show that on-policy algorithms are at least 3.5× more simulation-bound than off-policy algorithms. Finally, we profile a scale-up workload and demonstrate that GPU utilization metrics reported by commonly used tools dramatically inflate GPU usage, whereas RL-Scope reports true GPU-bound time. RL-Scope is an open-source tool available at https://github.com/UofT-EcoSystem/rlscope.

Author Information

James Gleeson (University of Toronto)
Srivatsan Krishnan (Harvard University)
Moshe Gabel (University of Toronto )
Vijay Janapa Reddi (Harvard University)
Eyal de Lara (University of Toronto)
Gennady Pekhimenko (University of Toronto)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors