Timezone: »

 
Oral
Blink: Fast and Generic Collectives for Distributed ML
Guanhua Wang · Shivaram Venkataraman · Amar Phanishayee · Nikhil Devanur · Jorgen Thelin · Ion Stoica

Tue Mar 03 02:30 PM -- 02:55 PM (PST) @ Ballroom A
in Efficient Model Training 2 »

Model parameter synchronization across GPUs introduces high overheads for data-parallel training at scale. Existing parameter synchronization protocols cannot effectively leverage available network resources in the face of ever increasing hardware heterogeneity. To address this issue, we propose Blink, a collective communication library that dynamically generates optimal communication primitives by packing spanning trees. We propose techniques to minimize the number of trees generated and extend Blink to leverage heterogeneous communication channels for hybrid, and faster, data transfers. Evaluations show that compared to the state-of-the-art (NCCL), Blink can achieve up to 8× faster model synchronization (AllReduce), and reduce end-to-end DNN training time for image classification tasks by up to 40%.

Author Information

Guanhua Wang (UC Berkeley)

I am a Ph.D. student in the AMPLab / RISELab, at UC Berkeley, advised by Prof. Ion Stoica.

Shivaram Venkataraman (University of Wisconsin, Madison)
Amar Phanishayee (Microsoft Research)
Nikhil Devanur (Microsoft)
Jorgen Thelin (Microsoft Research)
Ion Stoica (UC Berkeley)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors