Timezone: »

 
Oral
Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
Kiwan Maeng · Shivam Bharuka · Isabel Gao · Mark Jeffrey · Vikram Saraph · Bor-Yiing Su · Caroline Trippel · Jiyan Yang · Mike Rabbat · Brandon Lucia · Carole-Jean Wu

Thu Apr 08 03:40 PM -- 04:00 PM (PDT) @
in Session 12: Training (II) »

The paper proposes and optimizes a partial recovery training system, CPR, for recommendation models. CPR relaxes the consistency requirement by enabling non-failed nodes to proceed without loading checkpoints when a node fails during training, improving failure-related overheads. The paper is the first to the extent of our knowledge to perform a data-driven, in-depth analysis of applying partial recovery to recommendation models and identified a trade-off between accuracy and performance. Motivated by the analysis, we present CPR, a partial recovery training system that can reduce the training time and maintain the desired level of model accuracy by (1) estimating the benefit of partial recovery, (2) selecting an appropriate checkpoint saving interval, and (3) prioritizing to save updates of more frequently accessed parameters. Two variants of CPR, CPR-MFU and CPR-SSU, reduce the checkpoint-related overhead from 8.2--8.5% to 0.53--0.68% compared to full recovery, on a setup emulating the failure pattern and overhead of a production-scale cluster. While reducing overhead significantly, CPR achieves model quality on par with the more expensive full recovery scheme, training the state-of-the-art recommendation model using Criteo’s Terabyte CTR dataset. Our results also suggest that CPR can speed up training on a real production-scale cluster, without notably degrading the accuracy.

Author Information

Kiwan Maeng (Carnegie Mellon University)
Shivam Bharuka (Facebook Inc.)

I am a production engineer, supporting the AI infrastructure at Facebook. My research interests include exploring new computer architectures and designing scalable distributed systems. Before joining Facebook, I graduated with a master’s and a bachelor’s degree from University of Illinois at Urbana-Champaign, studying Computer Engineering.

Isabel Gao (Facebook)
Mark Jeffrey (University of Toronto)
Vikram Saraph (The Johns Hopkins University Applied Physics Laboratory)
Bor-Yiing Su (Facebook)
Caroline Trippel (Stanford / Facebook)
Jiyan Yang (Facebook Inc.)
Mike Rabbat (Facebook FAIR)
Brandon Lucia (Carnegie Mellon University)
Carole-Jean Wu (Facebook AI)

Related Events (a corresponding poster, oral, or spotlight)

More from the Same Authors