Close

Presentation

MixPipe: Efficient Bidirectional Pipeline Parallelism for Training Large-Scale Models
DescriptionIt is very challenging to train large DNN models at scale. This paper proposes MixPipe, a novel bidirectional pipeline scheme for efficiently training large-scale models in synchronous scenarios. Compared with existing synchronous pipeline approaches, MixPipe achieves a better balance between pipeline efficiency and device utilization, which benefits from the flexible regulating for the number of micro-batches injected into pipelines at the beginning. MixPipe also features a mixed schedule for balanced memory usage and less bubbles. Evaluation results show that: for Transformer based language models, MixPipe improves the training throughput by up to 2.39x over the state-of-the-art synchronous pipeline approaches.
Event Type
Research Manuscript
TimeTuesday, July 11th4:25pm - 4:40pm PDT
Location3006, 3rd Floor
Topics
Design
Keywords
AI/ML System and Platform Design