📄Publications

(* Equal contribution)

Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning
Pengfei Zheng, Rui Pan, Tarannum Khan, Shivaram Venkataraman, Aditya Akella
USENIX NSDI 2023
Abstract

Dynamic adaptation has become an essential technique in accelerating distributed machine learning (ML) training: Recent studies have shown that dynamically adjusting model structure (e.g., lottery ticket hypothesis) or hyperparameters (e.g., batch size) can significantly accelerate training without sacrificing accuracy. However, existing ML cluster schedulers are not designed to handle dynamic adaptation. We show that existing schemes fail to provide fairness and degrade system efficiency when the training throughput changes over time under dynamic adaptation. We design Shockwave, a scheduler with future planning that builds on two key ideas. First, Shockwave extends classic market theory from static settings to dynamic settings to co-optimize efficiency and fairness. Second, Shockwave utilizes stochastic dynamic programming to handle uncertain, dynamic throughput. We build a system for Shockwave and validate its performance with both trace-driven simulation and cluster experiments. Results show that for traces of ML jobs with dynamic adaptation, Shockwave improves makespan by 1.3× and fairness by 2× when compared with existing fair scheduling schemes.
Efficient Flow Scheduling in Distributed Deep Learning Training with Echelon Formation
Rui Pan*, Yiming Lei*, Jialong Li, Zhiqiang Xie, Binhang Yuan, Yiting Xia
ACM HotNets 2022
Abstract

This paper discusses why flow scheduling does not apply to distributed deep learning training and presents EchelonFlow, the first network abstraction to bridge the gap. EchelonFlow deviates from the common belief that semantically related flows should finish at the same time. We reached the key observation, after extensive workflow analysis of diverse training paradigms, that distributed training jobs observe strict computation patterns, which may consume data at different times. We devise a generic method to model the drastically different computation patterns across training paradigms, and formulate EchelonFlow to regulate flow finish times accordingly. Case studies of mainstream training paradigms under EchelonFlow demonstrate the expressiveness of the abstraction, and our system sketch suggests the feasibility of an EchelonFlow scheduling system.

Some other non peer-reviewed write-ups include:

CS 759 Project Report: Cautiously Aggressive GPU Space Sharing for Improving Resource Utilization and Job Efficiency (pdf)
CS 744 Project Report: Comparing Black-Box Optimization Methods for Online DBMS Tuning (pdf)
AgDH: A System for Gathering and Disseminating Dairy Data (pdf)

Rui Pan 潘瑞

🧐About

📄Publications

Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning

Efficient Flow Scheduling in Distributed Deep Learning Training with Echelon Formation

🎓Education

🐅Princeton University

🦡University of Wisconsin-Madison

👔Experience

🇩🇪Research Intern @ Max Planck Institute for Informatics (MPI-INF)

😍Interests

🤙Contacts