📄Publications

Conference Paper Shockwave: Proactive, Fair, and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning
Pengfei Zheng, Rui Pan, Tarannum Khan, Shivaram Venkataraman, Aditya Akella
USENIX NSDI 2023 (to appear, PDF & code coming soon)
Abstract

Dynamic adaptation has become an essential technique in accelerating distributed machine learning (ML) training: Recent studies have shown that dynamically adjusting model structure (e.g., lottery ticket hypothesis) or hyperparameters (e.g., batch size) can significantly accelerate training without sacrificing accuracy. However, existing ML cluster schedulers are not designed to handle dynamic adaptation. We show that existing schemes fail to provide fairness and degrade system efficiency when the training throughput changes over time under dynamic adaptation. We design Shockwave, a scheduler with future planning that builds on two key ideas. First, Shockwave extends classic market theory from static settings to dynamic settings to co-optimize efficiency and fairness. Second, Shockwave utilizes stochastic dynamic programming to handle uncertain, dynamic throughput. We build a system for Shockwave and validate its performance with both trace-driven simulation and cluster experiments. Results show that for traces of ML jobs with dynamic adaptation, Shockwave improves makespan by 1.3× and fairness by 2× when compared with existing fair scheduling schemes.
Workshop Paper Flow Scheduling for Machine Learning
In submission, 2022
Abstract

[Redacted]
Poster AgDH: A System for Gathering and Disseminating Dairy Data
Rui Pan, Steven Wangen, Michael Ferris
Presented at the 3^rd Annual WID Symposium, 2020
Abstract

Dairy farms have been incorporating modern data-tracking services, which generate an enormous amount of data of myriad types (e.g., genetic, nutritional, reproductive). The organic nature by which the different types of automation systems have arisen and developed has resulted in a highly heterogeneous arrangement of different systems from different companies that often have difficulty integrating with one another. Additionally, dependency on high-cost hardware systems (e.g., milking machines) makes it difficult for milk producers to switch service providers, which can disincentivize adaptation of the existing technologies. Modern agricultural analytics relies on the ability to integrate data from all of these data streams. To that end, we present the Agricultural Data Hub (AgDH), which revolutionizes dairy data collection and interpretation by providing uniformed data for future analysis through an extraction, transformation, and loading (ETL) process. This system establishes the relationships between these data, integrates those data, and makes them available from a single source, thus making it easier for dairy farmers to make management decisions. Later, these uniformed data will be sent to the Dairy Brain analytics services for additional analyses, and facilitate the visualization of the raw data and the analysis outcomes for easier consumption.

Rui Pan 潘瑞

🧐About

📄Publications

Conference Paper Shockwave: Proactive, Fair, and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning

Workshop Paper Flow Scheduling for Machine Learning

Poster AgDH: A System for Gathering and Disseminating Dairy Data

🎓Education

🐅Princeton University

🦡University of Wisconsin-Madison

👔Experience

Research Intern @ Max Planck Institute for Informatics (MPI-INF)

😍Interests

🤙Contacts