Sotabase

Career

· Associate Professor, University of California, Berkeley2023–
· Associate Professor, Stanford University2022–2023
· Assistant Professor, Stanford University2016–2022
· Assistant Professor, Massachusetts Institute of Technology (MIT)2015–2016
· VP, Apache Spark, The Apache Software Foundation2014–
· CTO & Cofounder, Databricks2013–
· Hadoop Committer, Apache Software Foundation2009–
· Intern, Google2007–
· Computer Science PhD Student, UC Berkeley2007–2013
· PhD in Computer Science, University of California, Berkeley2007–2013
· Research Assistant, University of Waterloo2004–2007
· Bachelor of Mathematics, Computer Science, University of Waterloo2003–2007
· BMath, University of Waterloo

Publications (519)

A view of cloud computing
CACM · 2010
10,602
cited
Above the Clouds: A Berkeley View of Cloud Computing
2009
7,448
cited
On the Opportunities and Risks of Foundation Models
arXiv.org · 2021
5,795
cited
Spark: Cluster Computing with Working Sets
USENIX Workshop on Hot Topics in Cloud Computing · 2010
4,904
cited
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
Symposium on Networked Systems Design and Implementation · 2012
4,770
cited
Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
Symposium on Networked Systems Design and Implementation · 2011
2,029
cited
Improving MapReduce Performance in Heterogeneous Environments
USENIX Symposium on Operating Systems Design and Implementation · 2008
1,938
cited
MLlib: Machine Learning in Apache Spark
Journal of machine learning research · 2015
1,838
cited
ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Annual International ACM SIGIR Conference on Research and Development in Information Retrieval · 2020
1,817
cited
Apache Spark
Communications of the ACM · 2016
1,558
cited
Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling
European Conference on Computer Systems · 2010
1,540
cited
Spark SQL: Relational Data Processing in Spark
SIGMOD Conference · 2015
1,451
cited
Dominant Resource Fairness: Fair Allocation of Multiple Resource Types
Symposium on Networked Systems Design and Implementation · 2011
1,353
cited
Discretized streams: fault-tolerant streaming computation at scale
Symposium on Operating Systems Principles · 2013
1,139
cited
PipeDream: generalized pipeline parallelism for DNN training
Symposium on Operating Systems Principles · 2019
1,069
cited
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
International Conference for High Performance Computing, Networking, Storage and Analysis · 2021
1,006
cited
Managing data transfers in computer clusters with orchestra
Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication · 2011
673
cited
Sparrow: distributed, low latency scheduling
Symposium on Operating Systems Principles · 2013
670
cited
ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
North American Chapter of the Association for Computational Linguistics · 2021
595
cited
Beyond Data and Model Parallelism for Deep Neural Networks
USENIX workshop on Tackling computer systems problems with machine learning techniques · 2018
584
cited
Sotabase