Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations

Pipelined Backpropagation at Scale: Training Large Models without Batches

Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training

Weight Decay may matter more than muP for Learning Rate Transfer in Practice

Training Dynamics of the Cooldown Stage in Warmup-Stable-Decay Learning Rate Scheduler

Ghost Noise for Regularizing Deep Neural Networks

Rotational Optimizers: Simple & Robust DNN Training