After 10 years of Adam: what actually trains neural networks faster

Qminers Reading Group, Session 13. This post follows the format of our internal sessions: what the papers say, and what we think about it.

Optimization is the quiet engine of deep learning. Every model, from a small forecasting network to a frontier LLM, is the output of an optimizer grinding through a loss landscape. For roughly a decade, that optimizer was Adam, and the question seemed closed. The LLM era reopened it: when a single training run costs millions, a 10% efficiency gain is real money, and a wave of new methods has arrived to claim it. Session 13 traced the line from classic stochastic gradient descent to the current frontier, across seven papers and one influential blog post.

What the papers say

The classics. SGD updates parameters along the negative gradient, one noisy mini-batch at a time. Momentum smooths the noise by averaging recent gradients. AdaGrad (2011) introduced the key idea behind everything that followed: scale each parameter's step size by the history of its gradients, so rarely-updated parameters take bigger steps. Adam (2014) combined momentum with AdaGrad-style scaling and added bias correction, and its default hyperparameters proved robust enough that most practitioners never touched them. AdamW (2017) fixed a subtle bug in how Adam handles regularization: for adaptive methods, L2 regularization and weight decay are not the same thing, and decoupling them makes the learning rate and the decay strength independently tunable. AdamW remains the default for LLM training today.

The new wave. Adam treats every parameter as an independent scalar. But the parameters of a neural network come in matrices, and gradients of a matrix carry structure that scalar methods throw away. Shampoo (2018) exploits this: it preconditions each gradient matrix from the left and from the right, approximating the full second-order statistics through a Kronecker product, which turns an intractable 50,000 by 50,000 matrix inverse into two small ones. The paper sat quietly for years until a distributed PyTorch implementation (2023) made it practical, and Shampoo won the external tuning track of the AlgoPerf benchmark, the most serious independent comparison of training algorithms to date.

SOAP (2024) showed that Shampoo's update is equivalent to running Adafactor in Shampoo's eigenbasis, and simplified the recipe to: run Adam, but in a rotated coordinate system that is refreshed only occasionally. The result matches Shampoo's gains with fewer hyperparameters, reporting roughly 40% fewer iterations than tuned AdamW on a 660M-parameter model. Muon takes an even more radical shortcut: take the momentum matrix, orthogonalize it (via a fast Newton-Schulz iteration rather than an expensive SVD), and step in that direction. The update becomes a pure rotation in matrix space, all singular values squashed to one. It uses half of Adam's optimizer memory and is applied only to 2D hidden-layer weights, with embeddings and norms still handled by AdamW.

The reality check. The paper with our favourite title, Fantastic Pretraining Optimizers and Where to Find Them, benchmarks the whole zoo against a properly tuned AdamW baseline. Three findings stand out. Speedups do not exceed 1.4x, well below the 2x figures in the original papers, which mostly reflect weak baselines. The gains shrink with scale: matrix-based methods deliver about 1.3x on models under 520M parameters, but around 1.1x at 1.2B parameters trained at an 8x Chinchilla ratio. And early loss curves mislead: optimizer rankings can flip during learning rate decay, so judging methods by intermediate checkpoints is unreliable.

Schedules. The Road Less Scheduled attacks a different assumption. Theory says the optimal learning rate is constant and the final answer should average all iterates; practice uses decaying schedules and keeps the last iterate. The authors show these are two views of the same trick: linear decay implicitly weights gradients the way iterate averaging would, and the ubiquitous cosine schedule works mainly because it approximates linear decay, while wasting the last few percent of the run on learning rates too small to matter. Their schedule-free method achieves the same optimal rates without needing to know the total step count in advance, and it topped the self-tuning track of AlgoPerf.

What we think

The debunking paper is the most valuable of the eight. Not because the new optimizers are useless, but because it quantifies what they are worth: a well-tuned AdamW is a much harder baseline than the literature implies, and hyperparameters do not transfer between optimizers, so most published comparisons are unfair by construction.

That said, 1.1x to 1.4x is not nothing. At LLM pretraining budgets it is a large amount of compute, and for teams that retrain models frequently on moderately sized architectures, exactly the regime where matrix-based methods show their strongest gains, the case for moving beyond Adam is now credible. The finding that the optimal choice shifts with the data-to-model ratio (Muon wins near 1x Chinchilla, Kron and SOAP take over at 8x and beyond) is the kind of practical detail that separates benchmarks from marketing.

The schedule-free result changed how some of us think about learning rate schedules altogether. The argument that cosine decay, used in thousands of papers, is just a rougher approximation of linear decay, which itself is a stand-in for iterate averaging, is a rare case of theory cleanly explaining an empirical folk practice. One slide from the authors' talk summarized the group's mood: stop using cosine.

Our overall takeaway: after ten years, the optimizer question is genuinely open again. The answer is no longer "just use Adam", but it is also not any single successor. It depends on model size, data ratio, and how much tuning budget you have. Which is, inconveniently, the honest answer to most questions in this field.

Reading order we recommend: the Adam and AdamW papers for foundations, the Shampoo and SOAP papers for the matrix-preconditioning line, the Muon write-up for the current favourite, and Fantastic Pretraining Optimizers before you believe any speedup claim.