A model that knows when it is unsure: notes on Energy-Based Models

Qminers Reading Group, Session 11. This post follows the format of our internal sessions: what the papers say, and what we think about it.

Session 11 covered energy-based models (EBMs), a paradigm that approaches prediction from an unusual angle: instead of computing an answer, the model searches for it. Two properties caught our attention. First, EBMs can allocate variable compute to a single prediction. A hard question gets more processing than an easy one, which standard feedforward networks cannot do. Second, and more unusual for deep learning, an EBM reports how confident it is in its own answer, by construction rather than as an afterthought.

What the papers say

We read three papers spanning two decades: LeCun et al.'s 2006 tutorial that defined the framework, Du and Mordatch's 2019 paper on implicit generation, and the 2025 paper on Energy-Based Transformers that tests whether the idea scales.

The core concept fits in one paragraph. A standard classifier maps an input X directly to an output Y. An EBM instead learns an energy function E(Y, X) that scores how compatible a candidate answer Y is with the input X. Low energy means a good match. Prediction becomes a search problem: start from a random guess and follow the gradient of the energy function downhill until you land in a valley. The energy value at the bottom doubles as a confidence measure, and the Boltzmann distribution turns energies into proper probabilities when needed.

Training works by shaping this landscape. The contrastive approach pushes energy down at real data points and up at samples the model itself generates. Sampling uses Langevin dynamics, which is gradient descent on the energy with noise injected at each step; run it long enough with a small enough step size and the samples probably follow the model's distribution.

The 2019 paper delivered a result we found genuinely striking. On out-of-distribution detection, where a model should recognize inputs unlike anything it was trained on, the EBM was the only model in the comparison that performed better than chance, with an average AUROC of 0.62 against 0.47 for PixelCNN++ and 0.42 for Glow. Standard generative models confidently assign high likelihood to data they have never seen. The EBM notices that something is off.

The 2025 Energy-Based Transformers paper scales the recipe to language modelling and reports that EBTs improve with data, batch size, and depth at a faster rate than a strong Transformer++ baseline, up to 35% faster with respect to data. It also shows that extra gradient steps at inference help most on out-of-distribution inputs, and that selecting the lowest-energy candidate from several samples improves results further, with the gap growing as training progresses.

What we think

The 2025 paper is solid, but it wraps simple mechanisms in heavy branding. Our presenter's glossary, which got the biggest laugh of the session, translates the terminology:

What we liked. This is finally a method that gives you calibrated confidence in individual predictions, not just a softmax score that saturates near 1.0 regardless of merit. And EBMs compose well with the rest of the toolbox: chain-of-thought prompting, reasoning trained via RL, or VAE-style models for vision. Nothing about the energy framing forces you to abandon what already works.

What we did not like. The experiments run at model sizes far below the frontier, so the headline scaling claims rest on extrapolation. The training is finicky; the authors themselves write that hyperparameters are extremely important and can be highly sensitive towards performance, which in our experience is a warning label, not a footnote. And the scaling comparisons flatter the method: a 35% data-efficiency edge at small scale says little about behaviour at 100x the compute.

Why this matters to us

Quantitative research lives and dies on knowing when a model should not be trusted. A prediction paired with a native, well-behaved confidence estimate is worth more than a slightly more accurate prediction with no such signal, especially when the input distribution shifts, which in financial data is not an edge case but the normal state of affairs. EBMs are not production-ready at scale, and the hyperparameter sensitivity alone would rule them out today. But the direction, models that report their own uncertainty and spend compute where it is needed, is one we want on our radar. 

Interested in the papers? Start with the LeCun tutorial for the framework, then the 2019 paper for the training recipe, then the 2025 paper for the scaling debate.