Energy in Statistics and Machine Learning

Why This Matters

The word energy names several different mathematical objects in statistics and machine learning. They share useful analogies, but they are not one method:

• an energy-based model assigns a scalar score to a configuration and turns that score into a probability density;
• free energy may mean a log-partition quantity or a variational objective;
• energy distance compares two probability distributions through expected pairwise distances;
• distance covariance measures dependence through centered distances;
• the energy score evaluates a univariate or multivariate probabilistic forecast;
• Langevin methods use a potential energy whose gradient drives a stochastic process;
• computational energy is literal electricity consumed by training or inference.

Confusing these meanings causes real errors. An energy-distance two-sample test does not train an energy-based model. A VAE's variational free energy is not the same object as an EBM's log partition function. A classifier's OOD energy score is not the proper energy score used in probabilistic forecasting.

This page is the map. Follow the links for the full mathematics.

Quick Disambiguation

The common thread is weaker than the shared word suggests. Several meanings involve a scalar cost, distances, or a probability normalization. The formulas, assumptions, and inferential claims remain different.

1. Energy as an Unnormalized Probability Model

Definition

Energy-Based Model$p_{\theta }(x)\propto e^{-E_{\theta }(x)}$

An energy-based model (EBM) specifies a scalar function $E_\theta:\mathcal X\to\mathbb R$ and defines

$p_\theta(x)=\frac{\exp(-E_\theta(x))}{Z(\theta)},\qquad Z(\theta)=\int_{\mathcal X}\exp(-E_\theta(u))\,d\mu(u).$

Lower energy gives larger density relative to other points. The partition function $Z(\theta)$ makes the density integrate to one.

Proposition

Positive Densities Admit an Energy Representation

Statement

For any constant $c\in\mathbb R$, define

$E(x)=-\log p(x)+c.$

Then $p(x)=\exp(-E(x))/Z$ with $Z=e^{-c}$. Conversely, any finite energy with finite positive partition function defines a density that is strictly positive almost everywhere on the support of the base measure.

Intuition

Taking a negative logarithm converts relative probability into an additive cost. Adding the same constant to every energy changes $Z$ but leaves the normalized density unchanged.

Proof Sketch

Substitute $E(x)=-\log p(x)+c$: $\exp(-E(x))=e^{-c}p(x)$. Integrating gives $Z=e^{-c}$, so division by $Z$ returns $p(x)$.

Why It Matters

The representation is algebraically broad, but training methods are not. Writing a density in energy form does not make a GAN objective, VAE objective, or flow likelihood into the same learning algorithm as an unnormalized EBM.

Failure Mode

If $p(x)=0$ on part of the space, the corresponding energy is $+\infty$ there. If $\int e^{-E(x)}d\mu(x)$ diverges, the proposed energy does not define a probability density.

report a correction →

For a differentiable continuous EBM,

$$\nabla_x\log p_\theta(x)=-\nabla_x E_\theta(x),$$

because the partition function does not depend on $x$. This relation connects EBMs to score matching, diffusion models, and Langevin sampling. It is a relation between gradients, not a claim that all those methods use one objective.

2. Two Meanings of Free Energy

Log-partition or Helmholtz free energy

With inverse temperature $\beta>0$,

$p_\beta(x)=\frac{e^{-\beta E(x)}}{Z_\beta},\qquad F_\beta=-\frac{1}{\beta}\log Z_\beta.$

At $\beta=1$, many ML treatments use $F=-\log Z$. This is a property of the unnormalized energy function together with its base measure: $Z$ is the normalizer required to obtain the probability model. Sign and temperature conventions vary, so check the definition before comparing papers. In equilibrium thermodynamics, Helmholtz free energy is also a macroscopic state function with a work interpretation; this page uses the probabilistic identity.

Variational free energy

For observed $x$, latent $z$, joint model $p(x,z)$, and a trial distribution $q(z)$, define

$$\mathcal F(q;x) =\mathbb E_{q(z)}[-\log p(x,z)]-H(q) =\operatorname{KL}\!\left(q(z)\,\|\,p(z\mid x)\right)-\log p(x).$$

Therefore

$$\mathcal F(q;x)\ge -\log p(x),$$

with equality exactly when $q(z)=p(z\mid x)$ almost everywhere. Under the usual maximization convention, $\mathcal F$ is the negative ELBO.

These free energies are related by variational identities, but they should not be substituted for one another without specifying the model, temperature, and optimization variable.

Watch Out

Free energy is not one universal loss

The VAE variational free energy is exactly a negative ELBO. An EBM's log-partition free energy is a normalization quantity. Normalizing flows use change-of-variables likelihoods, flow matching uses vector-field regression, and Schrödinger bridges use path-space KL control. Thermodynamic language can compare their tradeoffs, but it does not turn them into one objective.

3. Energy as a Distance Between Distributions

Let $X,X'\sim P$ and $Y,Y'\sim Q$ be independent, and let $\rho$ be a semimetric of strong negative type with finite first moments. Define

$$\mathcal E_\rho(P,Q) =2\mathbb E\rho(X,Y) -\mathbb E\rho(X,X') -\mathbb E\rho(Y,Y').$$

Then $\mathcal E_\rho(P,Q)\ge0$, and under the stated condition, $\mathcal E_\rho(P,Q)=0$ exactly when $P=Q$.

This is energy distance. It compares cross-sample distances with within-sample distances. There is no energy function $E_\theta(x)$, no partition function, and no Boltzmann density.

Energy distance can still be used as an objective for fitting a parameterized generator. That use does not by itself make the generator an energy-based model: the fitted model need not assign a scalar energy to each configuration or define a density through a partition function.

The name comes from generalized potential-energy constructions in the energy-statistics literature. The displayed formula is not Newtonian $1/r$ potential energy and should be interpreted by its statistical definition. The full page develops its sample estimators, permutation test, connection to maximum mean discrepancy, and dependence analogue.

Watch Out

Energy distance is not Wasserstein distance

Wasserstein distance minimizes expected transport cost over couplings between $P$ and $Q$. Energy distance combines independent cross-distances and within-distribution distances. Both compare probability laws through a ground metric, but their geometry and estimators are different.

4. Energy Score Versus Classifier Energy

Two ML quantities called an energy score are unrelated.

For a predictive distribution $F$ on $\mathbb R^d$, observation $y$, and independent $X,X'\sim F$, the standard $\beta=1$ energy score in loss orientation is

$$\operatorname{ES}(F,y) =\mathbb E\|X-y\| -\frac12\mathbb E\|X-X'\|.$$

Lower is better. If the true distribution is $G$, the expected excess score from reporting $F$ rather than $G$ is

$$\mathbb E_{Y\sim G}\operatorname{ES}(F,Y) -\mathbb E_{Y\sim G}\operatorname{ES}(G,Y) =\frac12\mathcal E(F,G).$$

This identity connects a proper forecast score to energy distance. The broader energy-score family replaces each norm by its $\beta$ power for $0<\beta<2$. In one dimension, the $\beta=1$ member equals the continuous ranked probability score.

For classifier logits $f_1(x),\ldots,f_K(x)$, an OOD method instead defines

$$E_T(x)=-T\log\sum_{k=1}^K\exp(f_k(x)/T).$$

That scalar is useful for ranking inputs, but it is not the multivariate proper energy score above.

5. Potential Energy in Langevin Methods

Overdamped Langevin dynamics often appears as

$$dX_t=-\nabla U(X_t)\,dt+\sqrt{2/\beta}\,dW_t.$$

Under suitable regularity and integrability conditions, its stationary density is proportional to $e^{-\beta U(x)}$. Here $U$ is a potential, so the connection to the Section 1 EBM target is exact when $\beta U=E_\theta$ up to an additive constant. In particular, $U=E_\theta$ when $\beta=1$. The stochastic differential equation is still a sampling or dynamics method, not a two-sample statistic. This stationary-law statement is about the continuous-time process. An unadjusted finite-step discretization generally introduces bias unless an appropriate limit or correction is used.

6. Literal Energy Use

In systems work, energy means physical energy consumed. For computing systems, this is usually estimated from electrical energy input and reported in joules or kilowatt-hours. That estimate needs hardware power, utilization, duration, datacenter overhead, and regional generation data. A parameter count or floating-point-operation estimate alone does not determine electrical energy use.

This meaning has no mathematical dependence on energy distance or an EBM energy function. The shared word is historical and analogical. See Energy Efficiency and Green AI for measurement, carbon accounting, and efficiency methods.

A Reliable Reading Rule

When a paper uses energy, ask four questions:

1. What are the inputs? A configuration, a probability distribution, two samples, a forecast and outcome, a stochastic path, or a hardware meter?
2. What is the output? A scalar score, normalized density, discrepancy, test statistic, objective, or physical quantity?
3. What makes zero or a minimum meaningful? Equality of distributions, independence, a posterior match, a likely configuration, or lower power draw?
4. What assumptions carry the claim? Finite partition function, finite moments, strong negative type, exchangeability, stationarity, or calibrated measurement?

Those questions disambiguate the term faster than relying on the surrounding thermodynamic vocabulary.

Common Confusions

Watch Out

Any density can be written as an energy, but not every training method is EBM training

The negative-log representation is an algebraic fact about positive densities. EBM training usually means learning an unnormalized scalar energy while handling or avoiding an intractable partition function. GANs, VAEs, flows, and diffusion models have distinct objectives and computational problems.

Watch Out

A zero sample statistic is not a population theorem

The statement $\mathcal E(P,Q)=0\iff P=Q$ concerns the population distance under its assumptions. A finite-sample estimator can equal zero by chance, and an unbiased estimator can even be negative. Inference requires a null calibration such as a permutation test.

Watch Out

Thermodynamic terminology does not supply missing assumptions

Words such as energy, entropy, temperature, and free energy can expose useful mathematical structure. They do not by themselves prove normalization, convergence, calibration, or equality of objectives.

Exercises

ExerciseCore

Problem

Classify each object: (a) $-T\log\sum_k e^{f_k(x)/T}$, (b) $2\mathbb E\|X-Y\|-\mathbb E\|X-X'\|-\mathbb E\|Y-Y'\|$, and (c) $\mathbb E_q[-\log p(x,z)]-H(q)$.

Hint

Reveal Solution

ExerciseCore

Problem

Let $p(x)$ be a strictly positive normalized density and define $E_c(x)=-\log p(x)+c$. Show that every $c\in\mathbb R$ gives the same normalized density, and compute the corresponding partition function.

Hint

Reveal Solution

References

Energy-based models and score matching:

• LeCun, Chopra, Hadsell, Ranzato, and Huang, "A Tutorial on Energy-Based Learning" (2006), Sections 2 and 3.
• Hyvärinen, "Estimation of Non-Normalized Statistical Models by Score Matching" (Journal of Machine Learning Research, 2005), Sections 2 and 3.
• Song and Ermon, "Generative Modeling by Estimating Gradients of the Data Distribution" (NeurIPS, 2019), Sections 2 and 3.

Energy statistics and scoring rules:

• Székely and Rizzo, "Energy Statistics: A Class of Statistics Based on Distances" (Journal of Statistical Planning and Inference, 2013), Sections 2 through 4. DOI: 10.1016/j.jspi.2013.03.018.
• Sejdinovic, Sriperumbudur, Gretton, and Fukumizu, "Equivalence of Distance- Based and RKHS-Based Statistics in Hypothesis Testing" (Annals of Statistics, 2013), Theorem 22.
• Gneiting and Raftery, "Strictly Proper Scoring Rules, Prediction, and Estimation" (Journal of the American Statistical Association, 2007), Section 4.3.
• Liu, Wang, Owens, and Li, "Energy-Based Out-of-Distribution Detection" (NeurIPS, 2020), Sections 2 and 3.

Free energy and stochastic dynamics:

• Pathria and Beale, Statistical Mechanics (3rd ed., 2011), Chapter 3.
• Jordan, Ghahramani, Jaakkola, and Saul, "An Introduction to Variational Methods for Graphical Models" (Machine Learning, 1999), Sections 2 and 3.
• Risken, The Fokker-Planck Equation (2nd ed., 1989), Chapters 4 and 5.

Next Topics

• Energy-Based Models: energy functions, partition functions, score matching, and sampling.
• Energy Distance and Distance Covariance: distribution comparison and nonlinear dependence from pairwise distances.