Free Energy Across Generative Model Families

Generative AI is often presented as a parade of model names: VAEs, normalizing flows, diffusion models, flow matching, rectified flow, Schrödinger bridges. That view is useful when you are learning the field for the first time, but it can also make the area look more fragmented than it really is.

A better map is to ask what each family is doing with probability.

Some models approximate inference. Some learn an invertible transport map. Some learn a reverse noising process. Others learn a vector field through probability space, or treat generation as a controlled stochastic path between distributions. The machinery changes, but several questions recur: how should a model represent uncertainty, how should it move mass, and what price should it pay for complexity?

That is where the language of free energy becomes useful.

This article uses free-energy accounting as a comparison device, not as a claim that one loss governs every model family. For a VAE, variational free energy is the negative ELBO under the standard sign convention. For an energy-based model, Helmholtz free energy is proportional to the negative log partition function. Schrödinger bridges have an explicit path-space KL objective. A normalizing flow used as a variational posterior remains inside the ELBO; a stand-alone flow density model uses change-of-variables likelihood. Flow matching uses vector-field regression. Thermodynamic language beyond those identities is analogical unless a particular derivation says otherwise. The terminology map is on Energy in Statistics and Machine Learning.

This is also the lens taken by Max Welling, Sirui Lu, and Lars Holdijk’s forthcoming book Generative AI and Stochastic Thermodynamics: A Tale of Free Energies.^[1] The book is interesting because it does not treat modern generative models as unrelated inventions. It organizes a large part of the field around variational free energy, stochastic dynamics, entropy, transport, and control.

That kind of map is useful because architectures change quickly. The deeper questions last longer.

A comparison rule in plain language

The phrase “free energy” can sound more mysterious than it needs to.

For this comparison, a useful heuristic is to ask how a method balances fitting observations against the cost of the explanation. This is not a definition of free energy. The exact definition depends on the model and convention.

A model should explain the data. But explanations are not free. A latent representation can become too complex. A posterior can drift too far from a prior. A path through probability space can be inefficient. A stochastic process can deviate too much from its reference dynamics. The exact objectives, or an explicit accounting analogy, can make those costs visible.

The useful way to compare these families is not by asking which one is “best.” It is by asking what each family actually optimizes or constrains. The formulas differ even when the methods address related questions about fit, probability transport, and uncertainty.

The table is deliberately simplified. It does not make VAEs, flows, diffusion models, flow matching, and Schrödinger bridges identical. It shows why they are useful to compare while keeping their objectives distinct. Each family specifies a different combination of fit, structure, transport, uncertainty, and constraints.

This is not only a philosophical point. It changes how you read model families.

A VAE is not just an encoder and decoder. It is a compromise between reconstruction and regularization. A normalizing flow is not just an invertible neural network. Its density is computed by the change-of-variables formula. A diffusion model is not just a denoiser. It learns reverse-time dynamics across a sequence of noisy distributions. Flow matching is not just a faster sampler. It learns a vector field along a chosen probability path. A Schrödinger bridge is not just a fancy diffusion variant. It asks how to move between distributions while staying close to a reference stochastic process.

Once the shared questions and distinct objectives are separated, the field becomes less noisy.

VAEs: free energy in its most visible form

A variational autoencoder is the easiest place to see the free-energy view because the tradeoff is written directly into the objective.

Kingma and Welling’s VAE paper introduced a scalable stochastic variational inference method for directed probabilistic models with continuous latent variables and intractable posterior distributions.^[2] The key move was to make a variational lower bound trainable with standard stochastic gradient methods using the reparameterization trick.

That sounds formal, but the structure is simple. A canonical VAE learns an encoder, a decoder, and an approximate posterior while using a specified latent prior, often a fixed standard Gaussian. The encoder maps data into a distribution over latent variables. The decoder maps latent samples back into data. The objective rewards good reconstruction while penalizing latent distributions that move too far away from the prior.

The ELBO, or evidence lower bound, is the training objective behind this tradeoff. In plain language, the ELBO says: preserve enough information to reconstruct the data, but keep the latent space regular enough that sampling still works.

That is why the VAE is more than an old generative model. It is a clean demonstration of the basic accounting problem. If the model is too constrained, reconstructions are weak. If the model is too unconstrained, the latent space becomes less useful. The useful model sits between those extremes.

This is the VAE lesson: good generative modeling is not only about fitting the data. It is about fitting the data while controlling the cost of the explanation.

Normalizing flows: transport with exact accounting

Normalizing flows start from a different premise.

Instead of approximating an intractable posterior with a simpler one, a flow transforms a simple density into a more complex density through a sequence of invertible maps. Rezende and Mohamed introduced normalizing flows as a way to construct flexible approximate posterior distributions by applying invertible transformations to an initial density.^[3]

As a generative density model, a normalizing flow is a learned transport map. It starts with something easy to sample from, usually a Gaussian, and warps it into something that looks like the data distribution. Real NVP is a canonical early architecture for this direct density-modeling use.^[10]

The appeal is exact likelihood. Because the maps are invertible, the change-of-variables formula tells you exactly how density changes. When the map stretches or compresses space, the log determinant of the Jacobian accounts for that change.

That is the accounting analogy here. The exact mathematics is the change-of-variables formula: the Jacobian determinant records how the map expands or contracts volume. It is useful to compare this with free-energy tradeoffs, but it is not itself a universal free-energy objective.

The role matters. When a flow parameterizes a variational posterior, it is optimized inside the ELBO and the variational-free-energy connection is exact. When a flow directly models the data density, it is usually trained by change-of-variables likelihood. Calling that objective free energy requires the relevant information-theoretic or thermodynamic convention to be stated.

The cost is architectural. Invertibility and tractable Jacobians constrain what the network can be. This is one reason normalizing flows have not replaced diffusion models for high-fidelity image generation. But flows remain important because exact likelihood is not a minor feature. In density estimation, anomaly scoring, likelihood-ratio problems, and some scientific modeling settings, the ability to evaluate density directly can matter more than visual sample quality.

The lesson from flows is different from the lesson from VAEs. A VAE makes the inference tradeoff visible. A flow makes the transport accounting visible.

Diffusion: generation as learned reverse dynamics

Diffusion models approach the problem by turning generation into a time-indexed process.

The early diffusion paper by Sohl-Dickstein, Weiss, Maheswaranathan, and Ganguli was explicitly inspired by nonequilibrium statistical physics.^[4] The idea was to slowly destroy structure in the data through a forward noising process, then learn a reverse process that restores structure. Ho, Jain, and Abbeel later made denoising diffusion probabilistic models central to modern image generation by training a weighted variational bound connected to denoising.^[5]

The forward process is easy. Add noise to the data until the structure disappears. The reverse process is hard. Learn how to move from noise back toward the data distribution.

Score-based generative modeling gives this idea a clean continuous-time form. Song and collaborators describe a stochastic differential equation that gradually transforms the data distribution into a known prior, and a reverse-time SDE that transforms the prior back into the data distribution.^[6] The reverse dynamics depend on the score, which is the gradient of the log-density.

A score is a local direction of increasing probability. If the model learns that direction at many noise levels, it can start from noise and move step by step toward structure.

This is where the thermodynamic language starts to feel natural. Diffusion models involve forward and reverse processes, noise schedules, path probabilities, entropy, and the cost of restoring structure. The model is not simply generating an output. It is learning a reverse dynamics.

A careful point matters here. The DDPM variational bound includes per-time KL terms between the exact forward posterior conditioned on the original data and the model's learned reverse transition, together with boundary and reconstruction terms. Specified continuous-time formulations admit path-space KL or control identities. Many implementations instead optimize a simplified denoising or score loss. The exact relationship therefore depends on the stated objective and derivation.

Flow matching and rectified flow: making the path explicit

Diffusion models work well, but they often require an iterative path from noise to data. That raises a natural question: can we learn better paths?

Flow matching answers this question by learning a vector field along a probability path. A vector field tells each point where to move and how fast. Lipman and collaborators showed that flow matching can train continuous normalizing flows by regressing vector fields along chosen probability paths, including paths related to diffusion and optimal transport.^[7]

Rectified flow pushes a related idea. Liu, Gong, and Liu proposed learning an ODE that transports between distributions along straighter paths, with the goal of reducing discretization cost and improving sampling efficiency.^[8]

The intuition is geometric. If the path from noise to data is winding and stochastic, sampling may require many steps. If the path is smoother or straighter, fewer numerical steps may be enough. The model family changes because the path has become part of the design.

This is one reason flow matching and rectified flow feel like more than incremental variants. They move attention from endpoint quality to path quality. The question is not only whether the final sample looks right. The question is how efficiently and stably the model gets there.

The comparison still asks a useful question about route cost. Flow matching itself is trained by vector-field regression along a chosen probability path, so any free-energy identity must come from an additional derivation rather than from the name of the method.

Schrödinger bridges: when the path itself is the object

Schrödinger bridges are the most abstract part of this map, so they deserve a plain definition.

A Schrödinger bridge asks how to move one probability distribution into another through stochastic trajectories while staying close to a reference process, often Brownian motion. It is an entropy-regularized optimal transport problem on path space.

Unpack that sentence piece by piece.

Optimal transport asks how to move probability mass from one distribution to another. Entropy regularization keeps the solution from becoming too rigid. Path space means we care about full trajectories, not only starting and ending points. A reference process gives the system a baseline dynamics that it can deviate from, but not without paying a cost.

De Bortoli, Thornton, Heng, and Doucet connected Schrödinger bridges to score-based generative modeling, framing diffusion Schrödinger bridges as a way to build finite-time diffusion processes for generative modeling.^[9]

The mathematics is sharper here than in the flow analogy. Schrödinger bridges minimize relative entropy on path space while matching endpoint marginals. That KL-control objective is exact. Calling a derived quantity free energy still requires an explicit temperature and sign convention. Generation is not only about the endpoint distribution; it can also be about the path.

This is conceptually similar to type theory, where different proofs of the same statement need not be treated as identical. The theorem is the endpoint, but the proof still carries structure. Likewise, two generative models may target the same distribution while taking different paths through probability space. Those paths matter because they encode efficiency, stability, inductive bias, and control. In that sense, a generative model is not just a distribution. It is a program for producing one.

That matters in domains where trajectories have meaning. In robotics, the path of an action matters, not just the final pose. In scientific simulation, the evolution of a system matters, not just the final state. In inverse problems, the route from prior uncertainty to posterior structure can encode constraints, uncertainty, and cost.

Schrödinger bridges are not the simplest production tool. They are mathematically heavier than VAEs, normalizing flows, and many diffusion pipelines. But they are conceptually important because they illustrate one research direction beyond generating samples: controlling stochastic processes.

What this map explains, and what it does not

The free-energy view gives structure to a field that can otherwise look like a stream of unrelated paper titles. It explains why VAEs, flows, diffusion models, flow matching, and Schrödinger bridges keep touching the same ideas: inference, transport, entropy, uncertainty, cost, and control.

But the framing has limits. It is useful because it connects the training and transport side of generative modeling. It is not a substitute for evaluation, calibration, safety, or domain-specific validation.

The free-energy view tells us more about training and generation than evaluation. It helps explain how models learn distributions and move through probability space. It does not, by itself, tell us whether a generated molecule is synthesizable, whether a forecast is calibrated, whether a robot action is safe, or whether a generated proof is correct.

As generative models move deeper into science, engineering, robotics, forecasting, and software systems, sample quality is no longer enough. We need models that can be controlled, evaluated, constrained, and integrated into real workflows.

This is why the free-energy lens is useful but not final. It gives a serious language for inference, uncertainty, path cost, and control. It does not eliminate the need for domain-specific evaluation.

Conclusion

Generative AI is easier to understand when we stop treating model families as isolated inventions.

VAEs show the tradeoff between reconstruction and regularization. Normalizing flows show how exact likelihood can be preserved through invertible transport. Diffusion models show how generation can be framed as learned reverse dynamics. Flow matching and rectified flow make the probability path itself more explicit. Schrödinger bridges show how generation can become controlled stochastic transport on path space.

The questions recur, but the objectives remain distinct. VAEs combine a data-fit term with latent KL regularization. Normalizing flows use change-of-variables likelihoods. Diffusion and score models use stated variational, denoising, or score objectives. Flow matching regresses a vector field along a chosen path. Schrödinger bridges minimize path-space relative entropy subject to endpoint constraints.

That is why the free-energy view is valuable. It is not the whole answer, and it does not explain every generative model. But it gives a durable map of a field that often looks more chaotic than it is.

The next generation of generative systems will not be judged only by how realistic their samples look. They will be judged by whether they can control the generation process, quantify uncertainty, obey constraints, reduce cost, and justify their outputs in settings where mistakes matter.

Free energy is useful comparison language only when each exact objective, sign convention, and analogy is identified. Used that way, it connects inference, uncertainty, transport, and control without erasing the differences among the methods.