Active Inference as a Convex Markov Decision Process

Methodology

We therefore leave a convergence analysis to future work. For each method at convergence (k = K), the per-step imagined state occupancy ρ t (s) is shown at five evenly spaced timesteps t ∈ {0, ⌊H/4⌋, ⌊H/2⌋, ⌊3H/4⌋, H} = {0, 3, 6, 9, 12}.

Study Design

Results & Findings

Model-based policy optimization (MBPO) lets an agent learn a world model from environment interaction and optimize a reactive policy on imagined rollouts inside that model. The two decision processes are coupled because the variational distribution ν is used in variational model learning and in action selection through the behavior policy β = π • ν.

• Model-based policy optimization (MBPO) lets an agent learn a world model from environment interaction and optimize a reactive policy on imagined rollouts inside that model.
• The two decision processes are coupled because the variational distribution ν is used in variational model learning and in action selection through the behavior policy β.
• The agent acts using the behavior policy β = π • ν, fits its model by Variational Free Energy (VFE) minimization to obtain m = (p.
• The agent learns a generative model of the environment and selects actions by minimising an expected free energy (EFE) that scores imagined future outcomes against a.
• Implementations use a recurrent world model, an actor, and a critic trained on imagined environment interactions.

Practical Applications

Generalizing to infinite horizons is possible but comes with modeling choices. The parameter-based novelty terms in the EFE may induce complex π-dependence.

Contributions

The contributions section outlines the main findings: the structure of EFE as a convex functional, the development of a mirror-descent-inspired algorithm for EFE minimization, and the establishment of a performative policy-model pair under mild assumptions.

Episodic Active Inference

This section describes the episodic active inference framework, detailing the components of the POMDP and the behavior policy used by the agent. It explains the process of model learning and policy optimization in the context of active inference.

VFE: World Model Training

The VFE section explains the training of the world model through variational free energy minimization on real environment data. It discusses the generative model and recognition density used for inferring states.

Figures Explained

The paper’s visual material highlights the workflow and the main system components.

• Figure 1 :: Figure1: The episodic active inference setting considered in this work: A) the environment admits a true partially observable decision process, while the agent learns a latent fully observable process using variational inference. B) In contrast to standard variational inference, the two decision processes are coupled: the variational distribution ν is used both in variational model learning and in action selection through the behavior policy β = π • ν. The agent acts in the environment using the behavior policy β = π • ν for multiple episodes, then uses the data to fit its model by Variational Free Energy (VFE) minimization to obtain m = (p, ν), and uses that model to optimize its policy in imagination by minimizing the Expected Free Energy (EFE).
• 12 RLFigure 2 :: Figure 2: MD-AIF on gridworld environments. (A, top two rows) 5×5 deterministic gridworld. Left: EFE convergence (log-log scale) for MD-AIF, RL (no novelty), and EFE gradient descent, together with the theoretical O(1/k) rate (dashed). MD-AIF converges faster and to a lower value, driven by regularized dynamic programming. Right: Per-step imagined occupancy ρ t (s) at convergence (k = 100) for MD-AIF (top) and RL (bottom) at five timesteps t ∈ {0, 3, 6, 9, 12}. MD-AIF spreads probability mass broadly across the grid before concentrating toward the goal, whereas RL greedily channels mass along the direct path. Both rows share the same color scale. (B) 10 × 10 deterministic gridworld with uniform preference (pure epistemic drive). Left: Mean total-variation error TV(p, p ⋆ ) of the model fit as a function of environment steps under the interleaved model-policy loop. MD-AIF’s information-gain drive produces broader state coverage, accelerating model learning relative to RL and EFE gradient descent. AIF agents [2] plan myopically over short horizons (T = 3 exact; T = 5 via Monte Carlo with N mc = 100 samples). Right: Per-state model error TV(p, p ⋆ ) = E a∼π T TV(p(•|s, a), p ⋆ (•|s, a)) after 2500 environment steps.
• ) 5 🙂 6 :: π k+1 ← MD-AIF(π k ; m k+1 , pk+1 , T, η, q k+1 0 return π K To see this, consider the one pass of the full loop (2). Deploying π (behavior β = π•ν) in the real environment induces the history distribution D(π); the perception step returns the VFE-optimal model m ⋆ (D(π)) ∈ arg min m∈M F m; D(π) =: p π , ν π , and this refit model is what the planner sees: it supplies a linear pseudo-cost ℓ π and the augmented transition kernel P π of Lemma 6, ℓ π := ℓ m ⋆ (D(π)) , P π := p m ⋆ (D(π)) . Thus the decision-dependence is the composition π.
• Figure 4: s,a p(s ′ |s, a)π t (a|s)ρ t (s). Constant step size η = 0.05.Convergence plot. The y-axis shows Γ(ρ k ) -Γ ⋆ , where Γ ⋆ is the minimum across all three methods and all iterates. The O(1/k) reference line is fitted to MD-AIF’s initial gap: C/k with C = Γ(ρ 0 ) -Γ ⋆ . Both axes are logarithmic.
• 2 .: Deploy the resulting policy: sample E = 5 episodes of length H = 25 from the true environment by ancestral sampling (a t ∼ π t (•|s t ), s t+1 ∼ p ⋆ (•|s t , a t )).

Limitations and Cautions

A useful limitation and caution is that this article summarizes the available paper text and extracted evidence; readers should consult the source paper before treating any interpretation as definitive.

The paper’s conclusions may depend on its source selection, definitions, assumptions, and the scope of its analysis, so follow-up reading is important.

Source Paper Figures and Captions