- Statistical Mechanics, especially non-equilibrium stat-mech
- Reinforcement Learning
- Neuroscience and Predictive Coding
- Dynamic systems and (stochastic) differential equations
- Information theory
- Variational inference methods
- Path-integral formulations and the Principle of Least Action
- Embodied cognition
- Bayesian inference, action under uncertainty
- Clinical psychiatry and computational corelates of pyschopathology
For this post, don’t worry about that list, as I’ll introduce concepts as we need them!
To begin, let’s look at some motivating ideas, which we’ll keep coming back to, with more and more formalism to back them up.
You: a thing in the world
You are a thing (fully enlightened Buddhists with no sense-of-self don’t @me yet please, it’s just a starting point) and you live in a world which you do not fully understand (or at-least, it should feel that way, if you’re paying any attention to how crazy things are right now).
You get a constant stream of new information through your senses, and your brain somehow needs to use this information-deluge to do things like make exceptional scrambled eggs, renew your driver’s licence, floss your teeth, and other crucial survival-skills.
Our first big insight into the FEP is that if you want to keep doing the survival thing, you should care not about the sense data itself, but about the stuff out there that causes it - you need to keep your eye on the causal prize! The sense data is just receptors firing more or less often. What matters is that you have receptors that reliably fire in a pattern that tells you useful things like:
- there’s a tiger there and she looks pissed
- it smells like chocolate cookies have just come out the oven, so you might get fed soon
- the molten chocolate cookie you just inhaled is scalding your oesophagus and causing long-term scarring
So your sense-data isn’t random - it has external causes - but it’s not perfectly reliable either. It’s a noisy connection, and ambiguity abounds:
- is that a burglar standing in the corner of your dark room, or did someone leave a coat hanging on your hatstand?
- is that the rustling of the wind in the leaves, or another, even scarier tiger stalking you?
- is the pressure of your ‘deep-tissue sports-massage’ a welcome relief from stiffness, or categorical proof that your masseuse is a sadist?
In the time you read all of the above, you breathed several times, each of your cells used some ATP, some cells died or were phagocytosed, the state of your brain changed and a bunch of different neurons fired, and yet You are still a thing in the world. From this I infer that you did not suddenly dissolve into an unremarkable puddle of goo in the preceding 30 seconds. This is our second big insight into the FEP: as a system, many little things can change (you are dynamic), but you must keep yourself tightly bound into a larger pattern - you can be micro-random but must be macro-stable!. Friston often refers to this as possessing an ‘attracting set’ - a set of states that all of your bizarre chemical processes can wiggle around in and between, but not out of.
There are lots of ways you could configure all of the atoms that currently constitute ‘you’. Unfortunately, the vast majority are unworkable, probably because the most likely arrangement of those atoms is spread in a thin mist somewhere between here and Neptune. The small subset of all possible ways to configure yourself that keeps you ‘you’ is your phenotype, and homeostasis is basically the process of keeping yourself in those states through feedback and self-regulation. To survive long-term, you need to have a high probability of occupying those states that are compatible with life (and a low probability of becoming a thin mist somewhere between here and Neptune)
Let’s make a concrete example out of temperature…
It’s getting hot in here (and that’s surprising)
Your body keeps itself near 37 degrees (Celsius), all the time 5
If you were suddenly to sense a temperature of 800 degrees, that would be surprising. And bad. As an organism, 800 degrees is not compatible with your continued existence, and is not something your phenotype is equipped to deal with. In this sense, your body is implicitly making a claim that there is a low probability of you experiencing 800 degree heat, because if that were not the case, you would have a different body. Since 37 degrees seems cosy enough, your body itself is an implicit expectation of a high probability of sensing that temperature 6 .
So your evolved biology ‘expects’ 37 degrees , and when it doesn’t get it it’s surprised. This ‘surprisal’ is actually a formal term from information theory, where we quantify the amount of surprise as the negative natural logarithm of the probability of the observed outcome:
− ln P ( X = x ) - \ln P(X = x) − ln P ( X = x )
All this is saying is that the lower the probability we expect for an event, the more surprised we should be if we do in-fact observe that event, and conversely, if we think there is a high probability of an event happening, we should not be very surprised to see it occur. If you enter the lottery, and I tell you you’ve won, you’d be really really surprised, because you know the probability of that is small. If I tell you you’ve lost, you’re not really surprised at all, as that was always the likely outcome.
Let’s drill a little deeper into this business of sensing something and - on the basis of that sensation - forming accurate beliefs about the temperature of the environment and your body. It’s worth saying: you don’t have a nice digital-thermometer organ attached somewhere to your body which your brain can just look at. You have millions of tiny sensory receptors, which fire because of the energetic bumping and jostling of atoms hitting the receptor. For a temperature receptor, when it’s hotter, the atoms hitting it have a higher average energy, which makes it more likely that the neuron the receptor is attached to is depolarised and fires an action potential (“spikes”). We can roughly reason that in hotter environments, our temperature receptors are firing more often (but only on average - it’s still a “noisy” signal), and so maybe if our brain counted the number of spikes in a certain time, it could learn a mapping from the state of the sensory data it receives to the probable temperature of the environment.
All Reality is Virtual Reality
This gives us our third big clue about the FEP: we don’t directly experience the environment, only the noisy sensations that correlate with it - we’re one layer removed from the world, exploring the virtual reality of our sense data and inferring things about the reality behind it. This is really the jumping off point for the FEP: as an organism, we need to minimise our surprisal (we don’t want to find ourselves in 800 degree heat, and do want to find ourselves at 37 degrees, with high probability), but we only have access to our noisy sense data, and we don’t know what causally determines our temperature in the environment.
Getting Bayesian
As organisms, we want to update our beliefs about the true state of the environment, given some sense data as evidence. That’s right, it’s time for 7
(obligatory link to Yudkowsky on Bayes Theorem. If this is new, read this now)
Bayes theorem tells us the optimal way to update our beliefs given some new evidence. Doing this is sometimes called Bayesian inference, or Bayesian updating, or - controversially - as “not making up your beliefs”.
A brief note on integral notation: I will often write integrals like this: $ \ \int_<>^<>$ with the $dx$ at the front of the integral. If you're used to seeing it at the back, like this: $ \ \int_<>^<> f(x)\ dx$ don't freak out, they're exactly the same. Also, I'm going to leave the bounds of integration out for now. We can add them in if we need them later
What we want is to update our belief about the state of the environmental temperature, given our current sensory data. Let’s write the prior probability that our temperature is 37º as $P(T)$, and the probability that we receive 40 incoming neural impulses in a given time as $P(S)$. We want to know the updated probability that our temperature is 37º, given that we sense 40 neural spikes:
P ( T ∣ S ) P(T|S) P ( T ∣ S )
So our hypothesis is that the temperature is 37 degrees, and the evidence we’re using to evaluate it is the 40 spikes we received in the last few seconds.
P ( T | S ) = P ( S | T ) P ( T ) P ( S ) P\left( T \middle| S \right) = \frac
The $\blue$ term seems to cause trouble, so I'm going to spend a little extra time explaining it here, using the example of modelling the weather today based on yesterday's weather: The probability of it being sunny today, $P(\text)$ can be thought of as the sum of the two possibilities - it could be sunny today, given that yesterday was sunny, or it could be sunny, given that yesterday was **not** sunny: $$P(\text) =\\ P(\text|\text)P(\text) +\\ P(\text|\text)P(\text)$$ Instead of the sum of these two complementary outcomes, we could sum over each different possibility: $$P(\text) =\\ P(\text|\text)P(\text) +\\ P(\text|\text)P(\text) +\\ P(\text
Based on the box above, we want to write $P(S)$ like this:
P ( S ) = ∑ i P ( S ∣ T i ) P ( T i ) \blue> P ( S ) = i ∑ P ( S ∣ T i ) P ( T i )
but we must remember that, unlike our discrete weather states above, temperature is a continuous variable (it’s just a real number, which can take any value). Therefore, we replace the sum over states with an integral over all temperature values:
P ( T | S ) = P ( S | T ) P ( T ) ∫ P ( S | T ) P ( T ) dT P\left( T \middle| S \right) = \frac
And there you have it. All you need to do to perfectly understand the world is know the probability of experiencing some sensory data given a particular environmental state, and evaluate the probability of experiencing that sensory data under every possible hypothetical temperature. Very simple stuff!
Okay, so that integral is intractable, which is formal mathematical language for ‘Wolfram Alpha can’t compute it’. This is a problem for the FEP, and much of the rest of our effort will go into getting around that integral. When we can’t get an exact result, we turn to approximate methods. We’re going to manipulate our equation above until it’s in a form that allows us to do approximate Bayesian inference.
Approximating your posterior
We want to find the posterior probability (the probability we get after applying a Bayesian update) that:
P ( T = 37 ∣ S ) P(T=37|S) P ( T = 37∣ S )
It would really help us control our body temperature if we understood the causal influences determining it. Intuitively, we want a world-model that predicts that if we move closer to a heat-source, we’ll get sensory data that tells us we’re getting hotter.
The internal model that encodes how we expect our sense data to correlate with our environmental states is called a Generative Model, and is referred to by Friston as the G-density. Our G-density tells us the joint probability of experiencing some sense data and a corresponding environmental state: P ( T , S ) P(T,S) P ( T , S )
You can imagine this as a big table that assigns a probability to each possible pair of values. A good model in this case would assign a high probability to the combination:
P ( high temp, lots of spikes ) P(\text) P ( high temp, lots of spikes )
and a low probability to things like:
P ( high temp, few spikes ) P(\text) P ( high temp, few spikes )
We will also want a function that represents our current ‘best-guess’ as to the causes of our sensory input, which we’ll call the R-density (for Recognition): q ( T ) q(T) q ( T )
This is just a probability distribution over the state of the environment, and we’re using $q$ instead of $p$ to remind us that it’s a distribution we’re guessing at.
We now need another fancy piece of information theory: the Kullback-Liebler (KL) Divergence 8
The KL Divergence just tells us how different two different probability distributions are:
We want the KL divergence because we want to know how close or far away our best guess is to the true posterior belief if we could compute that ugly integral
We write the KL-divergence between our guess $q(T)$ about the environment and the optimal posterior belief about the environment from Baye’s theorem:
D KL ( q ( T ) ∣ ∣ P ( T ∣ S ) ) = ∫ dT q ( T ) ln q ( T ) P ( T | S ) D_>(q(T)||P(T|S))\ = \ \int_<>^<>\ q( T )\ln\frac
which we rearrange using the property of the logarithm to:
= ∫ d T q ( T ) [ ln q ( T ) − ln P ( T ∣ S ) ] = \ \int_<>^<>> = ∫ d T q ( T ) [ ln q ( T ) − ln P ( T ∣ S )]
Now, any R-density (that is, any $q(T)$) which minimises this KL divergence, must be a good approximation of our true posterior. The only problem is that we don’t know the true posterior (that’s the thing we’re trying to work out in the first place), and so can’t simply guess a $q(T)$ to see if it minimises the KL Divergence, because we don’t know what we’re comparing it to.
To get around this, we’ll rewrite the true posterior
P ( T ∣ S ) \orange P ( T ∣ S )
using some pretty basic probability theory identities:
P ( T ∣ S ) = P ( S ∣ T ) P ( T ) P ( S ) \orange
If we take the natural log of both sides:
ln P ( T | S ) = ln P ( T , S ) P ( S ) = ln P ( T , S ) − ln P ( S ) \ln<\orange
> = \ln<\frac<\blue >> ln P ( T ∣ S ) = ln P ( S ) P ( T , S ) = ln P ( T , S ) − ln P ( S ) Plugging this expression for $P(T | S)$ into our KL Divergence we get: D KL ( q ( T ) ∣ ∣ P ( T ∣ S ) ) = ∫ d T q ( T ) [ ln q ( T ) − ln P ( T , S ) + ln P ( S ) ] D_>(q(T)||\orange)\ = \int_<>^<> + \ln<\purple Combining the first two $\ln$ terms: D KL ( q ( T ) ∣ ∣ P ( T ∣ S ) ) = ∫ d T q ( T ) [ ln q ( T ) P ( T , S ) + ln P ( S ) ] D_>(q(T)||\orange)\ = \int_<>^<><\blue Something you may notice about this is that we now have our KL divergence in terms of only our R-density, $q(T)$, and our G-density, $P(T,S)$, plus a ‘surprisal’ term $\ln P(S)$. This looks like progress! Remember, our R density is just our current best guess about the true state of the environment, the G-density is our internal model of how sensory data and environmental states correlate, and surprisal tells us how unexpected some data was! We can pull the $\ln P(S)$ out from under the integral because integration is linear we’re integrating over $dT$ and so $\ln P(S)$ acts like a constant (it’s just a number): D KL ( q ( T ) ∣ ∣ P ( T ∣ S ) ) = ∫ dT q ( T ) ln q ( T ) P ( T , S ) + ln P ( S ) ∫ d T q ( T ) D_>(q(T)||P(T|S))\ = \int_<>^<>\ q(T)\ \ln<\frac Furthermore, we have the requirement that ∫ dT q ( T ) = 1 \green<\int_<>^<>\ q \left( T \right) = 1>> ∫ dT q ( T ) = 1 which just says that the total probability must add to 1. Using this, we get: D KL ( q ( T ) ∣ ∣ P ( T ∣ S ) ) = ∫ dT [ q ( T ) ln q ( T ) P ( T , S ) ] + ln P ( S ) × 1 D_>(q(T)||P(T|S))\ = \pink<\int_<>^<> \text\left\lbrack q \left( T \right)\ln<\frac F ≡ ∫ dT [ q ( T ) ln q ( T ) P ( T , S ) ] F\ \equiv \pink<\int_<>^<>\text\left\lbrack q\left( T \right)\ln<\frac D KL ( q ( T ) ∣ ∣ P ( T | S ) ) = F + ln P ( S ) D_>(q(T)||P\left( T \middle| S \right)) = F + \ln If you haven’t guessed already, the F we defined above is ‘free energy’!!. Now we have a key result in the Free Energy literature, one which Friston refers to all the time: Free energy is an upper bound on surprisal! You can see this if I tell you that the KL divergence has the property that it is always greater than or equal to zero: D KL ( q ( T ) ∣ ∣ P ( T | S ) ) ≥ 0 D_>(q(T)||P\left( T \middle| S \right)) \geq 0 D KL ( q ( T ) ∣∣ P ( T ∣ S ) ) ≥ 0 ⇒ F + ln P ( S ) ≥ 0 \Rightarrow F\ + \ \ln Free energy is the surprisal an organism experiences upon sampling some data, given a generative model. More specifically, free energy is the surprisal an organism experiences upon sampling some data, given a generative model. If you’re up for it, you should try matching those words to the parts of the equations that encode them! I’ll even put the full equation right here in-front of you, with no distracting colours: F ≥ − ln P ( S ) F\ \geq \ - \ln F ≥ − ln P ( S ) ∫ dT [ q ( T ) ln q ( T ) P ( T , S ) ] ≥ − ln P ( S ) \int_<>^<>\text\left\lbrack q\left( T \right)\ln<\frac So far, we have found that this quantity ‘free energy’ is an upper bound on surprisal, and we notice that minimising it means we are approximating the true posterior P ( T ∣ S ) P(T|S) P ( T ∣ S ) We’re going to need to massage this equation for F a bit more to see if something useful pops out. Again, using the linearity of integration and the properties of the logarithm, we have: F ≡ ∫ dT [ q ( T ) ln q ( T ) P ( T , S ) ] F\ \equiv \int_<>^<>\text\left\lbrack \\ q\left( T \right)\ln<\frac From statistics, we say that the expected value (or average) of a Random Variable $X$ is the sum over all values that variable can take, times the probability it takes that value: E [ X ] = ∑ i x i P ( X = x i ) E\lbrack X\rbrack\ = \ \sum_^<> If you need a sanity check, let's compute this for a fair 6-sided die. The probability for each face landing up is $1/6$ The values our random variable $X$ can take are $X = \$ $$E[X] = \sum_^\fracx = 3.5$$ which is indeed the correct average value for our die 🎲 Replacing the sum with an integral in the continuous case: E [ X ] = ∫ R x P ( x ) d x \displaystyle \operatorname [X]=\int _ <\mathbb We also have the entropy of a system (you should convince yourself that this is the “expected value” of the surprise $-\ln P(x)$): H ( X ) = − ∑ i = 1 n P ( x i ) log P ( x i ) \mathrm (X)=-\sum _^ <\mathrm (x_)\log \mathrm (x_>) H ( X ) = − i = 1 ∑ n P ( x i ) lo g P ( x i ) Entropy can be a somewhat tricky term, but I think this way of thinking about it is fairly intuitive: it’s just the amount you expect to be surprised by a given probability distribution. Some distributions are very tightly clustered around their average values, and so they are very unsurprising, hence low entropy. The opposite of this are the so-called maximum-entropy distributions, which means every sample is maximally surprising. An example entropy calculation: A coin is a maximum entropy system, because it's maximally unpredictable, you don't know if it's going to come up heads $h$ or tails $t$ $$H(coin) = -P(h)\log P(h) - P(t)\log P(t)$$ $$H(coin) = -0.5 * \log(0.5) - 0.5\log(0.5)$$ $$H(coin) \approx 0.3$$ Now imagine if we had an unfair coin that came up heads with a probability of 90%: $$H(coin) = -P(h)\log P(h) - P(t)\log P(t)$$ $$H(coin) = -0.9\log (0.9) - 0.1\log (0.1)$$ $$H(coin) \approx 0.14$$ So the more certain we are about an outcome, the lower the entropy! Equipped with these ideas, we define an energy-like function $\mathbf(T,S)$: E ( T , S ) ≡ − ln P ( T , S ) \red<\mathbf<Ε>(T,S)>\ \equiv \red> E ( T , S ) ≡ − l n P ( T , S ) And this allows us to rewrite our free energy term F as (remember the $\mathbf(T,S)$ is defined as the negative of the log above, so we get a positive in the end): F = ∫ dT q ( T ) ln q ( T ) − ∫ dT q ( T ) ln P ( T , S ) F\ = \purple<\int_<>^<>\ q(T)\ \ln>> -\int_<>^<>\ q(T)\red<\ln Now, if we look at this formula (check that you see this from above) it looks like it’s saying that ‘free energy’ is equal to an average (expected value) of the energy-like function, minus something that looks a little like a continuous version of the entropy 9 . This version of the formula is something you’ll hear Friston refer to often, because it’s analagous to the Helmholtz free energy from thermodynamics/statistical mechanics: $F = U-TS$. The Helmholtz free energy is defined as the difference between the internal energy $U$ and the entropy $S$ of the system, multiplied by the temperature $T$. Here the term ‘free energy’ acquires some physical sense, being the quantity of energy in our system that is available to do useful work. Here I’ve split the $+$ into two negatives to emphasise we have a potential energy minus the entropy. F = ∫ dT q ( T ) E ( T , S ) − ( − ∫ dT q ( T ) ln q ( T ) ) F = \int_<>^<>\ q\left( T \right) \mathbf\left( T,S \right) > - \left( - \int_<>^<>\ q \left( T \right)\ln Having gone through all of the above, we can now state the FEP explicitly: The Free Energy Principle states that a system in nonequilibrium steady state with its environment must minimise its free energy We’ll need another few posts to unpack everything there, but at minimum we now know what the ‘free energy’ term is and why minimising it might be useful! We motivated the Free Energy Principle with three big ideas about living organisms: Additionally, we learned: As an experiment in what Michael Nielsen and Andy Matuschak dub the Mnemonic Medium, I’ve made a small deck of Anki cards that complement this post. If you use them, they’ll make the tricky vocabulary we’ve developed here into something you have stored in your long-term memory for easy use. You can download them here In the next post, we’ll take the background we developed here and build on it. We’ll take a deeper look at the R and G densities and some simplifying assumptions that allow us to write neat versions of them. The result will show the deep connection between the Free Energy Principle and Predictive Coding in the brain. If something here doesn’t make sense, is clearly wrong, or got you interested, let me know here or on twitter @jnearestn If you don’t want to wait for the next post, The Free Energy Principle for Action and Perception: A Mathematical Review is the best place to start out on your own, and I owe it a great deal in helping write this post! Special thanks to Gianluca for his detailed comments on drafts of this post To live is to risk it all; otherwise you're just an inert chunk of randomly assembled molecules drifting wherever the universe blows you.Another form of the free energy:
Conclusion, Summary, and Anki Cards
