Mutual Information, and Density in Thingspace

❦

Suppose you have a system X that can be in any of 8 states, which are all equally probable (relative to your current state of knowledge), and a system Y that can be in any of 4 states, all equally probable. total The entropy of X, as defined in the previous essay, is 3 bits; we’ll need to ask 3 yes-or-no questions to find out X’s exact state. The entropy of Y is 2 bits; we have to ask 2 yes-or-no questions to find out Y’s exact state. This may seem obvious since 2³ = 8 and 2² = 4, so 3 questions can distinguish 8 possibilities and 2 questions can distinguish 4 possibilities; but remember that if the possibilities were not all equally likely, we could use a more clever code to discover Y’s state using e.g. 1.75 questions on average. In this case, though, X’s probability mass is evenly distributed over all its possible states, and likewise Y, so we can’t use any clever codes.

What is the entropy of the combined system (X,Y)?

You might be tempted to answer, “It takes 3 questions to find out X, and then 2 questions to find out Y, so it takes 5 questions total to find out the state of X and Y.”

But what if the two variables are entangled, so that learning the state of Y tells us something about the state of X?

In particular, let’s suppose that X and Y are either both odd or both even.

Now if we receive a 3-bit message (ask 3 questions) and learn that X is in state X₅, we know that Y is in state Y₁ or state Y₃, but not state Y₂ or state Y₄. So the single additional question “Is Y in state Y₃?,” answered “No,” tells us the entire state of (X,Y): X = X₅, Y = Y₁. And we learned this with a total of 4 questions.

Conversely, if we learn that Y is in state Y₄ using two questions, it will take us only an additional two questions to learn whether X is in state X₂, X₄, X₆, or X₈. Again, four questions to learn the state of the joint system.

The mutual information of two variables is defined as the difference between the entropy of the joint system and the entropy of the independent systems: I(X;Y) = H(X) + H(Y) − H(X,Y).

Here there is one bit of mutual information between the two systems: Learning X tells us one bit of information about Y (cuts down the space of possibilities from 4 possibilities to 2, a factor-of-2 decrease in the volume) and learning Y tells us one bit of information about X (cuts down the possibility space from 8 possibilities to 4).

What about when probability mass is not evenly distributed? Last essay, for example, we discussed the case in which Y had the probabilities 1/2, 1/4, 1/8, 1/8 for its four states. Let us take this to be our probability distribution over Y, considered independently—if we saw Y, without seeing anything else, this is what we’d expect to see. And suppose the variable Z has two states, Z₁ and Z₂, with probabilities 3/8 and 5/8 respectively.

Then if and only if the joint distribution of Y and Z is as follows, there is zero mutual information between Y and Z:

Z1Y1 : 3/16	Z1Y2 : 3/32	Z1Y3 : 3/64	Z1Y4 : 3/64
Z2Y1 : 5/16	Z2Y2 : 5/32	Z2Y3 : 5/64	Z2Y4 : 5/64 .

This distribution obeys the law

P(Y,Z) = P(Y)P(Z) .

For example, P(Z₁Y₂) = P(Z₁)P(Y₂) = 3/8 × 1/4 = 3/32.

And observe that we can recover the marginal (independent) probabilities of Y and Z just by looking at the joint distribution:

P(Y1)	=	total probability of all the different ways Y1 can happen
	=	P(Z1Y1) + P(Z2Y1)
	=	3/16 + 5/16
	=	1/2 .

So, just by inspecting the joint distribution, we can determine whether the marginal variables Y and Z are independent; that is, whether the joint distribution factors into the product of the marginal distributions; whether, for all Y and Z, we have P(Y,Z) = P(Y)P(Z).

This last is significant because, by Bayes’s Rule,

P(ZjYi)	=	P(Yi)P(Zj)
P(ZjYi)/P(Zj)	=	P(Yi)
P(Yi|Zj)	=	P(Yi) .

In English: “After you learn Z_j , your belief about Y_i is just what it was before.”

So when the distribution factorizes—when P(Y,Z) = P(Y)P(Z)—this is equivalent to “Learning about Y never tells us anything about Z or vice versa.”

From which you might suspect, correctly, that there is no mutual information between Y and Z. Where there is no mutual information, there is no Bayesian evidence, and vice versa.

Suppose that in the distribution (Y,Z) above, we treated each possible combination of Y and Z as a separate event—so that the distribution (Y,Z) would have a total of 8 possibilities, with the probabilities shown—and then we calculated the entropy of the distribution (Y,Z) the same way we would calculate the entropy of any distribution:

P(Z₁Y₁)log₂(P(Z₁Y₁)) + P(Z₁Y₂)log₂(P(Z₁Y₂)) +
P(Z₁Y₃)log₂(P(Z₁Y₃)) + … + P(Z₂Y₄)log₂(P(Z₂Y₄))

= (3/16)log₂(3/16) + (3/32)log₂(3/32) +
(3/64)log₂(3/64) + … + (5/64)log₂(5/64) .

You would end up with the same total you would get if you separately calculated the entropy of Y plus the entropy of Z. There is no mutual information between the two variables, so our uncertainty about the joint system is not any less than our uncertainty about the two systems considered separately. (I am not showing the calculations, but you are welcome to do them; and I am not showing the proof that this is true in general, but you are welcome to Google on “Shannon entropy” and “mutual information.”)

What if the joint distribution doesn’t factorize? For example:

Z1Y1 : 12/64	Z1Y2 : 8/64	Z1Y3 : 1/64	Z1Y4 : 3/64
Z2Y1 : 20/64	Z2Y2 : 8/64	Z2Y3 : 7/64	Z2Y4 : 5/64 .

If you add up the joint probabilities to get marginal probabilities, you should find that P(Y₁) = 1/2, P(Z₁) = 3/8, and so on—the marginal probabilities are the same as before.

But the joint probabilities do not always equal the product of the marginal probabilities. For example, the probability P(Z₁Y₂) equals 8/64, where P(Z₁)P(Y₂) would equal 3/8 × 1/4 = 6/64. That is, the probability of running into Z₁Y₂ together is greater than you’d expect based on the probabilities of running into Z₁ or Y₂ separately.

Which in turn implies:

P(Z1Y2)	>	P(Z1)P(Y2)
P(Z1Y2)/P(Y2)	>	P(Z1)
P(Z1|Y2)	>	P(Z1).

Since there’s an “unusually high” probability for P(Z₁Y₂)—defined as a probability higher than the marginal probabilities would indicate by default—it follows that observing Y₂ is evidence that increases the probability of Z₁. And by a symmetrical argument, observing Z₁ must favor Y₂.

As there are at least some values of Y that tell us about Z (and vice versa) there must be mutual information between the two variables; and so you will find—I am confident, though I haven’t actually checked—that calculating the entropy of (Y,Z) yields less total uncertainty than the sum of the independent entropies of Y and Z. That is, H(Y,Z) = H(Y) + H(Z) − I(Y;Z), with all quantities necessarily positive.

(I digress here to remark that the symmetry of the expression for the mutual information shows that Y must tell us as much about Z, on average, as Z tells us about Y. I leave it as an exercise to the reader to reconcile this with anything they were taught in logic class about how, if all ravens are black, being allowed to reason Raven(x) ⇒ Black(x) doesn’t mean you’re allowed to reason Black(x) ⇒ Raven(x). How different seem the symmetrical probability flows of the Bayesian, from the sharp lurches of logic—even though the latter is just a degenerate case of the former.)

“But,” you ask, “what has all this to do with the proper use of words?”

In Empty Labels and then Replace the Symbol with the Substance, we saw the technique of replacing a word with its definition—the example being given:

All [mortal, ¬feathers, bipedal] are mortal.

Socrates is a [mortal, ¬feathers, bipedal].

Therefore, Socrates is mortal.

Why, then, would you even want to have a word for “human”? Why not just say “Socrates is a mortal featherless biped”?

Because it’s helpful to have shorter words for things that you encounter often. If your code for describing single properties is already efficient, then there will not be an advantage to having a special word for a conjunction—like “human” for “mortal featherless biped”—unless things that are mortal and featherless and bipedal, are found more often than the marginal probabilities would lead you to expect.

In efficient codes, word length corresponds to probability—so the code for Z₁Y₂ will be just as long as the code for Z₁ plus the code for Y₂, unless P(Z₁Y₂) > P(Z₁)P(Y₂), in which case the code for the word can be shorter than the codes for its parts.

And this in turn corresponds exactly to the case where we can infer some of the properties of the thing from seeing its other properties. It must be more likely than the default that featherless bipedal things will also be mortal.

Of course the word “human” really describes many, many more properties— when you see a human-shaped entity that talks and wears clothes, you can infer whole hosts of biochemical and anatomical and cognitive facts about it. To replace the word “human” with a description of everything we know about humans would require us to spend an inordinate amount of time talking. But this is true only because a featherless talking biped is far more likely than default to be poisonable by hemlock, or have broad nails, or be overconfident.

Having a word for a thing, rather than just listing its properties, is a more compact code precisely in those cases where we can infer some of those properties from the other properties. (With the exception perhaps of very primitive words, like “red,” that we would use to send an entirely uncompressed description of our sensory experiences. But by the time you encounter a bug, or even a rock, you’re dealing with nonsimple property collections, far above the primitive level.)

So having a word “wiggin” for green-eyed black-haired people is more useful than just saying “green-eyed black-haired person” precisely when:

1. Green-eyed people are more likely than average to be black-haired (and vice versa), meaning that we can probabilistically infer green eyes from black hair or vice versa; or
2. Wiggins share other properties that can be inferred at greater-than-default probability. In this case we have to separately observe the green eyes and black hair; but then, after observing both these properties independently, we can probabilistically infer other properties (like a taste for ketchup).

One may even consider the act of defining a word as a promise to this effect. Telling someone, “I define the word ‘wiggin’ to mean a person with green eyes and black hair,” by Gricean implication, asserts that the word “wiggin” will somehow help you make inferences / shorten your messages.

If green-eyes and black hair have no greater than default probability to be found together, nor does any other property occur at greater than default probability along with them, then the word “wiggin” is a lie: The word claims that certain people are worth distinguishing as a group, but they’re not.

In this case the word “wiggin” does not help describe reality more compactly—it is not defined by someone sending the shortest message—it has no role in the simplest explanation. Equivalently, the word “wiggin” will be of no help to you in doing any Bayesian inference. Even if you do not call the word a lie, it is surely an error.

And the way to carve reality at its joints is to draw your boundaries around concentrations of unusually high probability density in Thingspace.