Why a few words
can say so much.
There are more possible meanings than there are words. A language must let some distinctions go.
Information bottleneck asks: which distinctions are worth keeping?
Meet the toy task ↓Many meanings → a few words → reconstructed meanings
A hidden parcel.
Four lockers. Two words.
A parcel is hidden in one of four neighboring lockers. A noisy scanner gives you a reading from 1 to 4. Your partner cannot see the scanner. You can send one word: A or B.
Your partner’s task is to form a belief about which locker contains the parcel. A reading of 1 favors locker 1, but the scanner can be wrong. The bars below show exactly how uncertain you are.
One parcel is behind one door.
U = the occupied locker.
Both people know how the scanner works.
You see the scanner.
Reading 1 · m₁
Your belief about the parcel’s location
p(u | m): a distribution over lockers
W = the word
It is heard perfectly.
They hear the word.
A means you saw reading 1 or 2. Each is equally likely.
After hearing A
Their belief about the parcel’s location
p(u | w): a mixture of your possible beliefs
Compare readings 1 and 2. Your belief changes, but your partner hears A either way and reconstructs the same mixture.
Changing the selected reading shows a different possible observation. It does not move the parcel to that numbered locker. Both bar charts always show probabilities for lockers 1–4, with heights on the same 0–100% scale.
M indexes the four readings and their associated beliefs. A value of M selects an entire set of four bars. U indexes the lockers along the bottom of those bars. A reading and a locker can share a number while playing different roles.
The task connects three questions: how uncertain is the reading, how much does a word reveal about it, and how useful is that word for locating the parcel?
Initially the four readings occur equally often. A and B are arbitrary word labels; later, the bottleneck will choose both the grouping and how many words to use.
Uncertainty has a shape.
In the locker task, the scanner can display one of four readings. Before you look, how uncertain are you about which reading you will see? Equally likely readings are hard to anticipate. A reading that almost always appears is easy to anticipate.
Give each outcome a rectangle. Its width is its probability. Its height is its surprise. The total area is entropy.
Four equally likely possibilities have two bits of uncertainty.
Make one reading common. Its rectangle gets wider and shorter. Rare readings get taller, but their small widths limit their contribution.
A rare outcome is very surprising when it happens. Entropy asks how much surprise we encounter on average.
Here the outcomes are scanner readings M. The slider changes their frequencies to illustrate entropy. From the next section onward, we return to four equally likely readings.
Why the logarithm?
A probability of ½ corresponds to one binary halving; ¼ corresponds to two. Independent surprises add because probabilities multiply. This gives −log₂p. The entropy is also the asymptotic average length of an optimal binary code, not necessarily the length of any individual word.
Zero-probability outcomes contribute zero: limp→0 p log(1/p)=0. The other three readings divide the remaining probability equally.
A message removes possibilities.
Recall the hidden parcel and noisy scanner. Its four readings are equally likely. You say A for readings 1–2 and B for readings 3–4. Your partner hears only the word. After A, they know you saw either reading 1 or reading 2.
For this first information measure, ask: what does the word reveal about the scanner reading? The four circles on the right are possible readings. We will use the locker probabilities in the next picture.
The symbol identifies a pair. One bit of uncertainty remains.
One bit survives the bottleneck.
The reading started with two bits of uncertainty. Hearing the word leaves one. The difference is mutual information: one bit.
H(M|W) averages the remaining uncertainty over messages. Here A and B give equally uncertain posteriors.
Now make word choice unreliable.
With probability η, you choose the other word: for example, B after reading 1. Your partner knows this rule. A no longer rules out readings 3 and 4, so the ribbons cross over and the belief spreads out.
This slider changes word choice. The scanner’s uncertainty about the parcel stays fixed.
Two words. Zero information.
At η=½, every reading produces either word equally often. Hearing A changes nothing.
The words still have one bit of entropy, but all of it is randomness in naming. Word variety and transmitted information are different quantities.
The same quantity, viewed from the words
Here H(W)=1 throughout, H(W|M)=h₂(η), and I(M;W)=1−h₂(η). Randomization belongs to the encoder q(w|m); the transmitted symbol itself is noiseless. R=I(M;W) measures the complexity of a naming system: the distinctions it retains about meanings.
Not every bit is equally useful.
Now return to your partner’s actual task: locate the parcel. Reading 1 favors locker 1; reading 2 favors locker 2. Both put most probability in the left two lockers. Readings 3 and 4 mostly favor the right two.
Compare two ways to pair the readings: 1 + 2 versus 3 + 4, or 1 + 3 versus 2 + 4. Either word identifies a pair and retains one bit about the reading. But the reconstructed locker beliefs are very different.
The listener mixes the beliefs of every reading that could have produced A. The reconstructed distribution shows how far that mixture moves from the prior.
M selects a row of four bars: a scanner reading and its belief. U selects a bar within that row: a possible locker. The naming rule groups rows into words W. Grouping overlapping rows preserves more information about the parcel’s location.
This is why information bottleneck needs two mutual informations: how much is retained, and how useful it is.
The scanner model, exactly specified
Both the locker U and reading M have uniform marginals. Each row below is the location belief after one scanner reading: p(u|m). Thus reading 1 means a 70% chance of locker 1, a 25% chance of locker 2, and small chances of lockers 3 and 4.
| u₁ | u₂ | u₃ | u₄ | |
|---|---|---|---|---|
| m₁ | .70 | .25 | .04 | .01 |
| m₂ | .25 | .50 | .21 | .04 |
| m₃ | .04 | .21 | .50 | .25 |
| m₄ | .01 | .04 | .25 | .70 |
Equivalently, generate an occupied locker uniformly, then generate a noisy reading. The likelihood p(m|u) is the same symmetric matrix here, and Bayes’ rule gives the displayed beliefs. These probabilities define a simplified sensor model.
The joint law is p(m,u,w)=p(m)p(u|m)q(w|m), so U ⟂ W | M. Before any compression, the readings contain I(M;U)=0.6145 bits about location. Adjacent pairs retain A=0.4512 bits; alternating pairs retain A=0.0889 bits.
Keep the distinctions that earn their cost.
One word is cheap, but tells the listener nothing. A separate word for every reading keeps all the available information, but preserves more distinctions than the task may need.
Keep the scanner model and reading frequencies fixed. Now let the naming rule use anywhere from one to four words, with soft assignments allowed. Move β to change the value of information about the locker. The word system and its position in the information plane change together.
At low β, extra distinctions are not worth paying for. At higher β, preserving information matters more. New categories can appear, and their boundaries can become sharper.
The widths of the ribbons are naming probabilities. A soft boundary is a real part of the solution, not an unfinished hard partition.
Reading the curve and the dashed line
Let R=I(M;W), A=I(W;U). A line of constant Fβ=R−βA has slope 1/β. Moving it upward lowers Fβ. At the optimum it supports the feasible set. Gray points mark the one-word, adjacent-pair, alternating-pair, and four-word hard encoders.
Data processing gives A≤min{R,I(M;U)}. For 0≤β≤1, Fβ≥(1−β)R≥0, and a constant word attains zero. A nontrivial optimum may begin at a larger β.
The frontier is estimated on a grid of β values using annealing and multiple initializations, with up to four symbols. The line connects numerical solutions; convergence does not guarantee a global optimum.
A word reconstructs a blur.
When A combines readings 1 and 2, the listener receives neither original belief. It receives their mixture. Select either reading below: the speaker’s belief changes, but the listener’s reconstruction stays the same.
KL divergence measures the extra log loss from using the listener’s belief instead of the speaker’s. Averaging over meanings and words gives a particularly useful identity:
The fixed scanner model supplies I(M;U)=0.615 bits about the locker before any word is spoken. The naming rule determines how much survives. The adjacent-pair code preserves 0.451 bits and loses about 0.163 bits. Maximizing informativeness therefore minimizes distortion.
The decoder, the identity, and the optimization update
The joint law gives U ⟂ W | M. For an arbitrary decoder r(u|w), the expected loss adds the nonnegative term Ew KL[p(u|w)∥r(u|w)]. The Bayesian mixture is therefore optimal for a fixed encoder.
A word is favored when it is common and its listener interpretation fits the meaning. Updating frequencies, listener mixtures, and assignments gives a self-consistent IB algorithm. The exponent uses base 2 because the divergences are measured in bits.
Change the meaning space.
Keep the machinery.
In the locker task, a scanner reading evokes a belief about where the parcel is. In the color paper, a presented chip c evokes a modeled belief mc about color. A color word groups these beliefs, and the listener reconstructs their mixture.
The physical chip c is a point in the color set 𝒰; the mental meaning mc is a distribution over that same set. In the graphs, M selects a belief curve; U runs along its horizontal axis.
Here is the same meaning → word → reconstructed meaning chain with sixteen color meanings. Click a chip, then watch where its probability goes.
This simplified circular model uses the same information measures as the color study. The listener curve shows the most likely word for the selected chip; overall scores average over all words.
Changing β moves along a tradeoff for a fixed task. Changing perceptual uncertainty or communicative need changes the task itself—and with it the frontier.
The study uses the World Color Survey’s 330 chips in CIELAB space. The observed data are the presented chips and people’s naming responses, which estimate q(w|m). The Gaussian beliefs p(u|m) are modeling assumptions. A shared need distribution specifies p(m). Together these fix I(M;U) across the languages in the main comparison.
The correspondence and the model assumptions
The toy probability table becomes Gaussian meanings mc(u) ∝ exp[−‖u−c‖²/(2σ²)], with σ²=64 in the paper. The uniform scanner-reading prior becomes the shared least-informative (LI) source, with a uniform baseline. Meanings and world states are both discrete in these implementations; soft beliefs and naming probabilities can still vary continuously.
The paper displays 110 WCS languages plus English. Fifteen WCS languages are excluded from LI estimation and quantitative evaluation for response-count requirements. LI is estimated from naming data, so the quantitative table uses fivefold cross-validation over languages. The figures use the source estimated from all folds.
The circular model uses 16 uneven coordinates xᵢ=i+0.8 sin(6πi/16), circumference 16, wrapped-distance Gaussian beliefs, and up to 16 words. Numerical solutions use annealing and multiple initializations. The published WCS results use the full CIELAB geometry.
Where do real languages land?
Each language is now an encoder, so it becomes a point in exactly the information plane we used for the toy task. The striking result is where those points sit.
Original Fig. 3. Same R and A; real naming data. The figure includes 111 languages. Select each annotation to read the picture.
For each empirical encoder, the authors fit β by minimizing its objective gap to the corresponding IB optimum. The result supports efficient compression under this model of meaning and need.
What “near optimal” means here
ε is a vertical gap to a supporting line, not generally the gap to the curved frontier at the empirical R. Under the LI source, Table 1 reports mean ε=0.18±0.07 and fitted β=1.03±0.01. β is fitted to the empirical encoder; it is not an independently measured population preference.
A hard map can hide a category.
Replacing every chip’s naming distribution with its winning word throws away uncertainty. In Culina, an emerging green category can have positive probability without winning any chip.
Switch between the two views below. The meanings have not moved; what changes is how much of the naming distribution we can see.
The top panel keeps only the most frequent name. A low-probability category can disappear from this view.
Original Fig. 4. Each pair of rows compares language data with the fitted IB encoder. The lower probability curves are slices through the color palette.
Efficiency is one claim; getting category structure right is another. The model’s advantage is strongest when the full naming distribution is evaluated: mean gNID is 0.18 for IB versus 0.47 for RKK+ under the LI source, with lower being better. RKK+ uses an earlier efficiency objective that predicts hard categories.
Category emergence and the limits of the result
As β changes, categories can emerge continuously before they become modal. Agarabi and Dyimini both have six major categories, but different naming probabilities and complexities. The optimization path is not an observed history of language change.
The model predicts yellow too early and English pink too late. Pooled speakers can create soft boundaries, so this population-level fit does not by itself establish an individual cognitive mechanism. Mode-map NID is similar for IB and RKK+ (0.31 versus 0.32); the soft-structure difference is larger.
Some distinctions make a pattern.
“Here,” “to here,” and “from here” organize space by distance and orientation. The same framework can score how a language groups these nine meanings.
But a good information tradeoff need not produce a regular paradigm. In the picture below, compare the pattern down the last column. Increasing the consistency preference removes an exception.
Each letter is a form. The pattern glyphs ignore letter names and preserve only which cells are equal. The search compares all 21,147 deterministic partitions, including the one-word baseline.
Here the typological data record categorical groupings rather than probabilistic naming trials. The analysis therefore uses deterministic encoders, making I(M;W)=H(W). The extra consistency term captures something that mutual information alone does not measure.
Across 220 languages with 34 paradigms, the study finds good information tradeoffs under suitable communicative costs. Adding consistency helps account for frequent systems, although the reported optimal set still misses the most frequent paradigm, shared by 71 languages.
The spatial result, assumptions, and comparison class
A meaning is m=(d,n), a world state u=(d′,n′), and p(u|m) ∝ μ^(|d−d′|+C(n,n′)), with μ=0.2. The default costs are C(place,source)=1.32, C(place,goal)=0.8, and C(goal,source)=2.12. The source uses the rounded Finnish frequencies in Table 2, normalized.
The paper fits orientation costs; better fits favor place–source being more costly than place–goal. The interactive calculation compares all B₉=21,147 partitions, including the one-word baseline; the paper reports 21,146. Information is measured in bits and γ in bits per pattern. These calculations use the published cost parameters without refitting the 220-language data. Global optimality is within the deterministic candidate class.
Compression chooses what to forget.
A language does not need to preserve every difference. It needs distinctions that let a listener recover what matters. Information bottleneck turns that intuition into a measurable tradeoff.
The claim is always relative to a model of meanings, communicative need, and the pressures included in the objective.