Back to Feed
Measuring the entropy of English
The Signal
Claude Shannon, a mathematician and cryptographer who pioneered information theory, established that printed English is theoretically compressible to roughly one bit per character given sufficient context. He argued that predicting the next character requires contextual inference that exceeds simplistic statistical frequency tables, leading to a broader, contested thesis: that compression effectively serves as a proxy for intelligence.
The Case
- Shannon began by attempting to predict subsequent characters using n-gram statistics, such as tracking which letters followed the sequence “TH,” but discovered this approach failed for longer strings because many sequences never recurred in finite samples.
- To overcome this data sparsity, Shannon utilized human prediction tasks, including an informal experiment where his wife, Betty, guessed next-letters from a text to measure how much information was actually required to reconstruct a passage.
- In his 1950 paper, "Prediction and Entropy of Printed English," Shannon refined this methodology by measuring the number of guesses participants required to correctly identify the next character, a proxy for measuring the implicit probability of the text.
- The narrator claims that because near-limit compression now relies on engineering sophisticated predictive models rather than simple tabulation, the process essentially "probes at something intelligent," though this intelligence-framing remains an interpretive, authorial project rather than a formal, proven technical equivalence.
The 1 Minute Signal Take
The video offers a coherent technical history of Shannon’s entropy experiments, though it occasionally drifts into thematic speculation regarding the nature of intelligence. Watch it for the clear logical pivot from raw frequency analysis to human-predictive measures, but view the series’ "compression-is-intelligence" narrative as a subjective frame rather than established academic consensus.
Time saved:
Tags
Back to Feed