The odd thing about Shannon information is a coin flip can have just as much information as text, yet the former would intuitively seem devoid of information and the latter information rich. It is unclear what exactly is the relationship between Shannon information and what we intuitively consider information. Any ideas?
Shannon information deliberately only concerns syntactic information (information content). Other, more recent work, focuses on semantic information (information with meaning for a receiver).
> Shannon information theory provides various measures of so-called "syntactic information", which reflect the amount of statistical correlation between systems. In contrast, the concept of "semantic information" refers to those correlations which carry significance or "meaning" for a given system. Semantic information plays an important role in many fields, including biology, cognitive science, and philosophy, and there has been a long-standing interest in formulating a broadly applicable and formal theory of semantic information. In this paper we introduce such a theory. We define semantic information as the syntactic information that a physical system has about its environment which is causally necessary for the system to maintain its own existence. "Causal necessity" is defined in terms of counter-factual interventions which scramble correlations between the system and its environment, while "maintaining existence" is defined in terms of the system's ability to keep itself in a low entropy state.
Roughly speaking: The amount of computation or energy needed to perfectly reproduce a random source, such as a coin flip, is high, while the significance or meaning, for the average receiver, is low. Natural language text requires less computation to reproduce [1], but, for the average receiver, the significance is higher.
Hmm, but you could compress text to an equally random sequence. I.e. all minimal programs are by definition Kolmogorov random.
Also, what about crystalline forms, which are very orderly and require minimal computation to reproduce, but are equally insignificant for the average receiver?
> you could compress text to an equally random sequence
More or less correct. The key difference is that you could not compress a random coin flip sequence (and that a compressed text is meaningless until decompressed to original).
> all minimal programs are by definition Kolmogorov random
Compression provides an upper bound to K. Kolmogorov Randomness itself is not computable. AKA: You can't ever know if you have a minimal program.
The best approach that I've seen is a combination of Shannon information and Kolmogorov complexity. If an object has high Shannon information, then it is not crystalline. If it also has low Kolmogorov complexity then it is not random. This seems to characterize the sweet spot where meaningful information occurs. Kolmogorov called this quantity "randomness deficiency".
Rather than see the coin flip as a useless toss, one might imagine it as a binary decision. This or that?
With this in mind, you can see text as the output of an algorithm (brain) taking many such decisions. The information entropy contained in this text reveals the complexity (amount of distinct binary decisions) which needed to take place in the machine (brain) in order for the text to occur.
