On Thu, 10 Jul 1997 23:42:17 -0400, Brian D Harper wrote:
[...]
>PM>And the information theory entropy has no relationship to the
>entropy as defined by thermodynamics.
>SJ>Someone better tell the author of my daughter's university
>physics textbook! As he points out, there is a "relationship"
>between "entropy as defined by thermodynamics" and "information
>theory":
>PM>I disagree since there is not second law of information content.
>SJ>Your claim was that "information theory entropy has no
>relationship to the entropy as defined by thermodynamics". I
>responded with a quote from a "university physics textbook" that
>there is a "`relationship' between `entropy as defined by
>thermodynamics' and `information theory' ". You replied with an
>"unsupported assertion" that you "disagree since there is not second
>law of information content." Please supply quotes or references to
>support yoour psoition "that information theory entropy has no
>relationship to the entropy as defined by thermodynamics."
BH>First let me apologize for not having kept up with this thread.
>In particular, I did not see the quote from your daughters physics
>text nor is it particularly clear to me exactly why you want to
>establish a relation between thermodynamic and informational
>entropy. At first glance, such a stance seems odd to me. Suppose
>there were a second law for informational entropy. Such a law would
>guarantee that the informational entropy increases with time. But
>informational entropy is just another word for information content.
>So this law guarantees the increase in information content with
>time. Is this what you want?
I have no particular desire to "establish a relation between
thermodynamic and informational entropy. But Pim asserted that
"information theory entropy has no relationship to the entropy as
defined by thermodynamics". I quoted from my daughter's university
Physics text that there is a relationship between entropy as defined
by thermodynamics and information theory:
"In general, we associate disorder with randomness salt and pepper in
layers is more orderly than a random mixture; a neat stack of
numbered pages is more orderly than pages strewn randomly about on
the floor. We can also say that a more orderly arrangement is one
that requires more information to specify or classify it. When we
have one hot and one cold body, we have two classes of molecules and
two pieces of information; when the two bodies come to the same
temperature, there is only one class and one piece of information.
When salt and pepper are mixed there is only one (uniform) class;
when they are in layers, there are two classes. In this sense,
information is connected to order, or low entropy. This is the
foundation upon which the modern field of information theory is
built." (Giancoli D.C, "Physics", 1991, p403)
BH>Well, I'm afraid I agree with Pim on this, but I'll quickly add
>that this issue is very controversial with noted experts on both
>sides of the aisle.
OK. In view of our past differences on what turned out to be
mostly semantics, let us be clear from the outset that what you
"agree with Pim on" is that "information theory entropy has *no*
relationship to the entropy as defined by thermodynamics", because
that is the only disagreement I have with Pim in this thread. If you
admit *any* "relationship" between "information theory entropy" and
"entropy as defined by thermodynamics", then I will take it that you
"agree" with me and dis-"agree" with Pim on this specific point. OK?
BH>For example, Shannon and his cohort Weaver disagreed on this
>point. After bringing up Shannon I cannot resist repeating the
>humorous anecdote as to how Shannon arrived at the name "entropy"
>for his measure of information content. Apparently Shannon wanted
>to call it a measure of information content but hesitated for fear
>of confusion since "information" has so many different meanings. He
>then discussed his little "problem" with his friend, Von Neumann,
>who advised him to call it "entropy" for two reasons: "First,
>the expression is the same as the expression for entropy in
>thermodynamics and as such you should not use two different names
>for the same mathematical expression, and second, and more
>importantly, entropy, in spite of one hundred years of history, is
>not very well understood yet and so as such you will win every time
>you use entropy in an argument."
Indeed the "expression" for entropy in information theory "is the
same as the expression for entropy in thermodynamics":
"The formula for entropy in information theory is identical in form
to that representing entropy in statistical mechanics. There are a
number of other connections between statistical mechanics and
information theory, and some authors now develop thermodynamics and
statistical mechanics from the standpoint of information theory."
(Gallager R.G., "Information Theory", Encyclopaedia Britannica, 1984,
9:576)
Since "The formula for entropy in information theory is identical in
form to that representing entropy in statistical mechanics" and "some
authors now develop thermodynamics and statistical mechanics from the
standpoint of information theory" do you still "agree with Pim" that
"information theory entropy has no relationship to the entropy as
defined by thermodynamics"?
BH>If only Shannon had not listened to von Neumann ;-)
But he did, and used the same term "entropy" for "information theory
entropy" and "entropy as defined by thermodynamics".
There is nothing in the above unreferenced quote which indicates that
"the information theory entropy has no relationship to the entropy as
defined by thermodynamics."
BH>OK, I'll finally answer your request by giving a few quotes that
>outline the reasons for saying that there is no relation between the
>informational and thermodynamic entropies. Regardless of the fact
>that experts have taken the opposing view, the arguments presented
>in the following quotes seem to me to be pretty convincing:
>
>==========================================================
>As a result of its independent lines of development in thermodynamics
>and information theory, there are in science today two "entropies."
>This is one too many (see also Denbigh, 1982). It is not science's
>habit to affix the same name to different concepts, since common
>names suggest shared meanings.
The real question is are they "different concepts"? Maybe the
"entropy" in "thermodynamics" and the "entropy" in "information
theory" special expressions of the same general law of order to
disorder:
"...the entropy of a system can be considered a measure of the
disorder of the system. Then the second law of thermodynamics can be
stated simply as: Natural processes tend to move toward a state of
greater disorder." (Giancoli D.C, "Physics: Principles with
Applications", 1991, p402)
BH>Given the inevitable tendency for connotations to flow from the
>established to the new, the Shannon entropy began from the
>beginning to take on colorations of thermodynamic entropy. In his
>introductory chapter to Shannon's paper, Weaver revealingly quotes
>Eddington as follows: "The law that entropy always increases-the
>second law of thermodynamics- holds, I think, supreme position among
>the laws of Nature" (Shannon and Weaver, 1949). Shannon [*] goes on
>to rhapsodize that "thus when one meets the concept of entropy in
>communication theory, he has a right to be excited--a right to
>expect that one has hold of something that may turn out to be basic
>and important" (Shannon and Weaver, 1949). <<[*] a misprint,
>this quote is from Weaver not Shannon--BH>
This seems to indicate that "Shannon and Weaver" *intended* "Shannon
entropy ...to take on colorations of thermodynamic entropy".
BH>To appreciate the importance of restricting "entropy" to
>thermodynamic applications--or, at most, to other probabilistic
>applications where microstate-macrostate relationships provide a
>condition of irreversibility--one need only reflect on these
>remarks. While an indisputably important contribution to science,
>the Shannon formulation does not make contact with the second law.
>Shannon himself made no claims that it did.
One would have thought that his giving it the same name as the
"entropy" in "the second law", is itself a "claim"?
In any event, this seems to be just a semantic argument, wanting to
restrict "entropy" to a highly specialised meaning. This depends on
one's perspective and values of what's more important.
BH>But as long as the term "entropy" buttresses the Shannon formula,
>the second law remains a steady source of justification for ideas
>that must find their own grounds of support. If it were possible to
>treat "entropy" simply as an equation, with properties dependent on
>area of application, calling Shannon's function by that name would
>be relatively unproblematic. In point of fact, most who use the
>term "entropy" feel something of Weaver's conviction about
>contacting a universal principle that provides sweeping laws of
>directional change.
It seems that this supports my position that "Shannon's function" is
part of "a universal principle".
BH>[...] Brooks and Wiley (1985) correctly point out that the
>Boltzmann equation did not originate with Boltzmann or with
>thermodynamic relationships, but with the eighteenth-certury
>mathematician DeMoivre's analysis of games of chance. This is just
>more evidence that equations are not equivalent to concepts.
>Everything that bears the stamp H = - k SUM P_i log P_i does
>not have the property of increasing in time. Irreversibility must
>be independently demonstrated. If one has a nonequilibrium system
>governed by stochastic dynamics and negotiable energy barriers, then
>entropy will increase. --- Wicken, J.S. (1988). "Thermodynamics,
>Evolution, and Emergence: Ingredients for a New Synthesis," in
><Entropy, Information, and Evolution>, Editors B.H. Weber, D.J.
>Depew and J.D. Smith, MIT Press, Cambridge, MA, pp. 139-169.
>==================================================
The point is that Boltzmann showed that at its most basic level the
second law of thermodynamics is essentially statistical and that
thermodynamic equilibrium is the most probable state:
"In the 1870s Boltzmann published a series of papers in which he
showed that the second law of thermodynamics, which concerns energy
exchange, could be explained by applying the laws of mechanics and
the theory of probability to the motions of the atoms. In so doing,
he made clear that the second law is essentially statistical and that
a system approaches a state of thermodynamic equilibrium (equal
energy distribution throughout) because equilibrium is overwhelmingly
the most probable state in which matter occurs." ("Boltzmann, Ludwig
Eduard", Encyclopaedia Britannica, 1984, ii:134)
"From these examples, it is clear that probability is directly
related to disorder and hence to entropy. That is, the most probable
state is the one with greatest entropy, or greatest disorder and
randomness. Boltzmann showed that, consistent with Clausius's
definition (/\S= Q/T), the entropy of a system in a given (macro)
state can be written: S=2.3 k log W, where k is Boltzmann's constant
(k = 1.38 x 10^-23 J/K) and W is the number of microstates
corresponding to the given macrostate; that is, W is proportional to
the probability of occurrence of that state. In terms of
probability, the second law of thermodynamics-which tells us that
entropy increases in any process- reduces to the statement that those
processes occur which are most probable. The second law thus becomes
a trivial statement. However, there is an additional element now.
The second law in terms of probability does not forbid a decrease in
entropy. Rather, it says the probability is extremely low. It is
not impossible that salt and pepper should separate spontaneously
into layers, or that a broken tea cup should mend itself. It is even
possible that a lake should freeze over on a hot summer day (that is,
for heat to flow out of the cold lake into the warmer surroundings).
But the probability for such events occurring is extremely small."
Giancoli D.C, "Physics", 1991, p406)
BH>That H and S [**] are irreconcilably different should have been
>clear from Shannon's original identification of "the entropy of the
>set of probabilities." The thermodynamic entropy S is a state
>function property of a physical system. A crystal, an organism, a
>star, each has a value of S characteristic of it. H, on the other
>hand cannot refer to a physical system since it is identified with a
>mathematical construct. Pursuing thermodynamics a step further, the
>quantity TS is an energy-related, well-defined physical parameter of
>any physical system at uniform temperature. What in the world is
>the quantity TH? For that matter, what is T for a "set of
>probabilities" or a sequence of symbols? If H and S are related in
>any way that is more profound than sharing a mathematical formalism,
>it should be possible to identify a temperature-analog that can be
>meaningfully and usefully applied to sets and sequences.
I cannot see how this necessarily follows, if "temperature" is just a
special case of a more general statistical law.
BH>Wicken has drawn biologists' attention to the incommensurateness
>of physical and informational "entropies": "One should begin then
>by distinguishing carefully between two distinct kinds of
>entropy.... First is thermodynamic entropy itself, which involves
>statistical ensembles of micro-structures.... Second is the entropy
>of a sequence of elements, which can be defined as the minimum
>algorithm or information required for its unambiguous
>specification.... It must be appreciated that sequence entropies
>are not thermodynamic entropies" (Wicken, 1983 p. 439).
If "micro-structures" are the same as "microstates", then they are
fundamentally statistical, not just thermodynamic:
"The ideas of entropy and disorder are made clearer with the use of a
statistical or probabilistic analysis of the molecular state of a
system. This statistical approach, which was first applied toward
the end of the nineteenth century by Ludwig Boltzmann (1844-1906),
makes a clear distinction between the "macrostate" and the
"microstate" of a system. The microstate of a system would be
specified when the position and velocity of every particle (or
molecule) is given. The macrostate of a system is specified by
giving the macroscopic properties of the system-the temperature,
pressure, number of moles, and so on. In reality, we can know only
the macrostate of a system. There are generally far too many
molecules in a system to be able to know the velocity and position of
every one at a given moment. Nonetheless, it is important to
recognize that a great many different microstates can correspond to
the same macrostate. Let us take a simple example. Suppose you
repeatedly shake four coins in your hand and drop them on the table.
Specifying the number of heads and the number of tails that appear on
a given throw is the macrostate of this system. Specifying each coin
as being a head or a tail is the microstate of the system." Giancoli
D.C, "Physics, 1991, pp404-405)
BH>Wicken is correct, but he does not go far enough, in my view.
>Retaining the name "entropy" for the minimum algorithm of a sequence
>ensures that confusion will accompany attempts to interpret this
>quantity. Elsasser has expressed this problem very cogently. As he
>points out, "Gravitational and electrostatic equilibrium are
>characterized by the same (Laplace's) differential equation. Nobody
>would think that gravitational attraction and electrostatic
>attraction or repulsion are otherwise identical". (Elsasser, 1983,
>p. 107). The reason that nobody makes this mistake is that gravity
>and electrostatics have different names. Let us call the minimum
>information of a sequence by some other name than entropy. Perhaps
>'information potential' would do, for reasons expressed below. ---
>Olmsted, J. III (1988). "Observations on Evolution," in <Entropy,
>Information, and Evolution>, Editors B.H. Weber, D.J. Depew and
>J.D. Smith, MIT Press, Cambridge, MA, pp. 263-274. <<[**] H
>and S are the common symbols for the Shannon and thermodynamic
>entropies respectively-- BH>
>======================================================
I wonder if even this "Gravitational and electrostatic equilibrium"
which "are characterized by the same...differential equation" have an
underlying unity at a deeper level? But even if they don't it does
not necessarily mean that "information theory entropy has no
relationship to the entropy as defined by thermodynamics."
BH>There are a number of papers in the literature in which an
>attempt is made to establish a relation between Shannon entropy and
>Maxwell-Boltzmann-Gibbs entropy. Eigen (1971) follows Brillouin
>(1962) in confusing Maxwell-Boltzmann-Gibbs entropy with Shannon
>entropy. This point is discussed in section 12.1. There is such a
>relationship if and only if the probability spaces are isomorphic;
>in that case they have the same entropy (section 2.2.l). For
>example, the entropy of the probability space characteristic of the
>tossing of two dice is not equal to the entropy of the probability
>space of the tossing of a fair coin because these probability spaces
>are not isomorphic.
There seems to be confusion here that unless two things are
"identical", they cannot be related?
BH>Furthermore, there is a Shannon entropy for each alphabet
>depending on the arbitrary choice of the number of letters. There
>is a relationship between the Shannon entropies of different
>alphabets if and only if they are related by a code. It is clear
>from section 2.2.1 that Shannon entropy and Maxwell- Boltzmann-Gibbs
>entropy do not pertain to probability spaces that are isomorphic.
>In addition, these probability spaces are not related by a code.
>For that reason Maxwell-Boltzmann-Gibbs entropy is not applicable to
>the probability spaces used in communication theory or in the
>genetic information system of molecular biology. Furthermore, in
>thermodynamics and statistical mechanics there is an integral of the
>motion of the system, namely, the conservation of energy, which has
>no counterpart in information theory. For these reasons there is,
>therefore, no relation between Maxwell-Boltzmann-Gibbs entropy of
>statistical mechanics and the Shannon entropy of communication
>systems (Yockey, 1974, 1977c). -- Yockey, H. (1992). <Information
>Theory and Molecular Biology>, Cambridge University Press., p. 70.
>=================================================================
Yockey may have shown they are not identical, but it does not
therefore follow that "there is...no relation between...entropy of
statistical mechanics and the...entropy of communication systems"
Thanks for your quotes from your "experts". I have already quoted
some of mine. Here are some more:
"Information Theory, theory concerned with the mathematical laws
governing the transmission and processing of information. More
specifically, information theory deals with the measurement of
information, the representation of information (such as encoding),
and the capacity of communication systems to transmit and process
information...In most practical applications, one must choose among
messages that have different probabilities of being sent. The term
entropy has been borrowed from thermodynamics to denote the average
information content of these messages. Entropy can be understood
intuitively as the amount of "disorder" in a system. In information
theory the entropy of a message equals its average information
content. If, in a set of messages, the probabilities are equal, the
formula for the total entropy can be given as H = log2N, in which N
is the number of possible messages in the set." ("Information
Theory," Microsoft Encarta, Microsoft Corporation, 1993)
"The concept of entropy also plays an important part in the modern
discipline of information theory, in which it denotes the tendency of
communications to become confused by noise or static. The American
mathematician Claude E. Shannon first used the term for this purpose
in 1948. An example of this is the practice of photocopying
materials. As such materials are repeatedly copied and recopied,
their information is continually degraded until they become
unintelligible. Whispered rumors undergo a similar garbling, which
might be described as psychological entropy. Such degradation also
occurs in telecommunications and recorded music. To reduce this
entropy rise, the information may be digitally encoded as strings of
zeros and ones, which are recognizable even under high "noise"
levels, that is, in the presence of additional, unwanted signals."
(Settles G., "Entropy", Grolier Multimedia Encyclopedia, 1995)
In view of this "expert" testimony, the burden of proof is on you and
Pim to show that "the information theory entropy has NO RELATIONSHIP
to the entropy as defined by thermodynamics" (emphasis mine). The
"experts" you cite do not actually say this, and the "experts" I cite
say there is a relationship between the two entopies, at a deeper
level.
God bless.
Steve
-------------------------------------------------------------------
| Stephen E (Steve) Jones ,--_|\ sejones@ibm.net |
| 3 Hawker Avenue / Oz \ Steve.Jones@health.wa.gov.au |
| Warwick 6024 ->*_,--\_/ Phone +61 8 9448 7439 (These are |
| Perth, West Australia v my opinions, not my employer's) |
-------------------------------------------------------------------