Essay About Central Paradigm Of Classic Information Theory And Thermodynamic Entropy

Information Theory
Essay Preview: Information Theory
Report this essay
1. Introduction
Information theory is the mathematical theory of data communication and
storage generally considered to have been founded in 1948 by Claude E.
Shannon. The central paradigm of classic information theory is the
engineering problem of the transmission of information over a noisy channel.
The main result of this theory is Shannons noisy-channel coding theorem,
which states that reliable communication is possible over unreliable channels. It
is possible to surround a noisy channel with appropriate encoding and
decoding systems, such that messages can be communicated at any rate less
than (but arbitrarily close to) the channel capacity with an arbitrarily small
probability of error.
Information theory in the 1950s was sometimes classified as a branch of the
then voguish field called “cybernetics”, which included many aspects of
potential machine representation of the world; it is a broad and deep
mathematical theory, with equally broad and deep applications, chief among
them coding theory.
Coding theory is concerned with finding explicit methods, called codes, of
increasing the efficiency and fidelity of data communication over a noisy
channel up near the limit that Shannon proved is all but possible. These codes
can be roughly subdivided into data compression and error-correction codes.
It took many years to find the good codes whose existence Shannon proved.
A third class of codes are cryptographic ciphers; concepts from coding theory
and information theory are much used in cryptography and cryptanalysis; see
the article on deciban for an interesting historical application.
Information theory is also used in intelligence, gambling, statistics, and even
music composition.
2. Redundancy
Redundancy in information theory is the number of bits used to transmit a
message minus the number of bits of actual information in the message. Data
compression is a way to eliminate such redundancy, while checksums are a
way of adding redundancy.
3. Entropy
Entropy is a concept in thermodynamics (see thermodynamic entropy),
statistical mechanics and information theory. The concepts of information and
entropy have deep links with one another, although it took many years for the
development of the theories of statistical mechanics and information theory to
make this apparent. This article is about information entropy, the
information-theoretic formulation of entropy.
The basic concept of entropy in information theory has to do with how much
randomness there is in a signal or random event. An alternative way to look
at this is to talk about how much information is carried by the signal.
As an example consider some English text, encoded as a string of letters,
spaces and punctuation (so our signal is a string of characters). Since some
characters are not very likely (e.g. z) while others are very common (e.g. e)
the string of characters is not really as random as it might be. On the other
hand, since we cannot predict what the next character will be, it does have
some randomness. Entropy is a measure of this randomness, suggested by
Claude E. Shannon in his 1948 paper A Mathematical Theory of
Communication.
Shannon offers a definition of entropy which satisfies the assumptions that:
* The measure should be proportional (continuous) – i.e. changing the
value of one of the probabilities by a very small amount should only
change the entropy by a small amount.
* If all the outcomes (letters in the example above) are equally likely then
increasing the number of letters should always increase the entropy.
* We should be able to make the choice (in our example of a letter) in
two steps, in which case the entropy of the final result should be a
weighted sum of the entropies of the two steps.
4. Source coding (data compression)
In computer science, data compression or source coding is the process of
encoding information using fewer bits (or other information-bearing units)
than a more obvious representation would use, through use of specific
encoding schemes. For example, this article could be encoded with fewer bits
if we accept the convention that the word “compression” is encoded as
“comp”. One popular instance of compression that many computer users are
familiar with is the ZIP file format, which, as well as providing compression,
acts as an archiver, storing many files in a single output file.
As is the case with