Unit 6: Processes, Lecture 1

Flash and JavaScript are required for this feature.

Download the video from iTunes U or the Internet Archive.

This video is the only lecture in this unit.

Instructors: Prof. Paul Penfield, Prof. Seth Lloyd

Readings

Notes, Chapter 7: Processes (PDF - 1.5 MB)

A summary in concise mathematics (mostly algebra) of many topics in information theory, coding, communications. There is a section on discrete memoryless channels.

Gallager, Robert G. "Claude E. Shannon: A Retrospective on His Life, Work, and Impact." IEEE Transactions on Information Theory 47 (November 2001): 2681-2695.

Homework

Problem Set 6 (PDF)

Problem Set 6 Solutions (PDF)

Resources

Technical

Shannon, Claude E. "A Mathematical Theory of Communication." Bell System Technical Journal 27 (July and October 1948): 379-423 (Part I), 623-656 (Part II).

These seminal papers are available in several forms (see This resource may not render correctly in a screen reader.bibliographic notes (PDF)):

  • PDF version of original papers (This resource may not render correctly in a screen reader.PDF)# (With corrections but without Shannon's 1949 modifications.)

  • Amazon logo Shannon, Claude E., and Warren Weaver. The Mathematical Theory of Communication. Urbana, IL: University of Illinois Press, 1998. ISBN: 9780252725463. (First edition published in 1949, incorporating a number of modifications and corrections by Shannon.)

  • Shannon, Claude E. "A Mathematical Theory of Communication." 50th Anniversary Edition, printed for the 1998 IEEE International Symposium on Information Theory, MIT. Cambridge, UK: Cambridge University Press, August 16-21, 1998. (Based on 1949 book, with corrections.)

  • Amazon logo Reprinted in Key Papers in the Development of Information Theory. Edited by D. Slepian. New York, NY: IEEE Press, 1974. ISBN: 9780879420284.

  • Amazon logo Reprinted in Claude Elwood Shannon: Collected Papers. Edited by N. J. A. Sloane, and A. D. Wyner. New York, NY: IEEE Press, 1993. ISBN: 9780780304345.

Because he treated a general case, Shannon was able to distinguish loss and noise, in the way done in 6.050J/2.110J. Many others treat only channels with similar inputs and outputs in which case often L = N so it may not be obvious why they are different concepts.

Historical

Claude E. Shannon (1916-2001); obituary, Tech Talk, February 28, 2001.

Gallager, Robert G. "Claude E. Shannon: A Retrospective on His Life, Work, and Impact." IEEE Transactions on Information Theory 47 (November 2001): 2681-2695.

The process model is also known as Markov Processes or Markov Chains after Andrei A. Markov (1856-1922).

Books

There are many excellent texts on communications, most of which assume a familiarity with mathematics beyond introductory calculus. Some have treatments of the discrete memoryless channel, but not with the same emphasis given here. Here are a few.

Amazon logo Gallager, Robert G. Information Theory and Reliable Communications. New York, NY: John Wiley & Sons, Inc., 1968. ISBN: 9780471290483.
One of the early textbooks, designed for first-year graduate students, by one of the pioneers in communications, an MIT faculty member. He was later awarded the IEEE Medal of Honor, its most prestigious award.

Amazon logo Hambley, Allan R. An Introduction to Communication Systems. New York, NY: W. H. Freeman, 1989. ISBN: 9780716781844.

Amazon logo Cover, Thomas M., and Joy A. Thomas. Elements of Information Theory. New York, NY: John Wiley & Sons, Inc., 2006. ISBN: 9780471241959.
Aimed at university seniors and first-year graduate students. One of several excellent books of that era. Professor Cover, at Stanford University, is one of the leaders in Information Theory.

Amazon logo Haykin, Simon. Communication Systems. 4th ed. New York, NY: John Wiley & Sons, Inc., 2000. ISBN: 9780471178699.