Reading Assignments
Reading assignments were taken from the course textbook:
Cover, Thomas, and Joy Thomas. Elements of Information Theory. 2nd ed. New York, NY: Wiley-Interscience, 2006. ISBN: 9780471241959.
| LEC # | TOPICS | READINGS |
|---|---|---|
| 1 | Introduction, entropy | Chapter 1, sections 2.1 - 2.5 |
| 2 | Jensen's inequality, data processing theorem, Fanos's inequality | Sections 2.6 - 2.8, 2.11 |
| 3 | Different types of convergence, asymptotic equipartition property (AEP), typical set, joint typicality | Sections 3.1 - 3.3, 8.6 |
| 4 | Entropies of stochastic processes | Chapter 4 |
| 5 | Data compression, Kraft inequality, optimal codes | Sections 5.1 - 5.4 |
| 6 | Huffman codes | Sections 5.5 - 5.7 |
| 7 | Shannon-Fano-Elias codes, Slepian-Wolf | Sections 5.8 - 5.9, section 14 through the end of 14.4.4.1 |
| 8 | Channel capacity, binary symmetric and erasure channels | Sections 8.1 - 8.3 |
| 9 | Maximizing capacity, Blahut-Arimoto | |
| 10 | The channel coding theorem | Sections 8.4, 8.7 |
| 11 | Strong coding theorem, types of errors | |
| 12 | Strong coding theorem, error exponents | |
| In-class midterm | ||
| 13 | Fano's inequality and the converse to the coding theorem | Section 8.9 |
| 14 | Feedback capacity | Section 8.12 |
| 15 | Joint source channel coding | Section 8.13 |
| 16 | Differential entropy, maximizing entropy | Chapter 9, sections 11.1 - 11.6 |
| 17 | Additive Gaussian noise channel | Sections 10.1 - 10.3 |
| 18 | Gaussian channels: parallel, colored noise, inter-symbol interference | Sections 10.4 - 10.5 |
| 19 | Gaussian channels with feedback | Section 10.6 |
| 20 | Multiple access channels | Sections 14.1 - 14.3 |
| 21 | Broadcast channels | Section 14.6 |
| In-class presentations (2 sessions) | ||
| 22 | Finite state Markov channels | |
| 23 | Channel side information, wide-band channels |
Supplementary Readings
The supplementary readings are optional, except for the paper that you select as part of your project. The supplementary readings are coded for difficulty. One star: Accessible. Two stars: Requires significant mathematical maturity. Three stars: Expert level.
[* Lectures 4-5] Elias, Peter. "Predictive Coding: Part I." IRE Transactions: Information Theory 1, no. 16 (1955): 16-24.
[* Lecture 6] Huffman, David. "A Method for the Construction of Minimum-Redundancy Codes." Proceedings of the IRE 40, no. 9 (1952): 1098-1101.
[** Lecture 7] Slepian, David, and Jack Wolf. "Noiseless Coding of Correlated Information Sources." IEEE Transactions on Information Theory 19, no. 4 (1973): 471-480.
[** Lecture 7] Elias, Peter. "Predictive Coding: Part II." IRE Transactions: Information Theory 1, no. 16 (1955): 24-33.
[** Lecture 7] Gallagher, Robert. "Variations on a Theme by Huffman." IEEE Transactions on Information Theory 24, no. 6 (1978): 668-674.
[** Lectures 10-12] Gallagher, Robert. "A Simple Derivation of the Coding Theorem and Some Applications." IEEE Transactions on Information Theory 11, no. 1 (1965): 3-18.
[* Lectures 14, 16-20] Cover, Thomas. "The Role of Feedback in Communication." In Performance Limits in Communication Theory and Practice. Edited by J. K. Skwirzynski. New York, NY: Springer, 1988, pp. 225-235. ISBN: 9789024736959.
[** Lecture 20] El Gamal, Abbas, and Thomas Cover. "Multiple User Information Theory." Proceedings of the IEEE 68, no. 12 (1980): 1466-1483.
[** Lecture 20] Cover, Thomas, Abbas El Gamal, and Masoud Salehi. "Multiple Access Channels with Arbitrarily Correlated Sources." IEEE Transactions on Information Theory 26, no. 6 (1980): 648-657.
[** Lecture 20] Gallagher, Robert. "A Perspective on Multiaccess Channels." IEEE Transactions on Information Theory 31, no. 2 (1985): 124-142.
[** Lecture 21] Cover, Thomas. "Comments on Broadcast Channels." IEEE Transactions on Information Theory 44, no. 6 (1998): 2524-2530.
[** Lecture 21] Bergmans, Patrick, and Thomas Cover. "Cooperative Broadcasting." IEEE Transactions on Information Theory 20, no. 3 (1974): 317-324.
[*** Lecture 21] Weingarten, Hanan, Yosef Steinberg, and Shlomo Shamai (Shitz). "The Capacity Region of the Gaussian Multiple-Input Multiple-Output Broadcast Channel." IEEE Transactions on Information Theory 52, no. 9 (2006): 3936-3964.
[** Lectures 21-22] Jindal, Nihar, Sriram Vishwanath, and Andrea Goldsmith. "On the Duality of Gaussian Multiple-Access and Broadcast Channels." IEEE Transactions on Information Theory 50, no. 5 (2004): 768-783.
[*** Lectures 22-23] Goldsmith, Andrea, and Pravin Varaiya. "Capacity, Mutual Information, and Coding for Finite-State Markov Channels." IEEE Transactions on Information Theory 42, no. 3 (1996): 868-886.
[*** Lecture 23] Médard, Muriel, and Robert Gallagher. "Bandwidth Scaling for Fading Multipath Channels." IEEE Transactions on Information Theory 48, no. 4 (2002): 840-852.