Readings
 When you click the Amazon logo to the left of any citation and purchase the book (or other media) from Amazon.com, MIT OpenCourseWare will receive up to 10% of this purchase and any other purchases you make during that visit. This will not increase the cost of your purchase. Links provided are to the US Amazon site, but you can also support OCW through Amazon sites in other regions. Learn more. |
The reading assignments are from the required course textbook, Cover, T. M and J. A. Thomas. Elements of Information Theory. New York, NY: John Wiley & Sons, 2001. ISBN: 0471062596.
Supplemental readings are also provided below the required readings.
| LEC # | TOPICS | READINGS |
|---|---|---|
| 1 | Introduction, Review of Random Variables, Entropy, Mutual Information, Chain Rules | Chapter 1, Sections 2.1 - 2.5 |
| 2 | Jensen's Inequality, Data Processing Theorem, Fanos's Inequality | Sections 2.6 - 2.8 and 2.11 |
| 3 | Markov Chain, Entropy Rate of Random Processes | Chapter 4 |
| 4 | Different Types of Convergence, Asymptotic Equipartition Property (AEP), Typical Set, Joint Typicality | Sections 3.1 - 3.3 and 8.6 |
| 5 | Data Compression, Kraft Inequality, Optimal Codes | Sections 5.1 - 5.5 |
| 6 | Huffman Codes, Sensitivity of Distribution, Elias Code | Sections 5.6 - 5.9 |
| 7 | Gambling | Chapter 6 |
| 8 | Channel Capacity, Symmetric and Erasure Channels | Sections 8.1 - 8.3 |
| 9 | Coding Theorem | Sections 8.4 - 8.9 |
| 10 | Strong Coding Theorem | |
| 11 | Strong Coding Theorem (cont.) | |
| 12 | Feedback Capacity | Section 8.12 |
| 13 | Joint Source Channel Coding | Section 8.13 |
| 14 | Differential Entropy | Chapter 9 |
| 15 | Gaussian Channel | Sections 10.1 - 10.3 |
| 16 | Gaussian Channels: Parallel, Colored Noise, Inter-symbol Interference | Sections 10.4 - 10.5 |
| 17 | Maximizing Entropy | Sections 11.1 - 11.6 |
| 18 | Gaussian Channels with Feedback | Section 10.6 |
| 19 | Fading Channels | |
| 20 | Types, Universal Source Coding, Sanov's Theorem | Sections 12.1 - 12.6 |
| 21 | Multiple Access Channels | Sections 14.1 - 14.3 |
| 22 | Slepian-Wolf Coding | Sections 14.4 - 14.5 |
| 23 | Broadcast Channels | Section 14.6 |
| 24 | Channel Side Information, Wide-band Channels |
Supplemental Readings
This section provides the optional supplementary readings for the course. Most of the readings are coded for difficulty:
One star (*): Accessible
Two stars (**): Requires significant mathematical maturity
Three stars(***): Expert level
Shannon, Claude. "A Mathematical Theory of Communication.'' Bell System Technical Journal 27 (July and October, 1948): 379-423 and 623-656.
***Vembu, Sridhar, Sergio Verdu, and Yossef Steinberg. "The Source-Channel Separation Theorem Revisited." IEEE Trans. on Info. Theory 41, no. 1 (January 1995).
***Biglieri, Ezio, John Proakis, and Shlomo Shamai. "Fading Channels: Information-Theoretic and Communications Aspects." IEEE Trans. on Info. Theory 44, no. 6 (October, 1998).
*Rimoldi, Bixio, and Rudiger Urbanke. "A Rate-Splitting Approach to the Gaussian Multiple-Access Channel." IEEE Trans. on Info. Theory 42, no. 2 (March, 1996).
*Bergmans, Patrick. "A Simple Converse for Broadcast Channels with Additive White Gaussian Noise." IEEE Trans. on Info. Theory 20, no. 2 (March, 1974).
*———. "Random Coding Theorem for Broadcast Channels with Degraded Components" IEEE Trans. on Info. Theory 19, no. 2 (March 1973).
**Cover, Thomas. "Comments on Broadcast Channels." IEEE Trans. on Info. Theory 44, no. 6 (October, 1998).
**Mushkin, Mordechai, and Israel Bar-David. "Capacity and Coding for the Gilbert-Elliott Channels." IEEE Trans. on Info. Theory 35, no. 6 (November, 1989).
***Feder, Meir, and Neri Merhav. "Hierarchical Universal Coding." IEEE Trans. on Info. Theory 42, no. 5 (September, 1986).
***Verdu, Sergio, and Te Sun Han. "A General Formula for Channel Capacity." IEEE Trans. on Info. Theory 40, no. 4 (July, 1994).
*Gallager, Robert. "Variations on a Theme by Huffman." IEEE Trans. on Info. Theory 24, no. 6 (November, 1978).
*Chomsky, Noam. "Three Models for the Description on Language." IEEE Trans. on Info. Theory 2, no. 3 (November, 1956).
**Thomas, Joy. "Feedback Can at Most Double Gaussian Multiple Access Channel Capacity." IEEE Trans. on Info. Theory 33, no. 5 (September, 1987).
**Cheng, Roger, and Sergio Verdu. "Gaussian Multiaccess Channel with ISI: Capacity Region and Multiuser Water-Filling." IEEE Trans. on Info. Theory 39, no. 3 (May, 1993).
***Caire, Giuseppe, and Shlomo Shamai. "On the Capacity of Some Channels with Channel State Information." IEEE Trans. on Info. Theory 45, no. 6 (September, 1999).
*Gray, Robert, Mari Dunham, and R. Gobbi. "Eogodicity of Markov Channels." IEEE Trans. on Info. Theory 33, no. 5 (September, 1987).
***Goldsmith, Andrea, and Pravin Varaiya. "Capacity, Mutual Information, and Coding for Finite-State Markov Channels." IEEE Trans. on Info. Theory 42, no. 3 (May, 1993).
*Gallager, Robert. "A Perspective on Multiaccess Channels." IEEE Trans. on Info. Theory 31, no. 2 (March, 1985).
***Hirt, Walter, and James Massey. "Capacity of the Discrete-Time Gaussian Channel with Inter-symbol Interference." IEEE Trans. on Info. Theory 34, no. 3 (May, 1988).
**Shamai, Shlomo, and Aaron Wyner. "Information-Theoretic Considerations for Symmetric Cellular, Multiple-Access Fading Channels, Part I and II." IEEE Trans. on Info. Theory 43, no. 6 (May, 1997).
***Ozarow, Lawrence. "Random Coding for Additive Gaussian Channels with Feedback." IEEE Trans. on Info. Theory 36, no. 1 (January, 1990).
***Hughes, Brian, and Tony Thomas. "On Error Exponents for Arbitrarily Varying Channels." IEEE Trans. on Info. Theory 42, no. 1 (May, 1996).
***Medard, Muriel, and Robert Gallager. "Bandwidth Scaling for Fading Multipath Channels." IEEE Trans. on Info. Theory 48, no. 4 (April, 2002).
***Gallager, Robert. "A Simple Derivation of the Coding Theorem and Some Applications." IEEE Trans. on Info. Theory 11, no. 1 (January, 1965).
***Telatar, Emre, and David Tse. "Capacity and Mutual Information of Wideband Multipath Fading Channels." IEEE Trans. on Info. Theory 46, no. 4 (July, 2000).
**Telatar, I., "Capacity of Multi-antenna Gaussian Channels." European Transactions on Telecommunications 10 (Nov/Dec 1999): 585-595.
Verdu, Sergio. "On Channel Capacity per Unit Cost." EEE Trans. on Info. Theory 36, no. 5 (September 1990).
———. "Spectral Efficiency in the Wideband Regime." EEE Trans. on Info. Theory 48, no. 6 (June 2002).
Forney, David. "On Exponential Error Bounds for Random Codes on the BSC."
Jindal, Nihar, Sriram Vishwanath, and Andrea Goldsmith. "On The Duality of Gaussian Multiple-Access and Broadcast Channels." Preprint 2002.
