Help support MIT OpenCourseWare by shopping at Amazon.com! MIT OpenCourseWare offers direct links to Amazon.com to purchase the books cited in this course. Click on the book titles and purchase the book from Amazon.com, and MIT OpenCourseWare will receive up to 10% of all purchases you make. Your support will enable MIT to continue offering open access to MIT courses.

Schedule

• Lecture 7
• Recitation 8

Lecture Handouts

• Lecture Notes
  • Chapter 8: Inference (PDF)
  • Chapter 9: Principle of Maximum Entropy: Simple Form (PDF)
• Jaynes, E. T. "Information Theory and Statistical Mechanics." Physical Review 106, no. 4 (May 15, 1957): 620-630. (PDF - 2.2 MB)

Assignment

• Problem Set 7 (PDF)
• Problem Set 7 Solutions (PDF)

Resources

Technical

The person most responsible for use of maximum entropy principles in various fields of science is Edwin T. Jaynes (July 5, 1922 - April 30, 1998). 

• Jaynes, E. T. "Information Theory and Statistical Mechanics." Physical Review 106, no. 4 (May 15, 1957): 620-630. (PDF - 2.2 MB)
  This is the seminal paper which really started the modern use of the Principle of Maximum Entropy in physics
• ———. "Information Theory and Statistical Mechanics. II." Physical Review 108, no. 2 (October 15, 1957): 171-190. (PDF - 3.9 MB)
  Continuation of the previous reference.

Historical

• Jaynes Photograph
• Jaynes Biography
• Jaynes Bibliography, list of Unpublished Works, and Book Reviews

Jaynes knew, of course, about Thomas Bayes and when on sabbatical in England sought out and photographed the Bayes tombstone.

Books

• The philosophy of assuming maximum uncertainty is discussed in Chapter 3 of Tribus, M. Thermostatics and Thermodynamics. Princeton, NJ: D. Van Nostrand Co, Inc., 1961. ASIN: B000ARSH5S.
• Another good explanation, in terms of estimating probabilities of an unfair die is in E. T. Jaynes, "Information Theory and Statistical Mechanics," pp. 181-218 in Statistical Physics. Brandeis Summer Institute 1962, W. A. Benjamin, Inc., New York, NY; 1963. (PDF)
• Personal history by Jaynes, Edwin T. "Where Do We Stand on Maximum Entropy?" In The Maximum Entropy Formalism. Edited by Raphael D. Levine and Myron Tribus. Cambridge, MA: MIT Press, 1978, pp. 15-118. ISBN: 0262120801. (PDF)