This is an archived course. A more recent version may be available at ocw.mit.edu.

Syllabus

Course Meeting Times

Lectures: 1 session / week, 3 hours / session

Course Description

A computational camera attempts to digitally capture the essence of visual information by exploiting the synergistic combination of task-specific optics, illumination, sensors and processing. We will discuss and play with thermal cameras, multi-spectral cameras, high-speed, and 3D range-sensing cameras and camera arrays. We will learn about opportunities in scientific and medical imaging, mobile-phone based photography, camera for HCI and sensors mimicking animal eyes.

We will learn about the complete camera pipeline. In several hands-on projects we will build several physical imaging prototypes and understand how each stage of the imaging process can be manipulated.

We will learn about modern methods for capturing and sharing visual information. If novel cameras can be designed to sample light in radically new ways, then rich and useful forms of visual information may be recorded — beyond those present in traditional photographs. Furthermore, if computational process can be made aware of these novel imaging models, them the scene can be analyzed in higher dimensions and novel aesthetic renderings of the visual information can be synthesized.

In this course we will study this emerging multi-disciplinary field—one which is at the intersection of signal processing, applied optics, computer graphics and vision, electronics, art, and online sharing through social networks. We will examine whether such innovative camera-like sensors can overcome the tough problems in scene understanding and generate insightful awareness. In addition, we will develop new algorithms to exploit unusual optics, programmable wavelength control, and femto-second accurate photon counting to decompose the sensed values into perceptually critical elements.

Format

The course will consist of lectures, four project assignments, one mid-term exam and a final project. Additionally, each week a student will be assigned to take notes during the lecture, and distribute them afterward for class use. We will have a few guest talks by experts in the field.

The emphasis will be on hardware as well as software hands-on projects and we will progressively build the camera pipeline. Given the multi-disciplinary nature of the course, the class will be open and supportive of students with different backgrounds. Hence, we are going to try a two-track approach for homeworks: one software-intensive and the other with software-hardware (electronics/optics) emphasis.

We will use several camera elements such as optical elements (lenses, prisms, apertures, masks), light sources (programmable LEDs and projectors), sensors (high speed, thermal, multispectral, range-sensing) in our projects. However, this is not an optics class. The goal is to learn and build novel imaging devices.

Participants

The course is intended for students with interest in algorithmic and technical aspects of imaging and photography. Successful research in imaging requires a solid understanding in algorithms as well as technologies.

Prerequisites

Familiarity with imaging, camera techniques, applied optics, linear algebra and signal processing will be helpful but not necessary. We try to keep the mathematical prerequisites to a minimum, but we will introduce material from broad areas at a fast pace.

Related Courses

MAS.966 Camera Culture (Prof. Ramesh Raskar)

  • Graduate seminar with guest lecturers, discussion oriented

2.71/2.710 Optics (Prof. George Barbastathis)

  • Emphasis on Fourier optics and coherent imaging
  • [see Fall 2004 version in MIT OpenCourseWare]

6.815/6.865 Digital and Computational Photography (Prof. Fredo Durand)

  • Emphasis on software methods, graphics, and image processing

6.870 Computational Imaging (Prof. Berthold K. P. Horn) - last offered Spring 2006

  • Coding, nuclear and astronomical imaging, emphasis on theory

Readings

A list of suggested readings will be provided for each class. Suggested readings are also supplied for background on many of the project assignments.

Final Projects

The final project for the class should be novel and cool. Students will produce a conference quality paper describing the project, and we devote the last class period to presentations of the projects, followed by an award for the Best Project.

Grading

The credit weights are as follows:

ACTIVITIES PERCENTAGES
Project assignments 40%
Final project 30%
Mid-term exam 20%
Class participation 10%

 

To receive credit, you must attend regularly, complete the project assignment and develop a software or hardware prototype for final project.

Planned Schedule

Following is the planned schedule of topics per session. Some of the later classes may be subject to reordering or rescheduling.

LEC # TOPICS KEY DATES
1 Introduction and fast-forward preview of all topics Project assignment 1 out
2 Modern optics and lenses; ray-matrix operations  
3 Virtual optical bench, lightfield photography, fourier optics, wavefront coding

Project assignment 1 due

Project assignment 2 out

4 Digital illumination, Hadamard coded and multispectral illumination  
5

Emerging sensors: high speed imaging, 3D range sensors, femto-second concepts, front/back illumination, diffraction issues.

Project assignment 2 due

Project assignment 3 out

6

Beyond visible spectrum: multispectral imaging and thermal sensors, fluorescent imaging, 'audio camera'

 
7

Image reconstruction techniques, deconvolution, motion and defocus deblurring, tomography, heterodyned photography, compressive sensing

Project assignment 3 due

Project assignment 4 out

Final project start

8

Cameras for Human Computer Interaction (HCI): 0-D and 1-D sensors, spatio-temporal coding, frustrated TIR, camera-display fusion

Final project pre-proposals due
9

Short presentations on final project pre-proposals

Useful techniques in scientific and medical imaging: CT scans, strobing, endoscopes, astronomy and long range imaging

 
10 Mid-term exam  
11 Mobile photography, video blogging, life logs and online photo collections  
12

Optics and sensing in animal eyes: what can we learn from successful biological vision systems?

 
13 Future products and business models Final project proposals due
14 Final project presentations

Project assignment 4 due

Final project due

 

Acknowledgements

The course material has been prepared with slides, discussions and other contributions from many people. Only some of them are listed below and my apologies. Thank you all!

Jack Tumblin, Northwestern University, Shree Nayar, Columbia University, Amit Agrawal, MERL, Marc Levoy, Bennet Wilburn, Stanford U., Alyosha Efros, CMU, Steve Seitz, UW, Irfan Essa, Georgia Tech, Fredo Durand, MIT, Jingyi Yu, Delaware, Aseem Agrawala, U of Washington, Paul Debevec, USC, Todor Georgiev, Adobe, Hendrik Lensch, MPI.

I also want to thank many who have donated additional equipment, cameras, sensors, optics, software etc.

Fatih Porikli, Amit Agrawal, MERL, James Davis, UC of Santa Cruz, 3DV, Eddy Talvala and Andrew Adams, Stanford.