Syllabus
Course Meeting Times
Lectures: 1 session / week, 3 hours / session
Course Description
A computational camera attempts to digitally capture the essence of visual information by exploiting the synergistic combination of task-specific optics, illumination, sensors and processing. We will discuss and play with thermal cameras, multi-spectral cameras, high-speed, and 3D range-sensing cameras and camera arrays. We will learn about opportunities in scientific and medical imaging, mobile-phone based photography, camera for HCI and sensors mimicking animal eyes.
We will learn about the complete camera pipeline. In several hands-on projects we will build several physical imaging prototypes and understand how each stage of the imaging process can be manipulated.
We will learn about modern methods for capturing and sharing visual information. If novel cameras can be designed to sample light in radically new ways, then rich and useful forms of visual information may be recorded — beyond those present in traditional photographs. Furthermore, if computational process can be made aware of these novel imaging models, them the scene can be analyzed in higher dimensions and novel aesthetic renderings of the visual information can be synthesized.
In this course we will study this emerging multi-disciplinary field—one which is at the intersection of signal processing, applied optics, computer graphics and vision, electronics, art, and online sharing through social networks. We will examine whether such innovative camera-like sensors can overcome the tough problems in scene understanding and generate insightful awareness. In addition, we will develop new algorithms to exploit unusual optics, programmable wavelength control, and femto-second accurate photon counting to decompose the sensed values into perceptually critical elements.
Format
The course will consist of lectures, four project assignments, one mid-term exam and a final project. Additionally, each week a student will be assigned to take notes during the lecture, and distribute them afterward for class use. We will have a few guest talks by experts in the field.
The emphasis will be on hardware as well as software hands-on projects and we will progressively build the camera pipeline. Given the multi-disciplinary nature of the course, the class will be open and supportive of students with different backgrounds. Hence, we are going to try a two-track approach for homeworks: one software-intensive and the other with software-hardware (electronics/optics) emphasis.
We will use several camera elements such as optical elements (lenses, prisms, apertures, masks), light sources (programmable LEDs and projectors), sensors (high speed, thermal, multispectral, range-sensing) in our projects. However, this is not an optics class. The goal is to learn and build novel imaging devices.
Participants
The course is intended for students with interest in algorithmic and technical aspects of imaging and photography. Successful research in imaging requires a solid understanding in algorithms as well as technologies.
Prerequisites
Familiarity with imaging, camera techniques, applied optics, linear algebra and signal processing will be helpful but not necessary. We try to keep the mathematical prerequisites to a minimum, but we will introduce material from broad areas at a fast pace.
Related Courses
MAS.966 Camera Culture (Prof. Ramesh Raskar)
- Graduate seminar with guest lecturers, discussion oriented
2.71/2.710 Optics (Prof. George Barbastathis)
- Emphasis on Fourier optics and coherent imaging
- [see Fall 2004 version in MIT OpenCourseWare]
6.815/6.865 Digital and Computational Photography (Prof. Fredo Durand)
- Emphasis on software methods, graphics, and image processing
6.870 Computational Imaging (Prof. Berthold K. P. Horn) - last offered Spring 2006
- Coding, nuclear and astronomical imaging, emphasis on theory
Readings
A list of suggested readings will be provided for each class. Suggested readings are also supplied for background on many of the project assignments.
Final Projects
The final project for the class should be novel and cool. Students will produce a conference quality paper describing the project, and we devote the last class period to presentations of the projects, followed by an award for the Best Project.
Grading
The credit weights are as follows:
ACTIVITIES | PERCENTAGES |
---|---|
Project assignments | 40% |
Final project | 30% |
Mid-term exam | 20% |
Class participation | 10% |
To receive credit, you must attend regularly, complete the project assignment and develop a software or hardware prototype for final project.
Planned Schedule
Following is the planned schedule of topics per session. Some of the later classes may be subject to reordering or rescheduling.
LEC # | TOPICS | KEY DATES |
---|---|---|
1 | Introduction and fast-forward preview of all topics | Project assignment 1 out |
2 | Modern optics and lenses; ray-matrix operations | |
3 | Virtual optical bench, lightfield photography, fourier optics, wavefront coding |
Project assignment 1 due Project assignment 2 out |
4 | Digital illumination, Hadamard coded and multispectral illumination | |
5 | Emerging sensors: high speed imaging, 3D range sensors, femto-second concepts, front/back illumination, diffraction issues. |
Project assignment 2 due Project assignment 3 out |
6 | Beyond visible spectrum: multispectral imaging and thermal sensors, fluorescent imaging, 'audio camera' | |
7 | Image reconstruction techniques, deconvolution, motion and defocus deblurring, tomography, heterodyned photography, compressive sensing |
Project assignment 3 due Project assignment 4 out Final project start |
8 | Cameras for Human Computer Interaction (HCI): 0-D and 1-D sensors, spatio-temporal coding, frustrated TIR, camera-display fusion | Final project pre-proposals due |
9 |
Short presentations on final project pre-proposals Useful techniques in scientific and medical imaging: CT scans, strobing, endoscopes, astronomy and long range imaging | |
10 | Mid-term exam | |
11 | Mobile photography, video blogging, life logs and online photo collections | |
12 | Optics and sensing in animal eyes: what can we learn from successful biological vision systems? | |
13 | Future products and business models | Final project proposals due |
14 | Final project presentations |
Project assignment 4 due Final project due |
Acknowledgements
The course material has been prepared with slides, discussions and other contributions from many people. Only some of them are listed below and my apologies. Thank you all!
Jack Tumblin, Northwestern University, Shree Nayar, Columbia University, Amit Agrawal, MERL, Marc Levoy, Bennet Wilburn, Stanford U., Alyosha Efros, CMU, Steve Seitz, UW, Irfan Essa, Georgia Tech, Fredo Durand, MIT, Jingyi Yu, Delaware, Aseem Agrawala, U of Washington, Paul Debevec, USC, Todor Georgiev, Adobe, Hendrik Lensch, MPI.
I also want to thank many who have donated additional equipment, cameras, sensors, optics, software etc.
Fatih Porikli, Amit Agrawal, MERL, James Davis, UC of Santa Cruz, 3DV, Eddy Talvala and Andrew Adams, Stanford.