Mapping Measurable Outcomes

In this section, Professor Karen Willcox describes the measurable outcome index used in 16.90 Computational Methods in Aerospace Engineering. She describes her efforts to map measurable outcomes, how doing so benefits students, and how students have responded to these efforts.

Xoces: Visualizing Relationships between Measurable Outcomes

Measurable outcomes are skills students should be able to demonstrate after successfully completing a course. The outcomes are stated in a manner that they can be measured. For several years, I have been mapping measurable outcomes across the Department of Aeronautics and Astronautics curriculum through a project called Xoces. At this point, we have mapped outcomes for all of our undergraduate classes.

I began this project because, through teaching various parts of the classes and then being in the role of Associate Department Head, where I oversaw the educational aspects of our department, I became aware that, while a first important step is to have clear, measurable outcomes in your own class, a missing piece of the puzzle was understanding how the outcomes within individual classes connected across the departmental curriculum. I wanted to know how an outcome in one class might depend on outcomes in other classes and how the outcomes were working together to form an overall set. This kind of understanding was key to articulating, for accreditation purposes, what it meant to be a graduate of our program. It also became particularly salient when we were thinking about curriculum revision across the department. We realized that changes in one section of the curriculum would impact other parts of the curriculum.

Xoces visualization of learning outcomes across MIT Aeronautics and Astronautics subjects. See The Mapping Lab website for more about our educational mapping efforts.

The 16.90 Measurable Outcome Index

We have 59 measurable outcomes in 16.90 Computational Methods in Aerospace Engineering. Our Measurable Outcome Index provides students with links associated with each of the measurable outcomes for the course, arranged according to three units. Each unit corresponds to a set of outcomes students will be able to demonstrate upon successfully completing the course. The entire course is designed to help students achieve the outcomes. Further, the various assessment problems and exams in the course are designed to address one or more of the outcomes.

By clicking on a measurable outcome in the index, students see the content they can use to learn about that measurable outcome and the content that we use to assess their understanding of that measurable outcome. In fact, as we converted our notes for use on the online platform, we tagged every section of the notes and every reading question with the learning outcomes they addressed. We did this so that if students are sitting down to read a particular section of notes in preparation for class the next day, they will know exactly what we expect them to be able to do as they read the section. We want them to be able to view the measurable outcome tag, read the section, and then go to the assessment questions and ask themselves: "Can I demonstrate each measurable outcome?"

Benefits for Students

We have designed our online course site in such a way that if students are browsing by lecture they can see how each subsection in the lecture relates to the corresponding measurable outcome. But it’s also designed in such a way that students can use the Measurable Outcome Index to identify all of the places throughout the semester in either the lectures or the assessment questions where there is material that relates that particular outcome. I think the latter way of searching by outcome is particularly useful when students are reviewing for the final exam. They can click on a particular outcome and clearly identify the lectures in which the outcome was addressed. It helps them see how different parts of the class are connected.

This is important because there are relationships between the outcomes in each of the units in the course  For example, some of the skills students acquire in the first unit on Numerical Integration of Ordinary Differential Equations are used later in the second unit on Numerical Methods for Partial Differential Equations. There are also outcomes in other classes, such as 18.03 Differential Equations, that students need to master in order to be able to engage with the material in 16.90 Computational Methods in Aerospace Engineering. Furthermore, there are outcomes in 16.09 Statistics and Probability that relate directly to the outcomes in 16.90 Computational Methods in Aerospace Engineering.

Mapping these measurable outcomes makes these relationships explicit. Xoces, in turn, makes it possible for students to visualize the outcomes and their relationships. Providing this kind of mapping tool is important because when a student arrives in 16.90 Computational Methods in Aerospace Engineering, and I ask them to analyze the stability of integration methods for ordinary differential equations using Taylor series expansions, they can easily identify the prerequisite skills they need to master this outcome and in which courses they first learned them. They can quickly access the resources they need to refresh their understanding of the skills needed to complete the current task.

Students’ Perplexing Response

I like to understand how pieces of what I’m learning connect to each other. I would have loved to have had outcome mapping when I was a student. What I have struggled with is that my students don’t really seem to care that much about the outcomes or the outcome mapping. I have never had students say, “Oh, this is great! I wish we had this in all of our classes.” In fact, on the end-of-semester surveys, there’s a question that asks students to evaluate the degree to which the outcomes in the course were clear. Statistically, I get the same rating in all the years I’ve very taught the course, regardless of whether or not I share the outcome mapping with them.

I've talked with colleagues about students’ perplexing response to outcome mapping. One colleague suggested that, because we don’t use measurable outcomes across the Institute, we condition students not to value them. A different colleague thought perhaps students don’t have time to think about the measurable outcomes. They just want to get through the material and get the best grade they can. But that’s exactly what mapping outcomes is designed to help them do! Another view is that perhaps the Xoces project is too meta for students. The mapping makes sense to faculty because faculty have, over many years, encountered all of the individual outcomes and understand how they they all fit together. But novices are just encountering the individual outcomes for the first time. Expecting them to appreciate the “big picture” until they’ve delved into the individual components may be unrealistic. That kind of perspective may only come with maturity and experience using the materials.

I want to understand why it is that students don’t find an outcome-driven course organization compelling. Understanding what the barrier is will be critical because if we ultimately want to provide truly flexible course experiences for students—ones in which they don’t have to learn the material sequentially, and can move through the material at their own pace—we will need measurable outcomes to form the foundation of our course structures.