Affecting U.S. education through assessment : new tools to discover student understanding

Abstract

We may have a serious problem with education in the United States. However, the literature suggests one can arrive at differing conclusions about the efficacy of the American educational system depending on what we measure, how we measure it, when we decide to measure, and why we measure. As will be shown, many testing methods currently in vogue in the United States generate data that can lead policy makers, parents, educators, and even students to arrive at invalid conclusions about schools; teachers, and student ability, especially when evaluating the ability of a student to apply (versus just know) concepts. It was hypothesized that, if applied in a manner aligned with accepted validity standards, modern computer technology could both dramatically improve the accuracy of our inferences, and provide significant new insights into student learning and understanding given present national and California state standards. As a "proof of concept", a quasi-experimental, interrupted time series study was conducted using a computerized learning and assessment tool to observe second semester high school chemistry students solving qualitative chemistry problems. The results presented here suggest that: -- Without intervention, once a student chooses a strategy to solve a problem, the student will continue to use the same type of strategy (in both the near- and long-term), whether or not that strategy has proven effective. These findings imply we now have the opportunity to both diagnose ineffective strategies as they are developing, and tailor interventions to individual student needs. -- Because technology allows us to look both at a student's answer and how s)he arrived at that answer, we conclude that we can accurately infer whether a student really understands the concepts of a particular knowledge domain. Our findings suggest that, if properly employed, technology can offer new, real-time insights into student understanding. The paper concludes by discussing the applicability of this research to other knowledge domains, some avenues of future research, and particular pedagogical interventions which the results suggest might be most promising.