Identification and resolution of problems with methodology used in selection of technological concepts for R&D support

Abstract

How to better evaluate the merits of competing technological concepts. This project is concerned with evaluation of the merits of competing technological concepts and managing the R&D investments needed to bring them to fruition. A weakness of the current R&D process is that arguments regarding who should receive funding come from a concept's proponents, who are usually interested and biased. More objective evaluation methods are needed. As part of this work it is important to understand better how qualified experts evaluate technological concepts. Can a probabilistically formulated method of integrating knowledge of various performance attributes provide better understanding of the likely performance of a technological concept? This is the question of interest. A nuclear power plant example (impetus for the actual study below). The impetus for the study began with The U.S. Department of Energy's Generation IV advanced reactor technology program, the program that will select the next generation of nuclear reactors. Generation IV chose twenty-seven criteria for use in determining which nuclear power plant concepts would be best for a given mission. These criteria came in the form of twenty-seven questions asked of prospective concept designers.

(cont.) The concept designers ranked their own design over a range of seven bins and specified a peak in the most likely bin. The 27 criteria were assumed to be independent and were used in creating three major goals (sustainability, safety & reliability, and economics). That is, the score assigned in each of the 27 areas was rolled into 3 major scores called goals in this study. Weights, unknown to the concept designers, were assigned to individual questions and the three major goals, and then probability mass functions were created predicting the success of a given design. A robot design course. At the Massachusetts Institute of Technology (MIT) between Fall semester and Spring semester (i.e., during January) an independent activities period (IAP) offers the opportunity for students to design robots in MASlab (Mobile Autonomous Systems Laboratory) Robotics Competition, also known as course 6.186. Therefore, course 6.186 provides an opportunity for evaluating technological concepts (i.e., in the form of a robot design as well as operational contests of those designs). Course 6.186 provides an opportunity for students to act as consultants in offering their expertise in the evaluation of robots designed by themselves and their competitors.

(cont.) The evaluations are composed of questionnaires similar to those described in the nuclear power example. The consultants' responsibilities are to evaluate the strengths and weaknesses of other robots. The quality of the evaluations is indicated by the results of the robot competition. From this experience we learn more about how objective evaluations of the performance of competing concepts can be made. As coordinators of this effort, we identify the methodology of those consultants who were most successful in identifying, before testing, the best robot designs. The methodologies thus identified can be extended to large-scale projects in general such as identification of the best, among competing, technological concepts.