|
Abstract:
|
We propose and test new "polyhedral" question design and estimation methods that
use recent developments in mathematical programming. The methods are designed to
offer accurate estimates after relatively few questions in problems involving many
parameters. With polyhedral question design, each respondent's questions are adapted
based upon prior answers by that respondent to reduce a feasible set of parameters as
rapidly as possible. Polyhedral estimation provides estimates based on a centrality
criterion (the "analytic center" of the feasible parameter set). The methods require
computer support but can operate in both Internet and other computer-aided
environments with no noticeable delay between questions.
We evaluate the proposed methods using two approaches. First, we use Monte Carlo
simulations to compare the methods against established benchmarks in a variety of
domains. In the simulations we compare polyhedral question design to three
benchmarks: random selection, efficient Fixed designs, and Adaptive Conjoint
Analysis (ACA). We compare polyhedral estimation to Hierarchical Bayes estimation
for each question design method. The simulations evaluate the methods across different
levels of respondent heterogeneity, response accuracy, and numbers of questions. For
low numbers of questions, polyhedral question design does best (or is tied for best) for
all domains. For high numbers of questions, efficient Fixed designs do better in some
domains. The best estimation method depends on respondent heterogeneity and
response accuracy. Polyhedral (analytic center) estimation shows particular promise
for high heterogeneity and/or for low response errors.
The second evaluation employs a large-scale field test. The field test involved 330
respondents, who were randomly assigned to a question-design method and asked to
complete a web-based conjoint exercise. Following the conjoint exercise, respondents
were given $100 and allowed to make a purchase from a Pareto choice set of five
new-to-the-market laptop computer bags. The respondents received their chosen bag
together with the difference in cash between the price of their chosen bag and the
$100. We compare the question-design and estimation methods on both internal
validity (holdout tasks) and external validity (actual choice of a laptop bag). The field
test findings are consistent with the simulation results and offer strong support for the
polyhedral question design method. The preferred estimation method varied based on
the question design method, although Hierarchical Bayes estimation consistently
per-formed well in this domain.
The findings reveal a remarkable level of consistency across the validation tasks. They
suggest that the proposed methods are sufficiently promising to justify further
development. At the time of the test, the bags were prototypes. Based, in part, on the
results of this study the bags were launched successfully and are now commercially
available. Sales of the features of the laptop bags were consistent with
conjoint-analysis predictions. |
