Users as Oracles: Semi-automatically Corroborating User Feedback

Andy Podgurski
Case Western Reserve University
http://vorion.case.edu/~andy/andy_podgurski.htm

Abstract:

Once software is deployed to the field, ordinary users become the
primary arbiters of whether its behavior is acceptable.  Surprisingly,
few software developers equip their products with a mechanism to
elicit useful reports concerning "soft" failures, which do not crash
or hang the product.  Anecdotal evidence suggests that one reason for
this is that developers consider user reports about such failures to
be unreliable.  To address this issue, we are investigating methods
for corroborating feedback from individual users by relating it to
feedback from other users concerning executions with similar internal
dynamics.  In this talk we will describe some of these methods and
present evidence that they can compensate for a substantial degree of
mislabeling by users.