2.4 Factors affecting defect "reporting"

Mib Software
High Reuse Software
Development and Consulting
We extend and customize
reusable and open software.

Some Implications of Bazaar Size

Third Draft. Aug 11, 1998
Copyright 1997-1998, Forrest J. Cavalier, III Mib Software
All rights reserved. Comments welcome!

2.4 Factors affecting defect "reporting"

In the bazaar model as described by Raymond, it is assumed that any defect which is noticed is reported back to the bazaar. But this is not necessarily true.

The likelihood of "defects reported after being detected" is primarily influenced by social factors. A dedicated, involved developer is likely to complain because of an awareness of importance of reporting defects. But "dedicated developer" describes a minority of most participants.

Factors reducing effective size

Effort to report. "Advertising" a defect takes work. Traditional marketing studies of dissatisfied customers can indicate a 10:1 or 100:1 ratio of dissatisfied vs complaining. This has profound implications for defect reporting also.
Difficulty to report. Is there a known point of contact and easy method of reporting? Are international users able to participate? (Is there a language barrier?)
Existence of convenient alternatives. Need-driven users are likely to remain silent and use an alternative (a previous implementation, work around, or competing product, or just "go without.") in the time it takes to report.
Incorrect attribution to operator error. If need-driven consumers have a low expectation of defects, they have a strengthened motivation to assume that erroneous operation is due to their own mistake instead. This is a learned response....Since they are not expert users, they will frequently be encountering problems that they did (in fact) cause. When the occasional defect does pop up, it will be dismissed because the previous observations were resolved by correcting their own mistake.
Reluctance to duplicate existing defect reports. "This one is so obvious, they must already know." Or: "I'm using an older release, I have to expect bugs." Or: "No one is interested in hearing about bugs in older releases." (Sometimes older releases are used for convenience, even when a new release is known. Sometimes newer releases are abandoned for older due to problems.)

In fact, there can be many defects "carried over" to successive releases when there is a"release often" policy.
Reluctance to risk a "false alarm" (not a defect, but due to a mistake or incorrect usage by the user.) Does the community embarrass those who report false alarms? Does it complain about the work necessary to deal with false alarms?
Lack of motivation/reward for reporting defects. Regardless of whether the defect report is valid or a false alarm, the participant who reports it would appreciate assistance. This may be a pointer to an appropriate fix, or workaround, or a reference to documentation which explains the proper usage. If the team is overwhelmed temporarily or chronically, and cannot respond (or has no-one assigned to respond) then the user will not even know that their report was received, much less appreciated. The reporting participant may have no choice but to work around the defect, or find other alternatives (because waiting for a solution is an unacceptable delay.)

Strategies to counter low effective size

Make it easy to report. Provide a well-known and simple method accessible to all users. (e-mail) Build it into the software as a feature. Cultivate "points of contact" who can function as translators for language barriers.
Make the novice user aware of what events are actually reportable defects, not due to their mistake. Do internally detected and critical defects have a different error coding scheme and logging location which makes them "stand out"?
Reduce user error rate. Products which are complicated to use and setup are going to have a large number of false positives - so lowered complexity helps a lot.
Reward the defect reporters. It is important to provide personal reward. A message sent by an auto-responder "that your report was received" is not personal reward. A detailed, prompt response which assists the user is preferred. If this is not possible (responding to bug reports requires effective power) then find alternatives: Perhaps a searchable hypertext defect repository or FAQ. Just take a look at many existing FAQs: many of the questions have to do with reported errors. The reporter will be motivated to search, find out that it was their problem, and be given the solution. This helps them, and eliminates a report of a false alarm.

It is important to respond to all defect reports. Unfortunately, in a mature product, many defect reports are not actually defects, but "false alarms": and incorrect usage by the user. Responding to every report may require writing 99 responses to get 1 real defect report. This may actually be cost-effective.
Cultivate a culture of 0 expected defects. Make it obvious that reporting defects in old versions is important too, since it might be a defect carried over to a new version.
Don't punish false alarm reports.
Motivate complainers (to increase the number of complainers.)

Strategies to work with reduced effective size:

Remove the human factor. Detect and report defects back to the developers automatically.
Cultivate the attitudes of a select "small group" who will always report defects and anything they notice.
Motivate complainers (to increase the quality of each complainer.) The bazaar debugging model will support a fewer number of customers if the customers are motivated to complain. Even if they complain about features which do not exist yet, it will provide data for deciding what should be done next.

Next: 3. Alternate Debugging methods

Up to: 2. Debugging by testing dramatically lowers effective size
Up to: Some Implications of Bazaar Size