2.1 Factors affecting defect "installation"

Mib Software
High Reuse Software
Development and Consulting
We extend and customize
reusable and open software.

Some Implications of Bazaar Size

Third Draft. Aug 11, 1998
Copyright 1997-1998, Forrest J. Cavalier, III Mib Software
All rights reserved. Comments welcome!

2.1 Factors affecting defect "installation"

A defect remains hidden until it is installed and that code used.

"Installation Erosion"
Over the product development lifetime, the number sites where certain code is installed is generally a decreasing percentage of total sites.

After a product stabilizes at a some level of installed sites (determined by market share with respect to other products, market growth, etc) there are tendencies which will fragment or "erode" the effective working size, even if market share for all versions of the product is maintained (or even grows!)

Competition. If a competing product is installed, obviously the defect is not reachable. New products may take some market share from existing products.
Upgrade resistance. If a new version is released, some sites won't upgrade (cost, hassle, etc.) Unless there is a compelling reason to upgrade, many sites will not wish to risk changing something which is already working. The latest version won't be installed in as many sites as the previous release.
Feature saturation. As a software product matures, the features which are added are more specialized and therefore desired and in use at a smaller and smaller percentage of sites. The features added are not compelling reasons for upgrades, and if an upgrade is done, the new features may not ever be used.

Raymond briefly mentions this "erosion" effect due to what I term "upgrade resistance" and "feature saturation", but does not discuss its implications.

Actually, as I revise in late May 1997 the list is beginning to lose members for an interesting reason. Several people have asked me to unsubscribe them because fetchmail is working so well for them that they no longer need to see the list traffic! Perhaps this is part of the normal life-cycle of a mature bazaar-style project.

Large, popular bazaars are not immune to the effects of erosion. For example, although the customer base for all versions of Linux is quite large, with each release the customer base for the "current" release erodes. And secondly, additional features are going to have only a subset of the current user base as interested (and testing) stakeholders. Large bazaars will also indicate an attractive market inviting commercial or competition by other bazaars (which is less frequent.)

One implication of this observation is that it is wrong to assume that all code is covered equally by testing for a certain length of calendar time just because your "bazaar is large." Later releases will require a longer calendar time to reach the same level of run-time hours. This will slow progress as the product matures.

Fewer sites upgrading in addition to fewer sites which will be using those new features is a "double whammy" for defect detection..

(There is another way that installation erosion occurs: bazaar fracturing. See 4. Other Factors which erode effective size for a discussion.)

Strategies to maintain effective size:

Discourage alternatives: Provide More value than the competition
(One problem with "debugging by testing" is that some customers will expect a certain level of quality, (perhaps "production stable") and they will be disappointed when they find defects in a quickly released version. They may be induced to switch to a competing product with a lower expected number of defects.)
Expand into new markets/customer bases. This gains new size AND provides new stress points.
Provide compelling reasons to upgrade
- "beta" versions which expire.
- Compelling new features. (May be hard to do unless the new capability is actually necessary to keep up with the status quo (traffic growth, etc, grows to require the upgrade.) or an obvious new use with utility in most sites.
- Culture/Social issues and motivators.
Reduce the risk and effort to upgrading.
Less frequent releases might also be a strategy. If a release is to add new features, with no "core" code changes, then frequent releases may not be harmful: Consumers will jump on as soon as they see the feature they want, and they will be testing it. (This still does leave out the users who would have been using the core code, but found a work-around, or decided to "do without" and settled for previous version.) If core code is changing, then frequent releases dilutes the number of users testing the changes of core code. Unfortunately, less frequent releases is at odds with the mechanism outlined by Raymond:

"In practice, the theoretical loss of efficiency due to duplication of work by debuggers almost never seems to be an issue in the Linux world. One effect of a ``release early and often policy'' is to minimize such duplication by propagating fed-back fixes quickly."
Perhaps releasing defect fixes often but releasing new features less often is a good strategy. This is in fact how many open development projects work. Defect fixes are often available as "patches" or "minor version number upgrades."

Next: 2.2 Factors affecting defect "reachability" and "triggering"

Up to: 2. Debugging by testing dramatically lowers effective size
Up to: Some Implications of Bazaar Size