9. As internal releases are produced, start evaluating them, and feed results back

Ideally, each capability can be used as a usable product, but that is not always possible, as it is often the case that several capabilities are needed to create a usable solution. So the usability of each capability by itself is one of the variables that must be decided. In any case, completing a working capability is a major milestone, because it eliminates a risk: the risk that the capability cannot be achieved.

Define development processes such that they frequently produce complete internal releases of the product with all of the complete and incomplete capabilities. Frequent internal releases are important for two reasons:

  1. They help you to practice, refine, and harden the processes needed to produce a release.

  2. They enable you to test the current capabilities as an integrated product.

SpaceX does this with its rocket designs: when they develop a new rocket, they frequently build complete systems, only to discard them. That is why there are several completed vehicles on display at their Boca Chica facility—early versions that will never be used again.

Real Users, Real Usage

Internal releases should be tested by real users who have volunteered to be early adopters or trial users. They should ideally use the internal releases for real work, as long as that does not put them or others at risk. In other words, we want to test the product outside of the test environment, to the extent possible without endangering anyone.

Highly Instrumented

Internal releases should be highly instrumented, so that when capabilities fail, there is ample diagnostic information available to determine the cause.

The resulting test data should be published so that all of the teams can access it for analysis. Teams might want to examine aspects of system performance that they did not anticipate prior to a test. Make it easy for them to obtain the data.

Revise Key Metrics

Evaluations of internal releases reveal the overall product performance. This might add clarity to what is actually important. It might be that the key metric that has been watched is actually not the right one: perhaps a slightly different metric is actually more revealing. Always be reflecting on what is being measured and what the key metrics are.

Systematize

As Elon Musk puts it, you are “building the machine that makes the machine”. The result of repeated evaluations of internal releases should be a robust system that is measuring the right things. The behaviors used to evaluate releases, reflect, and revise metrics are repeatable behaviors. They are not a process per se, but they are patterns that those involved will use again and again.



Subtopics


 

Step 8 <—

—> Step 10