Page History

...

RC is frequently described as being dependent on RA.  However, during the Lakelse release, it became apparent that RC is, at best, loosely coupled to RA.  More accurately, RC seems to be based primarily on OpenStack releases. The apparent assumption is that this will result in sufficient, although unknown, coverage for RA.  There doesn't appear to have been any attempt to understand or even read the RA requirements during the development of RC.

1. M2 requires a description of the coverage of RA.  I was provided a link to RC documentation that does not provide specific mapping to RA requirements.  In fact, there aren't even any references to RA requirement numbers.
   1. This means that we do not have an accurate, or even approximate, measurement of RA coverage.
   2. We also cannot describe how coverage has changed from one release to the next.
   3. This is a contradiction of the RC documentation, itself.  For example:  "The conformance specifications must provide the mapping between tests and requirements to demonstrate traceability and coverage."
2. The RC documentation includes lists of test case names.  However, there are no links to detailed information about the test cases.  
   1. For example, a requirement author, or other stakeholder, might want to understand how a requirement is being tested.
   2. The engineering team conducts a review of RA documentation for each release.  Sometimes they want to see how a requirement is being tested.

...

1. M1 requires a validation plan for RC.  Instead of providing a validation plan, I was provided a pointer to the RC documentation.  The result of this is that the community has no documented description of how RC is being validated.
2. Based on feedback from the meeting, the validation testing does not include insertion of faults to ensure that test cases fail when they should.  This may result in false negative tests.
3. The validation testing seems to be conducted in just a single configuration/environment.  This may result in configuration/optimization around a single environment and in turn, inaccurate test results in a valid, but different environment.
4. There is no dashboard or other convenient means to see the results of validation testing.  Also, there are no periodic reports or other updates on the results.