Versions Compared

Old Version 28

changes.mady.by.user Unknown User (blsaws)

Saved on

compared with

New Version 29

changes.mady.by.user Unknown User (christopherprice)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

This page is intended as a working draft of how we will look to leverage and develop test cases for the dovetail project.

Table of Contents

Brainstorming

This section is intended to help us discuss and collaboratively edit a set of guiding principles for the dovetail program in OPNFV.
  • If you want to edit this page, please make sure that you add items/bullets/sub-bullets with you name identified as e.g. [bryan] so that we can track who has contributed the perspectives. We should work together to tweak the items/descriptions, so perspectives and responses added by others should be addressed asap, so that this becomes an effective tool for driving consensus.
  • This is itself a strawman proposal for how we can discuss this important topic. Alternatives are welcome, although the basic goal is that we can come to collaborative consensus as quickly as possible, with minimal barriers to participation in the discussion.
1) Focus on OPNFV role and unique aspects, e.g.
  • Integrated reference platforms
  • Component diversity
  • OPNFV-driven features 
  • The  basic assumption is that the scope of compliance testing falls within the included features of a given OPNFV release (as its upper bound)
2) Leverage upstream frameworks
  • wherever possible, verify components using upstream frameworks as a complete package (vs curating specific tests)
  • work proactively to further develop those frameworks to meet OPNFV needs.
3) Deliver clear value: release only tests that add clear value to SUT providers, end-users, and the community
4) Agile development with master and stable releases
  • fast rollout with minimum viable baseline, and continuous agile extension for new tests
  • Support SUT assessment against master and stable releases.
5) Establish and maintain a roadmap for the program
  • basic platform verification
  • well-established features, i.e. features that have become broadly supported by current OPNFV distros)
6) Move tests upstream asap
  • as soon as a home community is found, focus on upstreaming all tests  
  • I'm not sure how this will fit with a logo program, because in those programs you want to keep your tests stable and usually under your control, this seems to be more of a principle for the general test projects.
  • Upstream test cases that we use are always version referenced so the stability is tied the speed of development in that community. The issue is less of stability but more of the compatibility of different versions of different components in the reference platform. I think this may be one of the differences between traditional testing scheme and open source.
  • By the way, I think the integrated tests (inter-components) could reside in OPNFV, uniqueu features are another example, we may also be the natural home for nfv use cases etc.
  • one more point: upstream testing is sometimes developer-oriented, not always focused on compliance. in such cases, we need a way of either enhancing the upstream test suite or coming up a methodoly in dovetail.

Further Details

Dovetail Test Suite Purpose and Goals

...

hongbo: the same as those defined in the phase 1 and phase 2

Additional Brainstorming

This section is intended to help us discuss and collaboratively edit a set of guiding principles for the dovetail program in OPNFV.
  • If you want to edit this page, please make sure that you add items/bullets/sub-bullets with you name identified as e.g. [bryan] so that we can track who has contributed the perspectives. We should work together to tweak the items/descriptions, so perspectives and responses added by others should be addressed asap, so that this becomes an effective tool for driving consensus.
  • This is itself a strawman proposal for how we can discuss this important topic. Alternatives are welcome, although the basic goal is that we can come to collaborative consensus as quickly as possible, with minimal barriers to participation in the discussion.
1) Focus on OPNFV role and unique aspects, e.g.
  • Integrated reference platforms
  • Component diversity
  • OPNFV-driven features 
  • The  basic assumption is that the scope of compliance testing falls within the included features of a given OPNFV release (as its upper bound)
2) Leverage upstream frameworks
  • wherever possible, verify components using upstream frameworks as a complete package (vs curating specific tests)
  • work proactively to further develop those frameworks to meet OPNFV needs.
3) Deliver clear value: release only tests that add clear value to SUT providers, end-users, and the community
4) Agile development with master and stable releases
  • fast rollout with minimum viable baseline, and continuous agile extension for new tests
  • Support SUT assessment against master and stable releases.
5) Establish and maintain a roadmap for the program
  • basic platform verification
  • well-established features, i.e. features that have become broadly supported by current OPNFV distros)
6) Move tests upstream asap
  • as soon as a home community is found, focus on upstreaming all tests  
  • I'm not sure how this will fit with a logo program, because in those programs you want to keep your tests stable and usually under your control, this seems to be more of a principle for the general test projects.
  • Upstream test cases that we use are always version referenced so the stability is tied the speed of development in that community. The issue is less of stability but more of the compatibility of different versions of different components in the reference platform. I think this may be one of the differences between traditional testing scheme and open source.
  • By the way, I think the integrated tests (inter-components) could reside in OPNFV, uniqueu features are another example, we may also be the natural home for nfv use cases etc.
  • one more point: upstream testing is sometimes developer-oriented, not always focused on compliance. in such cases, we need a way of either enhancing the upstream test suite or coming up a methodoly in dovetail.