Versions Compared

Old Version 7

changes.mady.by.user Unknown User (wenjing)

Saved on

compared with

New Version 8

changes.mady.by.user Unknown User (wenjing)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Timeline:
    1. Project planning complete March 10. TSC review on March 21. Submit to TSC and Board for review/feedback. Test strategy doc by end of April.
    2. Launch announcement in OPNFV Summit (12-15 June). Vendor participation. Tutorial & training. 
    3. Danube release: end of June (~Danube + 3 mon). Launch the CVP program: ~July.
    4. Draft milestones timetable: see below "Project Milestones and Due Dates"
      Action: review and agree on Milestone table below
  2. General strategic approach to "Danube Qualification Testing" for CVP:
    1. Scope: Danube qualification testing will qualify compliance to OPNFV Danube release and will test OPNFV as a software platform, i.e. a well-defined set of components with associated interfaces, behaviors and features.
                  Danube qualification testing will include interface compliance testing, capability compliance testing, feature compliance testing of the OPNFV platform.                
                  The OPNFV Danube platform is defined as the sum of all the scenarios participating in OPNFV Danube release. Note that simple interface compliance testing is not seen as sufficient for Danube platform qualification.
    2. OPNFV brand value through Danube Qualification Testing: Danube qualification testing is to ensure that all values provided by the OPNFV Danube release are covered. The Danube qualification testing is to set a "high bar" for passing qualification tests, to ensure OPNFV Danube is understood as high-value.
    3. Controlled process: The Danube qualification testing needs to ensure that test results are objective, repeatable, well-documented and cannot be modified or cheated on. Testing and test results documentation needs to be carried out by an entity independent from the party providing the system under test.

      [Wenjing]: Refer to bullet b for scope, and refer to value or high level approach discussions in C&C committee. Here we focus on project planning of dovetail given those guidances.
      Action: None. See CVP document
  3.  The SUT for Danube cycle CVP is:
    1. Products from vendors (not OPNFV release artifacts themselves). Development tooling, such as CI/CD, is not part of the SUT. The vendors can bring up the SUT to a pre-Dovetail-test state in any way they choose (and Dovetail will provide documentation to help do so)
    2. The SUT can consist of hardware, NFVI software, and VIM software in one System Under Test. Or, a software only vendor can use third party or white box hardware to be tested as a whole and if the combined whole passes the test suite, the CVP label applies to the software. The Danube cycle does not plan to test hardware-only systems.
    3. To conduct the test (self test or third party test), the tester needs to compose the System Under Test (hardware, NFVI, and VIM) and bring it to the pre-Dovetail-test state. The tester then conducts the Dovetail tests using its automation tool which will report the result directly to a database managed by OPNFV.  (see bullet #4)
      Action: agree on SUT
  4.  Test strategy document - blueprint of what to test, how to test
    1. links to Bryan's etherpads: https://etherpad.opnfv.org/p/dovetail-principles,  https://etherpad.opnfv.org/p/dovetail-priorities (already entered into the test case requirement wiki page under brainstorming)
    2. publish by end of April
      1. Defines scope of what should be tested, and what is out of scope
      2.  Refers to how tests are executed, what types of tests are executed
      3.  Defines the test environment, and prerequisites for SUT being tested
      4. Specifies tools to be used for the test process
      5. https://en.wikipedia.org/wiki/Test_strategy  <- :D
      6. How (and why) to create one: http://www.guru99.com/how-to-create-test-strategy-document.html

        [Wenjing]: in order to finalize a plan by March 10, we need the main outline of the strategy document (and the main issues) agreed to by March 10 also. Not all details, but main ones that can impact the credibility of the plan.
        Action 1: Summarize the brainstorm information and update criteria in wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Case+Requirements. Publish it. Open call for review.
        Action 2: User facing documentation
  5. Scope and precise details of the Danube test suite are to be determined by the following process:
    1. Dovetail scope is a subset of functionalities contained in the OPNFV reference platform(s). When we say Danube based, it further narrows down to the functionalities found in Danube release cycle.  In other words, this sets the upper bound. 
    2. C&C has further reduced the scope for Danube cycle to be NFVI+VIM, and performance testing is currently out of scope.
      1. Plug-in C&C addendum on Danube scope. 2/27?
    3. A set of criteria is to be met by the test cases to be included in the Dovetail test suite. A draft of this set of criteria is in the wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Case+Requirements.  A draft superset of all possible tests is currently maintained in the wiki: https://wiki.opnfv.org/display/dovetail/Dovetail+Test+Areas+and+Test+Cases. However, we are taking a fresh look again with a new work/review process, tracked by Jira, to determine how to organize the test suite, what test areas and what test cases for each test area should be included in Danube CVP. The Jira ticket is https://jira.opnfv.org/browse/DOVETAIL-345. Once this work is concluded, the wiki's will be refreshed to reflect its outcome. Proposal for DOVETAIL-345:
      1. Openstack defcore /refstack integrated in functest
      2. SDN controllers as integrated via Neutron plugins including nosdn, odl, onos, ocl. Test cases driven via Neutron interface. (covers os-nosdn-*-ha, os-odl*-*-ha, os-onos-*-ha and os-ocl-*-ha)
      3. Nfv network basic validation: vPing, vRouter, ...
      4. Nfv use case: vIMS as integrated in functest
      5. HA : as in HA project
      6. IPv6 : as in IPv6 project
      7. others: we can call for proposal, once criteria published
    4. This process is open to all in the community and consensus based. Participation is on Jira (and aided by etherpads as needed) and dovetail calls publicized in the tech-discuss mailing list. In specific cases where consensus can't be reached by this process, requirement/scope issues go to C&C committee/Board, and technical issues go to TSC for determination.
      Action: General agreement on starting point, supporting data needed, and extended review sessions for DOVETAIL-345 until resolved (do we need more than the weekly call time?)

      /* next week march 17, we will continue on 6, 7, 8 */
  6.  Dovetail will deliver all toolings, documentations and the test suite based on Danube, with the end of June as the final release date. Deliverables include:
    1. standalone test automation tooling
    2. an official test suite for CVP (as the result of 2d process), an extended test suite for feedback or experimental review (ie useful/desirable for other benefits but not accepted by the 2d process)
    3. documentation that includes detailed guides on how to prepare and conduct the Dovetail test, pass criteria, report/review process. Documentation that includes a summary description of test areas and test cases with links to online details.
    4. the above test tooling and test suites and docs must be validated using OPNFV and product references. (ci process, open test/review process, vendor beta process)
    5. hosting the dovetail test result database, and any additional work needed to enable CVP administrative process (eg UI). test the hosting.
    6. need to make quick design decisions on: (i) test result in databases (vs. in files) (ii) use refstack directly is feasible in Danube (vs. later in E). (already in progress)
      Action: agreement on the list and put into JIRA 

  7.  Working with upstream
    1. We are currently evaluating to work with OpenStack Interop WG/Defcore by using refstack tooling for OpenStack testing needs within dovetail. Jose/Howard had a meeting during Atlanta PTG on this topic and OpenStack community seems to be very supportive. Working with upstream helps streamline work and leverage expertise. The current outstanding action item is to have the quick evaluation to help us determine (i) if this is the right direction (ii) if yes, can we deliver this in time for Danube cycle? If time does not permit us doing so in Danube cycle, we will roadmap it to E.
    2. Longer term plan for other upstream projects should follow the roadmap process below.
      Action: pursue Openstack for Euphrates(?)
  8.  Long term planning of roadmap
    1. Desired features for dovetail, if not possible/accepted in Danube, should be road mapped using Jira enhancement tickets. Features targeted for E release should start immediately in order to influence E release planning that is ongoing in the community.
    2. EUAG and other user inputs should be incorporated into the roadmap using the same Jira backlog/to-do ticket mechanism. (Write a story for each request)
    3. Dovetail's E cycle planning will be based on these Jira backlog/to-do tickets. Use fixVersion "Future Release" instead of "Dovetail Danube".
      Action: all inputs should be entered in JIRA for dovetail consideration

...