This page is a temporary location to draft and validate the testing guide documentation. Once it's in a decent draft shape, I will submit it to documentation gerrit and respective section authors can complete the doc through gerrit review.

Reference: CVP OVP and Dovetail Terminology

I will use the terms defined in the above WIKI. Those terms will also be copied to the documentation's term section when they are in decent shape.

...

• Ubuntu 16.04.2 LTS (Xenial) for x86_64

• Ubuntu 14.04 LTS (Trusty) for x86_64
• CentOS-7-1611 for x86_64
   

...

The current CVP test suite includes Openstack DefCore testing through RefStack as a component. Therefore it is advised that testers should also consult documentation of Defcore qualification testing for additional guidance. The version of Refstack in CVP (should we use CVP1.x.y? )  is _1.0.0 is 2016.8. /*double check */

1.5.2.1 Openstack Versions

/* this list needs to be cross checked with refstack's support list */

/* also needs to be more precise what is listed here, what is not listed - we should probably list open source community tested only */

...

liberty

...

mitaka

...

newton

...

For now, following versions of open source Openstack have been tested:

• Kilo

• Liberty

• Mitaka

• Newton

It's not guaranteed that those versions not listed here or commercial versions of Openstack are well compatible with CVP_1.0.0. 

The list can be updated if additional versions have been tested.  

1.5.2.

...

2 Openstack POD Configuration

The SUT must be configured such that it consists of 3 controllers nodes in HA mode, and 2 compute nodes, as illustrated in the diagram below.

/*insert diagram */

.

• controller node( 3 or more)
• compute node( 2 or more)
• storage node(optional)
• network node(optional)

Above is the recommended deployment, but you can still try your own combinations of nodes.

The topologies vary in different deployment, below is a typical deployment:

/*insert diagram */

Additional Additional explainations /*what else needed here?? */

1.5.2.

...

3 Required Openstack Components

The following Openstack components are required,

...

/* any storage configuration needed? */

2 Conducting CVP Tests using Dovetail

2.1 Dovetail CVP Testing Overview

The Dovetail testing framework consists of two major parts: the testing client that executes all test cases in a vendor lab (self-testing) or a third party lab, and the server system that is under the OPNFV's administration to store and view test results based on OPNFV Test API. The following diagram illustrates this overall framework.

/* here is a draft diagram that needs to be revised when exact information is known and fixed */

Image Removed

This section mainly focuses on helping the testers in the vendor's domain attempting to run the CVP tests using Dovetail.

Dovetail client tool (or just Dovetail tool or Dovetail for short) can be installed in the jumphost either directly as Python software, or as a Docker(r) container. Comments of pros and cons of the two options TBD.

In 2.2, we describe steps the tester needs to take to install Dovetail directly from the source. In 2.3, we describe steps needed for installing Dovetail Docker(r) container. Once installed, and properly configured, the remaining test process is mostly identical for the two options. In 2.4, we go over the steps of actually running the test suite. In 2.5, we discuss how to view test results and make sense of them, for example, what the tester may do in case of unexpected test failures. Section 2.6 describes additional Dovetail features that are not absolutely necessary in CVP testing but users may find useful for other purposes. One example is to run Dovetail for in-house testing as preparation before official CVP testing; another example is to run Dovetail experimental test suites other than the CVP test suite. Experimental tests may be made available by the community for experimenting less mature test cases or functionalities for the purpose of getting feedbacks for improvement.

2.2 Installing Dovetail

Before taking this step, testers should check the hardware and networking requirements of the POD, and the jumphost in particular, to make sure they are compliant.

In this section, we describe the procedure to install Dovetail client tool that runs the CVP test suite from the jumphost. The jumphost must have network access to both the public Internet and to the O&M (Operation and Management) network with access rights to all VIM APIs being tested.

2.2.1 Checking the Jumphost Readiness

While Dovetail does not have hard requirement on a specific operating system type or version, these have been validated by the community through some level of exercise in OPNFV labs or PlugFests.

• Ubuntu 16.04.2 LTS (Xenial) for x86_64
• Ubuntu 14.04 LTS (Trusty) for x86_64
• CentOS-7-1611 for x86_64
• Red Hat Enterprise Linux 7.3 for x86_64
• Fedora 24 Server for x86_64 
• Fedora 25 Server for x86_64
  

If anyone has testify the dovetail running on additional operation system, this list can be updated after a approve test. 

Non-Linux operating systems, such as Windows, Mac OS, have not been tested at this time.

The tester should also validate that the jumphost can reach the public Internet. For example,

% ping 8.8.8.8

% ping https://www.opnfv.org/cvp

/* Need login credential to cvp server */

And also validate that the jumphost can reach the Openstack API, for example,

/* Need refinement on how to test this */

% ping x.x.x.x

2.2.2 Configuring the Jumphost Environment

/* First, openstack env variables to be passed to Functest */

The jumphost needs to have the right environmental variable setting to enable access to the Openstack API. This is usually done through the Openstack credential file.

Sample Openstack credential file environment_config.sh:

          /*Project-level authentication scope (name or ID), recommend admin project.*/

          export OS_USERNAME=admin

          /* Authentication password.*/

          export OS_PASSWORD=secret

          /* Authentication URL, one of the endpoints of keystone service. If this is v3 version, there need some extra variables as follows.*/

          export OS_AUTH_URL='http://xxx.xxx.xxx.xxx:5000/v3'

          /* Default is 2.0. If use keystone v3 API, this should be set as 3.*/

          export OS_IDENTITY_API_VERSION=3

          /* Domain name or ID containing the user above.  Command to check the domain: openstack user show <OS_USERNAME>*/

          export OS_USER_DOMAIN_NAME=default

          /* Domain name or ID containing the project above.  Command to check the domain: openstack project show <OS_PROJECT_NAME>*/

          export OS_PROJECT_DOMAIN_NAME=default

          export DOVETAIL_HOME=$HOME/cvp

Export all these variables into environment by,

          % source <OpenStack-credential-file-path>

2.2.3 Installing Prerequisite on Jumphost

 1. Dovetail requires Python 2.7 and later

     Use the following steps to check if the right version of python is already installed, and if not, install it.

    % python --version

 2. Dovetail requires Docker 1.8.0 and later 

     Use the following steps to check if the right version of Docker is already installed, and if not, install it.

    % docker --version

    As the docker installation process is much complex, you can refer to the official document: https://docs.docker.com/engine/installation/linux/

2.2.4 Installing Dovetail on Jumphost

A tester can choose one of the following two methods for installing and running Dovetail. In part1, we explain the steps to install Dovetail from the source. In part2, an alternative using a Docker image with preinstalled Dovetail is introduced.

part1. Installing Dovetail directly

...

a) Ubuntu 

         sudo apt-get update

         sudo apt-get -y install gcc  git  vim  python-dev  python-pip  --no-install-recommends

 b) centos and redhat

         sudo yum -y update

         sudo yum -y install epel-release

         sudo yum -y install gcc  git  vim-enhanced  python-devel  python-pip

 c) fedora

         sudo dnf -y update

         sudo dnf -y install gcc  git  vim-enhanced  python-devel  python-pip  redhat-rpm-config

p.s  When testing SUT's https service, there need some extra packages, such as apt-transport-https. This still remains to be verified. 

...

/* resource/configuration needed by IPv6 */

/* resource/configuration needed by HA */

/* resource/configuration needed by sdnvpn */

2 Conducting CVP Tests using Dovetail

2.1 Dovetail CVP Testing Overview

The Dovetail testing framework consists of two major parts: the testing client that executes all test cases in a vendor lab (self-testing) or a third party lab, and the server system that is under the OPNFV's administration to store and view test results based on OPNFV Test API. The following diagram illustrates this overall framework.

/* here is a draft diagram that needs to be revised when exact information is known and fixed */

Image Added

This section mainly focuses on helping the testers in the vendor's domain attempting to run the CVP tests using Dovetail.

Dovetail client tool (or just Dovetail tool or Dovetail for short) can be installed in the jumphost either directly as Python software, or as a Docker(r) container. Comments of pros and cons of the two options TBD.

In 2.2, we describe steps the tester needs to take to install Dovetail directly from the source. In 2.3, we describe steps needed for installing Dovetail Docker(r) container. Once installed, and properly configured, the remaining test process is mostly identical for the two options. In 2.4, we go over the steps of actually running the test suite. In 2.5, we discuss how to view test results and make sense of them, for example, what the tester may do in case of unexpected test failures. Section 2.6 describes additional Dovetail features that are not absolutely necessary in CVP testing but users may find useful for other purposes. One example is to run Dovetail for in-house testing as preparation before official CVP testing; another example is to run Dovetail experimental test suites other than the CVP test suite. Experimental tests may be made available by the community for experimenting less mature test cases or functionalities for the purpose of getting feedbacks for improvement.

2.2 Installing Dovetail

Before taking this step, testers should check the hardware and networking requirements of the POD, and the jumphost in particular, to make sure they are compliant.

In this section, we describe the procedure to install Dovetail client tool that runs the CVP test suite from the jumphost. The jumphost must have network access to both the public Internet and to the O&M (Operation and Management) network with access rights to all VIM APIs being tested.

2.2.1 Checking the Jumphost Readiness

While Dovetail does not have hard requirement on a specific operating system type or version, these have been validated by the community through some level of exercise in OPNFV labs or PlugFests.

• Ubuntu 16.04.2 LTS (Xenial) for x86_64
• Ubuntu 14.04 LTS (Trusty) for x86_64
• CentOS-7-1611 for x86_64
• Red Hat Enterprise Linux 7.3 for x86_64
• Fedora 24 Server for x86_64 
• Fedora 25 Server for x86_64
  

If anyone managed to run the dovetail on additional operation system, this list can be updated after a double check. 

Non-Linux operating systems, such as Windows, Mac OS, have not been tested at this time.

The tester should also validate that the jumphost can reach the public Internet. For example,

%ping 8.8.8.8
%dig https://www.opnfv.org/cvp

2.2.2 Configuring the Jumphost Environment

/* First, openstack env variables to be passed to Functest */

The jumphost needs to have the right environmental variable setting to enable access to the Openstack API. This is usually done through the Openstack credential file. If you do not know how to fill the variables, just go to check OpenRC files on Openstack controller nodes, normally they are located in /opt.

Sample Openstack credential file environment_config.sh:

          /*Project-level authentication scope (name or ID), recommend admin project.*/

          export OS_USERNAME=

          /* Authentication password. Use your own password*/

          export OS_PASSWORD=

          /* Authentication URL, one of the endpoints of keystone service. If this is v3 version, there need some extra variables as follows.*/

          export OS_AUTH_URL='http://xxx.xxx.xxx.xxx:5000/v3'

          /* Default is 2.0. If use keystone v3 API, this should be set as 3.*/

          export OS_IDENTITY_API_VERSION=3

          /* Domain name or ID containing the user above.  Command to check the domain: openstack user show <OS_USERNAME>*/

          export OS_USER_DOMAIN_NAME=default

          /* Domain name or ID containing the project above.  Command to check the domain: openstack project show <OS_PROJECT_NAME>*/

          export OS_PROJECT_DOMAIN_NAME=default

2.2.3 Installing Prerequisite on Jumphost

 1. Dovetail requires Python 2.7 and later

     Use the following steps to check if the right version of python is already installed, and if not, install it.

    % python --version

 2. Dovetail requires Docker 1.12.3 and later 

     Use the following steps to check if the right version of Docker is already installed, and if not, install it.

    % docker version

    As the docker installation process is much complex, you can refer to the scripts.

    It will install the latest version of docker, if you’re not intend to update your docker version, be careful to run the script:

    % wget -qO- https://get.docker.com/ | sh

2.2.4 Installing Dovetail on Jumphost

A tester can choose one of the following two methods for installing and running Dovetail. In Method1, we explain the steps to install Dovetail from the source. In Method2, an alternative using a Docker image with preinstalled Dovetail is introduced.

Method 1. Installing Dovetail directly

1. Update and install packages
   

   a) Ubuntu 

            sudo apt-get update

            sudo apt-get -y install gcc  git  vim  python-dev  python-pip  --no-install-recommends

    b) centos and redhat

            sudo yum -y update

            sudo yum -y install epel-release

            sudo yum -y install gcc  git  vim-enhanced  python-devel  python-pip

    c) fedora

            sudo dnf -y update

            sudo dnf -y install gcc  git  vim-enhanced  python-devel  python-pip  redhat-rpm-config

   p.s  When testing SUT's https service, there need some extra packages, such as apt-transport-https. This still remains to be verified. 

   
   

2. Installing Dovetail

          Now we are ready to install Dovetail.

          /* Version of dovetail is not specified yet? we are still using the latest in the master - this needs to be fixed before launch. */

          First change directory to $DOVETAIL_HOME,

          % cd $DOVETAIL_HOME

          % sudo git clone https://git.opnfv.org/dovetail

Image Added

          % cd $DOVETAIL_HOME/dovetail

          % sudo pip install -e ./

          /* test dovetail install is successful */

          % dovetail -h

Method 2. Installing Dovetail Docker Container 

       The Dovetail project also maintains a Docker image that has Dovetail test tools preinstalled. 

% sudo docker pull opnfv/dovetail:<tag>

The only <tag> is 'latest'.

% sudo docker run --privileged=true -it -v <openrc_path>:<openrc_path> \

-v $DOVETAIL_HOME/results:$DOVETAIL_HOME/results \

-v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \

-v /home/opnfv/dovetail/userconfig:/home/opnfv/dovetail/userconfig \

-v /var/run/docker.sock:/var/run/docker.sock \

--name <DoveTail_Container_Name> (optional) \

opnfv/dovetail:<Tag> /bin/bash

2.3 Running CVP Test Suite

2.3.1 Running Test Suite

The Dovetail client CLI allows the tester to specify which test suite to run. By default the results are stored in a local file $DOVETAIL_HOME/dovetail/results.

% dovetail run --testsuite <test suite name> --openrc <path-to-openrc-file> /*?? */

Multiple test suites may be available, testsuites named "debug" and "proposed_tests" are just provided for testing. But for the purpose of running CVP test suite, the test suite name follows the following format,

CVP.<major>.<minor>.<patch>  /* test if this format works */

For example, CVP_1_0_0

% dovetail run --testsuite CVP_1_0_0

When the SUT's VIM (Virtual Infrastructure Manager) is Openstack, its configuration is commonly defined in the openrc file. In that case, you can specify the openrc file in the command line,

% dovetail run --testsuite CVP_1_0_0 --openrc <path-to-openrc-file>

In order to report official results to OPNFV, run the CVP test suite and report to OPNFV official URL,

% dovetail run --testsuite <test suite name> --openrc <path-to-openrc-file> --report https://www.opnfv.org/cvp

The official server https://www.opnfv.org/cvp is still under development, there is a temporal server to use http://205.177.226.237:9997/api/v1/results

2.3.2 Special Configuration for Running HA test cases

HA test cases need to know the info of all nodes of the OpenStack. It should include every node's name, role, ip, user and key_filename or password. The info should be written in file $DOVETAIL_HOME/dovetail/userconfig/pod.yaml.  There is a sample file $DOVETAIL_HOME/dovetail/userconfig/sample_pod.yaml.

Two methods to login nodes

      Method1. Use private key to login nodes. For example the info of node1,

                name: node1

                role: Controller

                ip: 10.1.0.50

                user: root

                key_filename: /root/.ssh/id_rsa

      Method2. Use password to login nodes. For example the info of node1,

                name: node1

                role: Controller

                ip: 10.1.0.50

                user: root

                password: root

NOTE:

1. The name of each node must be node1, node2,...
2. node1 must be Controller node.
3. If use private key to login, the key_filename must be /root/.ssh/id_rsa in file $DOVETAIL_HOME/dovetail/userconfig/pod.yaml.
4. If use private key to login, you must copy the private key to $DOVETAIL_HOME/dovetail/userconfig/
         cp <private_key_for_login_nodes> $DOVETAIL_HOME/dovetail/userconfig/id_rsa

2.3.3 Making Sense of CVP Test Results

When a tester is performing trial runs, Dovetail stores results in a local file by default.

% cd $DOVETAIL_HOME/dovetail/results

1.  local file

a)  Log file: dovetail.log

      /* review the dovetail.log to see if all important information has been captured - in default mode without DEBUG */

     /* the end of the log file has a summary of all test case test results */

     Additional log files may be of interests: refstack.log, opnfv_yardstick_tcXXX.out ...

b)  Example: Openstack refstack test case example

     can see the log details in refstack.log, which has the  passed/skipped/failed test cases result, the failed test cases have rich debug information

     for the users to see why this test case fails. 

c) Example: OPNFV Yardstick test case example

    for yardstick tool, its log is stored in yardstick.log

    for each test case result in Yardstick, the logs are stored in opnfv_yardstick_tcXXX.out, respectively.

2. OPNFV web interface

    wait for the complement of LF, test community, etc.

2.3.4 Updating Dovetail or Test Suite

% cd $DOVETAIL_HOME/dovetail

% sudo git pull

% sudo pip install -e ./

This step is necessary if dovetail software or the CVP test suite have updates.

2.4 Other Dovetail Usage

2.4.1 Running Dovetail Locally 

DoveTail supports uploading results into database. The database can be either a local database or an official one. Before you can use the local database, you need to create the local database and the testapi service. They can be installed on any machine which can talk to the jumphost. Docker 1.12.3 and later should be installed on this machine. There are 3 steps for creating local database and testapi service.

      step1. Set ports for database and testapi service           

            The default ports of database and testapi are 27017 and 8000, respectively. Check whether they are used by other services already.

                  % netstat -nlt

            If 27017 and 8000 are used by other services, you need to set other ports in step2.

      step2. Run a Dovetail container

            Run a Dovetail container.

                  % sudo docker pull opnfv/dovetail:<Tag>

                  % sudo docker run -itd --privileged=true --name <DoveTail_Container_Name> \

                       -v /var/run/docker.sock:/var/run/docker.sock  opnfv/dovetail:<Tag> /bin/bash

                  % sudo docker exec -it <DoveTail_Container_Name> /bin/bash

            If you need to set ports for database and testapi service,

                  % export mongodb_port=<database_port>

                  % export testapi_port=<testapi_port>

      step3. Create local database and testapi service

            % cd /home/opnfv/dovetail/dovetail/utils/local_db/

            % ./launch_db.sh <localhost_ip_address>

            Exit this Dovetail container.

                  % exit

      step4. Check the status of database and testapi service

            Local database and testapi service are actually two containers named mongodb and testapi. You can check whether there are these two containers running.

                  % sudo docker ps -a

            You can try to get data from the database to make sure everything is OK.

                  % wget <localhost_ip_address>:<testapi_port>/api/v1/results

2.4.2 Running Dovetail with an Offline SUT

2.4.3

          Now we are ready to install Dovetail.

          /* Version of dovetail is not specified yet? we are still using the latest in the master - this needs to be fixed before launch. */

          First change directory to $DOVETAIL_HOME,

          % cd $DOVETAIL_HOME

          % sudo git clone https://git.opnfv.org/dovetail

Image Removed

          % cd $DOVETAIL_HOME/dovetail

          % sudo pip install -e ./

          /* test dovetail install is successful */

          % dovetail -h

part2. Installing Dovetail Docker Container 

       The Dovetail project also maintains a Docker image that has Dovetail test tools preinstalled. 

2.3 Running CVP Test Suite

2.3.1 Running Test Suite

The Dovetail client CLI allows the tester to specify which test suite to run. By default the results are stored in a local file $DOVETAIL_HOME/results.

% dovetail run --testsuite <test suite name> --openrc <path-to-openrc-file> /*?? */

Multiple test suites may be available, but for the purpose of running CVP test suite, the test suite name follows the following format,

CVP.<major>.<minor>.<patch>  /* test if this format works */

For example, CVP_1_0_0

% dovetail run --testsuite CVP_1_0_0

When the SUT's VIM (Virtual Infrastructure Manager) is Openstack, its configuration is commonly defined in the openrc file. In that case, you can specify the openrc file in the command line,

% dovetail run --testsuite CVP_1_0_0 --openrc <path-to-openrc-file>

In order to report official results to OPNFV, run the CVP test suite and report to OPNFV official URL,

% dovetail run --testsuite <test suite name> --openrc <path-to-openrc-file> --report https://www.opnfv.org/cvp

2.3.2 Making Sense of CVP Test Results

When a tester is performing trial runs, Dovetail stores results in a local file by default.

% cd $DOVETAIL_HOME/dovetail/results

/* local file */

Log file: dovetail.log

/* review the dovetail.log to see if all important information has been captured - in default mode without DEBUG */

/* the end of the log file has a summary of all test case test results */

Additional log files may be of interests: refstack.log, opnfv_yardstick_tcXXX.out ...

Example: Openstack refstack test case example

can see the log details in refstack.log, which has the  passed/skipped/failed test cases result, the failed test cases have rich debug information

for the users to see why this test case fails. 

Example: OPNFV Yardstick test case example

for yardstick tool, its log is stored in yardstick.log

for each test case result in Yardstick, the logs are stored in opnfv_yardstick_tcXXX.out, respectively.

/* OPNFV web interface */

wait for the complement of LF, test community, etc.

2.3.3 Updating Dovetail or Test Suite

% cd $DOVETAIL_HOME/dovetail

% sudo git pull

% sudo pip install -e ./

This step is necessary if dovetail software or the CVP test suite have updates.

2.4 Other Dovetail Usage

2.4.1 Running Dovetail Locally 

/*DB*/

...

Running Dovetail with Experimental Test Cases

2.4.

...

4 Running Individual Test Cases or for Special Cases

Refstack 1. Refstack client to run Defcore testcases

    a) By default, for Defcore test cases run by Refstack-client, which are consumed by DoveTail,  are run followed with automatically generated configuration file, i.e., refstack_tempest.conf.

    In some circumstances, the automatic configuration file may not quite satisfied with the SUT, DoveTail provide a way for users to set its configuration file according to its own SUT manually,

    besides, the users should define Defcore testcase file, i.e., defcore.txt, at the same time. The steps are shown as,

    when "Installing Dovetail Docker Container" method is used,

%              % sudo mkdir /home/opnfv/dovetail/userconfig

%              % cd /home/opnfv/dovetail/userconfig

            % touch refstack_tempest.conf defcore.txt

            % vim refstack_tempest.conf

            % vim defcore.txt

    the recommend way to set refstack_tempest.conf is shown in https://aptira.com/testing-openstack-tempest-part-1/

    the recommended way to edit defcore.txt is to open https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false and copy all the test cases into defcore.txt.

    Then use “docker run” to create a container,,

          % sudo docker run --privileged=true -it -v <openrc_path>:<openrc_path> \

           -v /home/opnfv/dovetail/results:/home/opnfv/dovetail/results \

           -v /home/opnfv/dovetail/userconfig:/home/opnfv/dovetail/userconfig \

           -v /var/run/docker.sock:/var/run/docker.sock \

          --name <DoveTail_Container_Name> (optional) \

          opnfv/dovetail:<Tag> /bin/bash

3 Dovetail Client CLI Manual

This section contains a brief manual for all the features available through the Dovetail client command line interface (CLI).

...

Dovetail has three commands: list, run and show.

...

3.2 List

...

3.2.1 List help

% dovetail list -h

...

3.2.2 List a test suite

List command will list all test cases belong to the given test suite.

...

The ipv6, example and nfvi are test areas. If no <TESTSUITE> is given, it will list all testsuites.

...

3.3 Show

Show command will give the detailed info of one certain test case.

...

3.3.1 Show help

% dovetail show -h

...

3.3.2 Show test case

...

3.4 Run

Dovetail supports running a named test suite, or one named test area of a test suite.

...

3.4.1 Run help

% dovetail run -h

There are some options:

• func_tag: set FuncTest’s Docker tag, for example stable,latest and danube.1.0

• openrc: give the path of OpenStack credential file

• yard_tag: set Yardstick’s Docker tag

• testarea: set a certain testarea within a certain testsuite

• offline: run without pull the docker images, and it requires the jumphost to have these images locally. This will ensure DoveTail run in an offline environment.

• report: push results to DB or store with files

• testsuite: set the testsuite to be tested

• debug: flag to show the debug log messages

...