This page is a temporary location to draft and validate the testing guide documentation. Once it's in a decent draft shape, I will submit it to documentation gerrit.
Reference: CVP and Dovetail Terminology
I will use the terms defined in the above WIKI. Those terms will also be copied to the documentation's term section when they are in decent shape.
The intended audience of this document is the technical staff in vendors who are interested in seeking CVP certifications using Dovetail testing or third party labs who are to assist their clients for the same purpose. Additional audience include anyone who is interested in using Dovetail for testing purposes other than CVP certification. We assume the readers have good technical background in IT (e.g. Linux, Docker containers, security), networking, NFV, in addition to test areas that CVP covers.
1 Creating a Compliant POD
Dovetail uses Pharos Specification (Pharos Specification) as a starting point of defining a compliant physical system as a part of the CVP's System Under Test (SUT). The current Pharos Specification is still being developed and updated for Danube as this time. It also has goals and constrains that are outside of the CVP's current scope, for example those required for use in the OPNFV community development. For these reasons, we provide a generalized guide here specifically for testers who need to create and configure a POD for the purpose of conducting CVP testing.
/* work in progress */
1.1 High Level Overview
1.2 Hardware
1.3 Networking
1.4 Management
1.5 Configuration Template
2 Conducting CVP Tests using Dovetail
2.1 Dovetail CVP Testing Overview
The Dovetail testing framework consists of two major parts: the testing client that executes all test cases in a vendor lab (self-testing) or a third party lab, and the server system that is under the OPNFV's administration to store and view test results based on OPNFV Test API. The following diagram illustrates this overall framework.
/* here is a draft diagram that needs to be revised when exact information is known and fixed */
This section mainly focuses on helping the testers in the vendor's domain attempting to run the CVP tests using Dovetail.
Dovetail client tool (or just Dovetail tool or Dovetail for short) can be installed in the jumphost either directly as Python software, or as a Docker(r) container. Comments of pros and cons of the two options TBD.
In 2.2, we describe steps the tester needs to take to install Dovetail directly from the source. In 2.3, we describe steps needed for installing Dovetail Docker(r) container. Once installed, and properly configured, the remaining test process is mostly identical for the two options. In 2.4, we go over the steps of actually running the test suite. In 2.5, we discuss how to view test results and make sense of them, for example, what the tester may do in case of unexpected test failures. Section 2.6 describes additional Dovetail features that are not absolutely necessary in CVP testing but users may find useful for other purposes. One example is to run Dovetail for in-house testing as preparation before official CVP testing; another example is to run Dovetail experimental test suites other than the CVP test suite. Experimental tests may be made available by the community for experimenting less mature test cases or functionalities for the purpose of getting feedbacks for improvement.
2.2 Installing Dovetail Directly
Before taking this step, testers should check the hardware and networking requirements of the POD, and the jumphost in particular, to make sure they are compliant.
In this section, we describe the procedure to install Dovetail client tool that runs the CVP test suite from the jumphost. The jumphost must have network access to both the public Internet and to the O&M (Operation and Management) network with access rights to all VIM APIs being tested.
2.2.1 Checking the Jumphost Readiness
While Dovetail does not have hard requirement on a specific operating system type or version, these have been validated by the community through some level of exercise in OPNFV labs or PlugFests.
- Xenial 16.04 (LTS)
- Trusty 14.04 (LTS)
- 64-bit version of CentOS 7
- 64-bit version of RHEL 7
- 64-bit version of Fedora 24 or 25
Additional examples of operating systems known to work well can be found in OPNFV Dovetail wiki pages. /* Suggest we maintain such a wiki for support purpose - Wenjing */
Non-Linux operating systems, such as Windows, Mac OS, have not been tested at this time.
The tester should also validate that the jumphost can reach the public Internet. For example,
% ping 8.8.8.8
% ping www.google.com
And also validate that the jumphost can reach the Openstack API, for example,
/* Need refinement on how to test this */
% ping x.x.x.x
2.2.2 Configuring the Jumphost Environment
The jumphost needs to have the right environmental variable setting to enable access to the Openstack API. This is usually done through the Openstack credential file (See https://docs.openstack.org/developer/python-openstackclient/man/openstack.html#environment-variables).
For example, /* need to fix this picture here - the example settings are not good */
OS_USERNAME: Authentication username, this can just be admin user.
OS_PASSWORD: Authentication password
OS_AUTH_URL: Authentication URL
OS_USER_DOMAIN_NAME: Domain name or ID containing user
OS_PROJECT_NAME: Project-level authentication scope (name or ID), this can just be admin project.
OS_IDENTITY_API_VERSION: Identity API version (Default: 2.0)
OS_IMAGE_API_VERSION: Image API version (Default: 2.0)
OS_PROJECT_DOMAIN_NAME: Domain name or ID containing project
2.2.3 Installing Prerequisite
Dovetail requires Python 2.7 or later and git /*version??*/, anything else?
Use the following steps to check if the right version of python is already installed, and if not, install it.
% python -V
/* Need contribution to complete this section */
/* reference web links to install python, git, ... */
2.2.4 Installing Dovetail
Now we are ready to install Dovetail.
/* Version of dovetail is not specified yet? we are still using the latest in the master - this needs to be fixed before launch. */
First, create a home directory for Dovetail,
% cd
% sudo mkdir dovetail
% export DOVETAIL_HOME=$HOME/dovetail
% cd $DOVETAIL_HOME
% sudo git clone https://git.opnfv.org/dovetail
/* steps missing in installing dovetail after the git clone */
to be completed
2.3 Installing Dovetail Docker Container
TBD. Is this useful or necessary? It seems the overall complexity is about the same.
2.4 Running CVP Test Suite
The Dovetail client CLI allows the tester to specify which test suite to run.
% dovetail run --testsuite <test suite name>
Multiple test suites may be available, but for the purpose of running CVP test suite, the test suite name follows the following format,
CVP.<major>.<minor>.<patch>
For example, CVP.1.0.0.
% dovetail run --testsuite CVP.1.0.0
When the SUT's VIM (Virtual Infrastructure Manager) is Openstack, its configuration is commonly defined in the openrc file. In that case, you can specify the openrc file in the command line,
% dovetail run --testsuite CVP.1.0.0 --openrc <path-to-openrc-file>
2.5 Making Sense of CVP Test Results
2.6 Other Dovetail Usage
2.6.1 Running Dovetail Locally
2.6.2 Running Dovetail with Experimental Test Cases
Dovetail Client CLI Manual
This section contains a brief manual for all the features available through the Dovetail client command line interface (CLI).
6.1 Check dovetail commands
% dovetail -h
Dovetail has three commands: list, run and show.
6.2 List
6.2.1 List help
% dovetail list -h
6.2.2 List a test suite
List command will list all test cases belong to the given test suite.
% dovetail list compliance_set
% dovetail list debug
The ipv6, example and nfvi are test areas. If no <TESTSUITE> is given, it will list all testsuites.
6.3 Show
Show command will give the detailed info of one certain test case.
6.3.1 Show help
% dovetail show -h
6.3.2 Show test case
6.4 Run
Dovetail supports running a named test suite, or one named test area of a test suite.
6.4.1 Run help
% dovetail run -h
There are some options:
func_tag: set FuncTest’s Docker tag, for example stable,latest and danube.1.0
openrc: give the path of OpenStack credential file
yard_tag: set Yardstick’s Docker tag
testarea: set a certain testarea within a certain testsuite
offline: run without pull the docker images, and it requires the jumphost to have these images locally. This will ensure DoveTail run in an offline environment.
report: push results to DB or store with files
testsuite: set the testsuite to be tested
debug: flag to show the debug log messages