...

1. --k8s should be used.
2. The testcase name should be explicitly specified – there should be a 'c' in the testcase. 
3. conf/12_k8s.conf is correctly configured - especially the K8S_CONFIG_FILEPATH, NETWORK_ATTACHMENT_FILEPATH & POD_MANIFEST_FILEPATH.
4. Important files:
   1. core/pod_controller.py - Based on the deployment, this decides how many pods to deploy.
   2. pods/papi.py - Does the actual deployment of Network attachments and pods.
   3. testcases/testcase.py - controls the flow
      1. Variables: _pod_ctl, _pod_list, _evfctl, 
      2. _pod_ctl: Reference to Pod Controller. This is a main handler using which we manage the pods.
      3. _pod_list : list of pods-references.
      4. _evfctl : Use to configure the external vswitch. Currently, only flow-entries and Interface Up/Down is managed 

There was an issue with the way the ViNeperf python scripts launch the ovs-vswitchd process. With the current code it seem we can only reserve ports once or twice before Trafficgen-pod fails to work. The issue is caused because the ViNeperf process stops reading the file descriptor which the ovs-vswitchd process is using for stdout after around ~30 sec.  This causes the ovs-vswitchd stdout buffer to get full and hang any ovs-vswitchd threads which write to stdout.

The reason this happens was because the ovs-vswitchd process is launched using pexpect and a thread is spawned to read the process stdout using the tools.tasks.Process.relinquish() function. The thread which is reading the stdout calls pexpect.pty_spawn.read_nonblocking() with default arguments which by default uses the timeout used for spawn which by default is 30 sec. 

https://github.com/opnfv/vswitchperf/blob/master/tools/tasks.py#L394

https://pexpect.readthedocs.io/en/stable/_modules/pexpect/pty_spawn.html#spawn.read_nonblocking

The modification is made to  the vineperf/tools/task.py file to call to read_nonblocking to pass timeout=None and it fixed the issue. With this, the trafficgen pod can reserve/release the ports multiple times.

[opnfv@worker vswitchperf]$ git diff tools/tasks.py

diff --git a/tools/tasks.py b/tools/tasks.py

index 4e03f85..06408ab 100644

--- a/tools/tasks.py

+++ b/tools/tasks.py

@@ -391,7 +391,7 @@ class Process(object):

         def run(self):

             while True:

                 try:

-                    self.child.read_nonblocking()

+                    self.child.read_nonblocking(timeout=None)

                 except (pexpect.EOF, pexpect.TIMEOUT):

                     break

Scenario-Specific:

1. Vswitch is managed by ViNePerf - North-South
   1. Single-Pod
   2. Multi-Pod*
2. Vswitch is NOT managed by ViNePerf - North-South
   1. Single-Pod
   2. Multi-Pod
3. Using with Non-vSwitch-based-CNI - North-South.
4. Vswitch is manage

...