The NFVbench project develops a toolkit that allows developers, integrators, testers and customers to measure and assess the L2/L3 forwarding performance of an NFV-infrastructure solution stack (i.e. OPNFV scenario) using a black-box approach.
NFVbench is available as a self-contained docker container that includes the the main application and the Trex software traffic generator.
NFVbench is agnostic of the NFV platform, the controller used (ML2/OVS, ODL...), the network stack used (OVS, OVS-DPDK, VPP, SR-IOV...) and supports VLAN and VxLAN.
NFVbench is currently widely deployed in production NFV clouds around the world by multiple service providers and is mostly used for
- on-site integrated performance benchmarking tool, which allows instant performance measurement on every deployed NFV platform
- precise performance planning tool with high VNF density (multi-chaining)
NFVbench is also currently used as the traffic generator solution for the CNCF CNF testbed based on packet.net: CNF testbed slides.
Documentation
Source code
Mailing list
There is no mailing list dedicated to NFVbench.
For inquiries and questions: send an email to anuket-tech-discuss@lists.anuket.io with a Subject line starting with
"#nfvbench"
The list archives can be consulted and searched here:
https://lists.anuket.io/g/anuket-tech-discuss/search?q=%23nfvbench&ct=1
TRex project
NFVbench Docker image
Where can I find NFVbench Docker images?
Available versions of NFVbench Docker images can be found on Docker hub:
https://hub.docker.com/r/opnfv/nfvbench/tags
Where is the change log for NFVbench Docker image?
No ChangeLog, look at NFVbench commit log on nfvbench.org.
Note: there is also a GitHub mirror for NFVbench, but it is more complicated to see the association between commit history and tags
Which version of TRex is associated to a given version of NFVbench?
For a given version of NFVbench container image on Docker hub, TRex version can be found in the TREX_VER environment variable in the image history. For
instance, NFVbench Docker image version 3.4.1 is based on TRex 2.56 as can be seen here.
For a given version of NFVbench code base, the version of TRex that would be embedded in a Docker container built from that code base is given in the
TREX_VER environment variable found in docker/Dockerfile For instance, NFVbench code base tagged 3.6.1 is meant to be used with TRex 2.61 as can be seen in the Dockerfile.
NFVbench loop VM image
The list of the artifacts generated by NFVbench project, including all versions of NFVbench loop VM, can be found here:
https://artifacts.opnfv.org/nfvbench.html. Look for nfvbench/images.
CPU cores and huge pages
How to configure the CPU logical cores to be used by TRex?
Reminder: a CPU logical core is either a SMT thread (ie when intel hyper-threading is enabled) or a CPU physical core (when SMT is disabled).
TRex needs:
1. one logical core for general purpose
2. one logical core for latency measurement
3. several logical cores for test traffic generation
Let's detail that on the following excerpt of a NFVbench config file:
traffic_generator:
...
generator_profile:
- name: trex-local
...
cores: 6
platform:
master_thread_id: '0'
latency_thread_id: '2'
dual_if:
- socket: 0
threads: [4,6,8,10,12,14]
...
The general purpose logical core is configured with master_thread_id here CPU logical core 0. TRex workload on that core is not CPU intensive so typically that core will be shared with the operating system.
The logical core for latency measurement is configured with latency_thread_id, here CPU logical core 2.
Traffic generation uses a number of logical cores specified in the cores config parameter, here 6. The list of logical cores to be used is given by the threads config parameter, here 4,6,8,10,12,14.
Note: the cores config parameter does not include the system
thread and the latency thread.
TRex will run a CPU intensive DPDK PMD thread on each logical core identified for latency measurement or traffic generation. So all those cores have to be
dedicated to a single instance of TRex. Besides, when SMT is enabled, only one logical core per physical core should be used.
The number of logical cores needed for traffic generation depends on the server hardware and the desired throughput. For instance, 6 cores are needed to
generate 2x 10Gbps with 64-byte packets on a server equipped with Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz processors and Intel(R) 10GbE 2P X710 Adapters.
On different hardware, that number of cores should be assessed with a loopback test.
How can I see the logical CPU cores actually used by TRex?
During a test, some logical CPU cores are running user processes with DPDK PMD threads at 100% CPU time.
An easy visual way to see it during a test: start the top command then 1 t (1 to show individual CPUs, t to display a bar graph of CPU usage).
Example: logical CPU core 2 is running the latency threads, and logical CPU cores 4, 6, 8, 10, 12 and 14 are running the test traffic threads.
top - 14:29:07 up 1 day, 23:10, 2 users, load average: 3.48, 1.74, 1.09
Tasks: 598 total, 1 running, 597 sleeping, 0 stopped, 0 zombie
%Cpu0 : 1.3/0.0 1[| ]
%Cpu1 : 0.0/0.0 0[ ]
%Cpu2 : 100.0/0.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu3 : 0.0/0.0 0[ ]
%Cpu4 : 100.0/0.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu5 : 0.0/0.0 0[ ]
%Cpu6 : 100.0/0.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu7 : 0.0/0.0 0[ ]
%Cpu8 : 100.0/0.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu9 : 0.0/0.0 0[ ]
%Cpu10 : 100.0/0.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu11 : 0.0/0.0 0[ ]
%Cpu12 : 100.0/0.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu13 : 0.0/0.0 0[ ]
%Cpu14 : 100.0/0.0 100[|||||||||||||||||||||||||||||||||||||||||||||||||||||]
%Cpu15 : 0.0/0.0 0[ ]
%Cpu16 : 0.0/0.0 0[ ]
%Cpu17 : 0.0/0.0 0[ ]
%Cpu18 : 0.0/0.0 0[ ]
%Cpu19 : 0.0/0.0 0[
<skip>
How many huge pages do I need for NFVbench (TRex)?
TRex reserves some memory as huge pages at startup. The default value is 1GB and is enough in our experience.
In NFVbench config file, the parameter to configure the amount of memory to reserve is traffic_generator.generator_profile.limit_memory. Description from the config file:
Specify the memory reserved for running the TRex traffic generator (in MB).
Limit the amount of packet memory used. (Passed to dpdk as -m arg)
In the default config file for NFVbench 3.4.1, we have:
traffic_generator:
generator_profile:
- name: trex-local
<skip>
limit_memory: 1024
We use 1GB huge pages and we configure Linux kernel to allocate them at boot time with the parameters default_hugepagesz=1G hugepagesz=1G hugepages=<nb-of-pages> on the kernel CLI. <nb-of-pages> has to be a
multiple of the number of NUMA nodes, because the same number of pages is allocated on each NUMA node. For instance, if the traffic generator server has two NUMA nodes, the minimum value for one instance of TRex would be
<nb-of-pages>=2.
Note: at startup, TRex takes all the available huge pages, then he releases the pages he does not need.
Test traffic
What is the TRex frame size?
When we specify a frame size to NFVbench eg with the -fs command line option, TRex generates *802.3 Ethernet frames* of that size. The size include the Ethernet header and the CRC checksum. The CRC checksum is 4-byte long. The Ethernet header is either 14-byte long (when NFVbench is configured without
VLAN ID) or 18-byte long (4 additional bytes for the 802.1Q tag when NFVbench is configured with VLAN IDs).