Testing and verifying DPDK is NOT easy. And it is even more challenging in VM environments.
After
investing in VMWare hypervisors that supposedly run DPDK, we wanted to
test and verify that a) it worked, and b) the performance was as
advertised.
Below is a list of steps we took to get the host and a DPDK-enabled VM ready:
- Hypervisor(s)
- Enabled
the ixgben_ens drivers on the host. There are some ESXI CLI commands
you can run to ensure that these are loaded and working.
- VM Settings
- VMXNet3 adaptor types on VM
- Latency Sensitivity = High (sched.cpu.latencySensitivity=TRUE)
- Hugepages enabled in VM (sched.mem.lpage.enable1GPage TRUE
- Reserve all Guest Memory
- Enable Multiple Cores for High I/O Workloads (ethernetX.ctxPerDev=”1”
)
- CPU Reservation
- NUMA Affinity (numaNodeAffinity=X)
After this, I launched the VM. I was smart enough to launch the VM with 3 NICs on it.
- eth0 - used as a management port, for ssh and such.
- eth1 - this one to be used for DPDK testing
- eth2 - this one to be used for DPDK testing
Launching a VM (i.e. a RHEL Linux VM) with these settings, does NOT mean that you are ready for DPDK!! You still need a DPDK-compiled application on your OS. DPDK applications need to use DPDK-enabled NIC drivers on the VM, and on a Linux VM, these drivers are typically run as kernel modules. There are several different types and kinds of DPDK drivers (kernel modules), such as vfio, uio-pci-generic, igb_uio, et al.
To prepare your VM for testing, we decided to install DPDK, and then run the TestPMD application.
Installing DPDK
To get DPDK, you can go to dpdk.org, and download the drivers, which comes as a tar.gz file that can be unpacked. Or, there is a github site that you can use the clone the directories and files.
# git clone http://dpdk.org/git/dpdk
It is important to read the instructions when building DPDK, because the old-style "make configure", "make", "make install" process has been replaced by fancier build tools like meson and ninja that you need to install. I chose to install by going to the top of the directory tree, and typing:
# meson -Dexamples=all build
This does not actually compile the code. It sets the table for you to use Ninja to build the code. So the next step was to type:
# ninja
Followed by:
# ninja install
The "ninja install" puts a resultant set of DPDK executables (some ELF, some Python), in /usr/local/bin directory (maybe installs some stuff in other places too).
Right away, I hit a snag. When I tried to run dpdk_setup.py, to bind the VM The kernel module igb_uio.ko was nowhere to be found.
I was completely at a loss about this, until I realized that some other DPDK packages (test load generators) compile DPDK and the igb_uio.ko drivers, either by including them outright, or copying the sources into the build process. Trex, for example, builds the drivers. And so does a package called DTS. After I decided to git clone the DTS package, I stumbled upon some documentation in an archived package called DTS (DPDK Testing Suite). In the DTS package, in the /opt/github/dts/doc/dts_gsg/usr_guide there is a file called igb_uio.rst which describes how to compile the igb_uio.ko drivers for use with DTS. This was the missing link. The section of the file up front, described that the drivers have been moved into a different github repository - and are now separated from DPDK!
Get Source Code - note: assumption is that you are doing this in /opt directory.
---------------
Get igb_uio::
git clone http://dpdk.org/git/dpdk-kmods
git clone git://dpdk.org/dpdk-kmods
Get DPDK::
git clone git://dpdk.org/dpdk
git clone http://dpdk.org/git/dpdk
The author of this igb_uio.rst file described the process that can be used to fuse DPDK and the drivers back together into a single build - the way it used to be. How convenient. Here is how that is done.
Integrate igb_uio into DPDK
---------------------------
Assume you have cloned the dpdk and dpdk-kmods source code
in opt/dpdk and opt/dpdk-kmods.
Step 1
# Copy dpdk-kmods/linux/igb_uio/ to dpdk/kernel/linux/:
[root@dts linux]# cp -r /opt/dpdk-kmods/linux/igb_uio /opt/dpdk/kernel/linux/
you should see igb_uio in your output:
[root@dts linux]# ls /opt/dpdk/kernel/linux/
igb_uio kni meson.build
Step 2:
# enable igb_uio build in meson:
since we have copied the directory over to /opt/dpdk, we will edit the meson.build there.
* add igb_uio in /opt/dpdk/kernel/linux/meson.build subdirs as below:
subdirs = ['kni', 'igb_uio']
NOTE: this is an important step not to miss because it will not build if you don't do this.
Step 3:
* create a file of meson.build in /opt/dpdk/kernel/linux/igb_uio/ as below:
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2017 Intel Corporation
mkfile = custom_target('igb_uio_makefile',
output: 'Makefile',
command: ['touch', '@OUTPUT@'])
custom_target('igb_uio',
input: ['igb_uio.c', 'Kbuild'],
output: 'igb_uio.ko',
command: ['make', '-C', kernel_dir + '/build',
'M=' + meson.current_build_dir(),
'src=' + meson.current_source_dir(),
'EXTRA_CFLAGS=-I' + meson.current_source_dir() +
'/../../../lib/librte_eal/include',
'modules'],
depends: mkfile,
install: true,
install_dir: kernel_dir + '/extra/dpdk',
build_by_default: get_option('enable_kmods'))
How wonderful. To recap, here is what we did:
- copy the source files from dpdk-kmods into the proper directory of dpdk
- snap in the proper meson build file (which the author graciously provides)
- uninstall (previous build, assuming you built DPDK before doing all of this)
- rebuild
- reinstall
Step 3:
# cd /opt/dpdk/build
# ninja uninstall
Step 4:
# ninja
Step 5:
# ninja install
A quick find command shows that the kernel module was built.
[root@acdcchndnfvdpk0001 dpdk]# find . -print | grep ko
./build/drivers/net/octeontx/base/libocteontx_base.a.p/octeontx_pkovf.c.o
./build/lib/librte_table.a.p/table_rte_table_hash_cuckoo.c.o
./build/lib/librte_hash.a.p/hash_rte_cuckoo_hash.c.o
./kernel/linux/igb_uio/igb_uio.ko
./kernel/linux/igb_uio/.igb_uio.ko.cmd
./drivers/net/octeontx/base/octeontx_pkovf.c
./drivers/net/octeontx/base/octeontx_pkovf.h
./lib/hash/rte_cuckoo_hash.h
./lib/hash/rte_cuckoo_hash.c
./lib/table/rte_table_hash_cuckoo.h
./lib/table/rte_table_hash_cuckoo.c
Now, we have something we can use to bind our adaptors to the drivers!!!
You can bind the adaptors to the drivers using a couple of different methods. You can use a utility that is supplied by DPDK to do it (dpdk-devbind.py), or you can also use a nifty Linux utility called driverctl, which I prefer (this needs to be installed typically with a package manager as it generally does not roll onto the OS with a default installation).
A script I use to do the binding looks like this:
# cat bind-pci.sh
#!/bin/bash
lshw -class network -businfo | grep pci
while :
do
echo "Linux Interface to override (e.g. p1p1, p1p2, p1p3, p1p4):"
read iface
if [ ${iface} == "skip" ]; then
break
fi
lshw -class network -businfo | grep pci | grep ${iface}
if [ $? -eq 0 ]; then
pci=`lshw -class network -businfo | grep pci | grep ${iface} | awk '{printf $1}' | cut -f2 -d"@"`
echo "We will override the kernel driver with igb_uio for PCI address: ${pci}"
driverctl set-override ${pci} igb_uio
break
fi
done
When you run this script, you can check to see if the binding was successful by running a DPDK command:
# python3 /usr/local/bin/dpdk-devbind.py --status
And this command will show you whether the binding worked or not.
Network devices using DPDK-compatible driver
============================================
0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3
0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3
Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio *Active*
NOTE: To remove a driver binding, "driverctl unset-override ${pci address}" would be used. In which case, the driver will now become visible to the Linux OS in the Virtual Machine again.
So we now have one adaptor that the Linux Networking Kernel sees (eth0), but the two adaptors Linux saw prior to the binding (eth1, and eth2), have now been "reassigned" to DPDK. And the OS does not see them at all, actually, anymore.
If we run an ifconfig, or an "ip a" command to see the Linux network interfaces in the VM, this is what it now looks like.
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:b7:83:1a brd ff:ff:ff:ff:ff:ff
inet 192.168.2.10/24 brd 192.168.2.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::250:56ff:feb7:831a/64 scope link
valid_lft forever preferred_lft forever
NOTE: No eth1 or eth2 shows up in Linux anymore, as they have been handed over to DPDK, which bypasses the Linux Kernel entirely.
Okay, now we have adaptors set up for DPDK. Now what? In our next step, we will do some simple verification with TestPMD.
