DPDK Hands-On - Part VII - Creating a Physical DPDK NIC Port on OpenVSwitch

First off, you can NOT add a DPDK port for a physical NIC onto OpenVSwitch, if the NIC is not using a DPDK driver! And that driver must be working!!!

Now, when I got started, I was using vfio as my DPDK driver. Why? Because, per the DPDK and DPDK+OVS websites, vfio is the newer driver and the one everyone is encouraged to use.

In order to use a poll mode driver (on Linux), the appropriate kernel module(s) need(s) to be loaded. Once this kernel module is loaded, you can use a couple of methods (at least) to tell Linux to use the vfio driver for your PCI NIC. The two tools I use, are:

• driverctl - this ships with the OS (CentOS at least)
• dpdk-devbind - to use this utility, dpdk-tools package needs to be installed, or if you have downloaded and compiled DPDK by source code you can use it that way.
  

Using driverctl, you can set your driver by "overriding" the default driver, as follows:

# driverctl set-override 0000:01:00.0 vfio-pci

or, if you are using uio_pci_generic, 

# driverctl set-override 0000:01:00.0 uio_pci_generic

NOTE: I had to use uio_pci_generic with my cards. The vfio driver was registering an error that I finally found in the openvswitch log.

I find the dpdk-devbind tool best for checking status, but you can also use driverctl for that also. I will show both commands and their output. The driverctl is not as clean in its output format as the dpdk-devbind command is.

# driverctl -v list-devices | grep -i net
0000:00:19.0 e1000e (Ethernet Connection I217-LM)
0000:01:00.0 uio_pci_generic [*] (82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (PRO/1000 PT Dual Port Server Adapter))
0000:01:00.1 uio_pci_generic [*] (82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (PRO/1000 PT Dual Port Server Adapter))
0000:03:00.0 e1000e (82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (PRO/1000 PT Dual Port Server Adapter))
0000:03:00.1 e1000e (82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper applications) (PRO/1000 PT Dual Port Server Adapter))

 # dpdk-devbind --status

Network devices using DPDK-compatible driver
============================================
0000:01:00.0 '82571EB Gigabit Ethernet Controller 105e' drv=uio_pci_generic unused=e1000e
0000:01:00.1 '82571EB Gigabit Ethernet Controller 105e' drv=uio_pci_generic unused=e1000e

Network devices using kernel driver
===================================
0000:00:19.0 'Ethernet Connection I217-LM 153a' if=em1 drv=e1000e unused=uio_pci_generic
0000:03:00.0 '82571EB Gigabit Ethernet Controller 105e' if=p1p1 drv=e1000e unused=uio_pci_generic
0000:03:00.1 '82571EB Gigabit Ethernet Controller 105e' if=p1p2 drv=e1000e unused=uio_pci_generic

After you load your driver for the NIC of choice, you need to check a few things to ensure that the driver is working properly. Otherwise, you might pull your hair out trying to debug problems, without realizing that the underlying driver is the culprit.

Next, we will add the port to OpenVSwitch, but before you do, there are two important things to make sure are done:

 

1. make sure the openvswitch you are running has been compiled for DPDK, and initialized.

The best way to do this, is to dump the switch parameters, as shown below. I have highlighted all of the settings you should typically encounter for a properly compiled and configured DPDK-OpenVSwitch.

# ovs-vsctl list open_vswitch

_uuid               : 2d46de50-e5b8-47be-84b4-a7e85ce29526
bridges             : [14049db7-07eb-4b00-89d6-b05201d2978e, 221b954d-c2dc-48b2-923b-86a87765ed7b, db7a8c3c-5a03-4063-a5e8-23686f319473]
cur_cfg             : 936
datapath_types      : [netdev, system]
db_version          : "7.16.1"
dpdk_initialized    : true
dpdk_version        : "DPDK 18.11.8"
external_ids        : {hostname=maschinen, rundir="/var/run/openvswitch", system-id="35b95ef5-fd71-491f-8623-5ccbbc1eca6b"}
iface_types         : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, erspan, geneve, gre, internal, "ip6erspan", "ip6gre", lisp, patch, stt, system, tap, vxlan]
manager_options     : [a4329e63-5b63-4e8b-8dac-0e9bc8492c28]
next_cfg            : 936
other_config        : {dpdk-init="true", dpdk-lcore-mask="0x2", dpdk-socket-limit="2048", dpdk-socket-mem="1024", pmd-cpu-mask="0xC"}
ovs_version         : "2.11.1"
ssl                 : []
statistics          : {}
system_type         : centos
system_version      : "7"

2. make sure that your specific bridge, is created for DPDK (netdev datapath rather than system)

Trying to add a PCI NIC that is using poll mode drivers (vfio or uio) to an OpenVSwitch bridge that does not have the datapath set correctly (to netdev) will result in an error when you add the port. This error will be reported both on an "ovs-vsctl show" command, and also reflected in the log file /var/log/openvswitch/ovs-vswitchd.log file.

This was one of the most common and confusing errors to debug, actually. In newer versions of OVS, they have added the datapath to the "ovs-vsctl show" command. My switch, however, predates this patch, so I have to examine my datapath another way:

# ovs-vsctl list bridge br-tun | grep datapath
datapath_id         : "0000001b21c57204"
datapath_type       : netdev
datapath_version    : "<built-in>"

If your datapath on the bridge is NOT using netdev, you can change it by changing your datapath.

# ovs-vsctl set bridge br-tun datapath_type=netdev

So if your switch looks good, and your bridge is set with the right netdev datapath type, adding your NIC should be successful with the following command:

# ovs-vsctl add-port br-tun dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs=0000:01:00.0 ofport_request=1

Let me comment a bit on this command to shed some additional insight.

1. I always add my dpdk NICs - whether they are physical or virtual ports - with the name "dpdk" as a prefix as a matter of practice.

2.  A physical PCI NIC is going to use type dpdk. Virtual interfaces, do NOT use that type, and use vhostuser of vhostuserclient types. The distinction between these two is a subsequent discussion.

3. The dpdk-devargs is where you map the port to a specific PCI address. You need to be SURE you are using the correct one, and not inadvertently switching these. A lot of mistakes are made by using the wrong PCI addresses, especially in cases where there are multiple ports on a NIC card where one may be 0000:01:00.0 and another 0000:01:00.1!

4. The ofport_request where you are giving your port a port number on the switch. If you set this to 1, and 1 is taken, you will get an error. But in general, I try to always make a physical NIC port 1. It is extremely rare to add more than one physical NIC to a bridge.

So after adding your NIC to the bridge, you can check the status of it with the ovs-vsctl show command:

# ovs-vsctl show

    Bridge br-tun
        fail_mode: standalone
        Port "dpdk0" --> you will see an error here if the port did not add correctly!
            Interface "dpdk0"
                type: dpdk
                options: {dpdk-devargs="0000:01:00.0"}
        Port br-tun
            Interface br-tun
                type: internal

Another useful command, is to dump your specific bridge, by port number. We can use the "ofctl" command, rather than "vsctl" command.

# ovs-ofctl show br-tun
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000001b21c57204
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
 1(dpdk0): addr:00:1b:21:c5:72:04
     config:     0
     state:      0
     current:    1GB-FD AUTO_NEG
     speed: 1000 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:00:1b:21:c5:72:04 --> LOCAL port maps to the bridge port in ovs-vsctl command.
     config:     0
     state:      0
     current:    10MB-FD COPPER
     speed: 10 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0

And this concludes our post on adding a physical DPDK PCI NIC to an OpenVSwitch bridge.

DPDK Hands-On - Part VI - Configuring OpenVSwitch DPDK Parameters

In earlier posts, we talked about the fact that we needed to check our compatibility with NUMA, HugePages, and IOMMU. And when we decided we had that compatibility, we "enabled" these technologies by passing them into the Linux kernel through grub.

But - now that everything is enabled, OpenVSwitch needs to be configured with a number of parameters that, to the layman's eye, look quite scary and intimidating.  Fortunately, there is a website that elaborates on these parameters much better than I could, so I will list that here:    OVS-DPDK Parameters

The first thing to do, is initialize OpenVSwitch for DPDK. Without this, OpenVSwitch is deaf,dumb and blind as to DPDK-related directives.

# ovs-vsctl set Open_vSwitch . other_config:dpdk-init=true

Next, we will reserve memory, in Hugepages, for DPDK sockets. The command below will reserve 1 Hugepage with an upper limit of 2 Hugepages (each Hugepage = 1G).

# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024"
# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-limit="2048"

NOTE: The Hugepages use can be checked a couple of ways. See prior post on Hugepages.

Next, we instruct OVS to place DPDK threads on specific cores the way we want to. We will do that by specifying a CPU mask for each of two settings. First, let's discuss the OVS directives (settings):

• dpdk-lcore-mask - lcore are the threads that manage the poll mode driver threads
  
• pmd-cpu-mask

The dpdk-lcore-mask is a core bitmask that is used during DPDK initialization and it is where the non-datapath OVS-DPDK threads such as handler and revalidator threads run.

 

The pmd-cpu-mask is a core bitmask that sets which cores are used by OVS-DPDK for datapath packet processing.

Source: https://developers.redhat.com/blog/2017/06/28/ovs-dpdk-parameters-dealing-with-multi-numa/

 

The masks in particular are quite esoteric, and need to be understood properly.

Calculating Mask Values

I found a pmd (poll mode driver) mask calculator out on GitHub, at the following link:

Poll Mode Driver Mask Calculator - Written in Python

Another calculator I found, is at this link:

Mask Calculator - OVS DPDK Parameters

So let's take our box as an example. We have one Numa Node, with 4 cores. One thread per core. This makes things a bit simpler since we don't have to consider sibling pairs.

This mask will set the lcore to run on CPU core 0.

# ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x2

This mask will set the pmd threads to run on CPU cores 2 and 3.

# ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask="0xC"

It will be EASY to see if your pmd cores are set correctly, because if you run top or htop, these cpus will be consumed 100% because the poll mode drivers consume those cores polling for packets!

So per the screen shot below, we can see that cores 2-3 are fully consumed. This is a classic example of why affinity rules matter! The OS scheduler would never attempt to place a process on a cpu that was fully consumed, but if you wanted your OS to run on core 0 (along with the lcore threads), you only have one core remaining (on this system anyway), to run VMs unless you wanted to share cores 0-1 for userspace processes that are launched.

NOTE: in htop, cores 2-3 are shown as 3-4 because htop starts with 1 and not with 0.

DPDK Hands-On - Part V - Custom Compiling DPDK and OpenVSwitch

So in our last blog, I pointed out that you could "just use yum" to install your DPDK and your OpenVSwitch packages.

There are several problems with this. Let me go through them.

OpenVSwitch is not compiled for DPDK by default

First, and most importantly, when you use yum to install OpenVSwitch, you do NOT get an OpenVSwitch that has been compiled with DPDK support. And this is a very time-consuming and painful lesson to learn.

I just assumed, at first, that the reason nothing seemed to be working was that maybe DPDK wasn't enabled. No. It in fact has to have a special compile flag --with-dpdk in order to support DPDK.

Versions Matter - for BOTH DPDK and OpenVSwitch

At this OpenVSwitch link, http://docs.openvswitch.org/en/latest/faq/releases/ there are two very very important tables you need to examine and consider when choosing your DPDK and OpenVSwitch.

• Kernel version - DPDK version compatibility
• DPDK version - OpenVSwitch compatibility

There are also some specific feature tables as well, so that you don't use the wrong version for a specific feature you want.

So given that yum makes it easy to install a particular version of DPDK and OpenVSwitch, the versions are pinned to those in their repository, and the OVS is not compiled for DPDK. Neither version may line up with the kernel you happen to be running.

For example, I am still on a 3.x kernel on this system. Not a 4.x kernel. I wound up choosing:

1. DPDK 17.11.10
2. OVS 2.10.2

Note: I learned how important these versions are, because I have older Intel e1000e NICs on this little Dell Precision T1700 box. Apparently these NICs were all the rage early on in the development cycles for DPDK and OpenVSwitch. But after a while, these NICs are no longer tested, and in fact may no longer work as new drivers are introduced. So in my case, I was advised to go backwards on DPDK and OVS to ensure that I could find a driver that worked and was tested on the e1000e (more on that in a later post).

Uninstalling OpenVSwitch will break your OpenStack!!!
Before compiling a new from-scratch version of DPDK and OpenVSwitch, you need to remove the DPDK and OpenVSwitch that yum had installed from the default repositories (CentOS 7 in my case). When I did a "yum remove" on DPDK, that went smoothly enough, but when I ran "yum remove openvswitch", there are a plethora of packages that have dependencies on this, and yum removed those as well. All of my Neutron OpenVSwitch packages, for example, were removed. So I saved off the names of these packages, so that I could install them later after I custom compiled my DPDK and OpenVSwitch.

Compiling DPDK

I read the documentation on how to download and compile DPDK and OpenVSwitch. How hard could it be, right? make, make configure, make install. bang bang bang. 

And, within a couple of hours, I had compiled DPDK and OpenVSwitch (for DPDK). I was able to bind a NIC using the vfio driver, initialize OpenVSwitch for DPDK, and add ports to a bridge.

 

NOTE: Later, I will learn that the drivers did not work properly and revert to igb_uio drivers.

Then I realized, that there was no way to start up OpenVSwitch like a typical service. And, as mentioned, my OpenStack was not there anymore because of the fact that I ran a yum remove on openvswitch.

And THAT, is why you want to install packages instead of just compiling stuff.

When we install packages on Linux, we take for granted the rather esoteric and complicated process of compiling, linking, and copying resultant files into a package that can be installed on any POSIX-compliant Linux system. 

 

To make an rpm, the tool called rpmbuilder is required. rpmbuilder parses something called a "spec file" and uses that as the map for creating an rpm.

As I was googling around how to build a spec file for these two hand-compiled packages, I stumbled onto the fact that most "responsible" packages, include a spec file already in them. So that you can just run the rpmbuild command on them.

One issue I had was that I had no experience running rpmbuild. I didn't know what options to use. I got some tutelage from a developer on the OpenVSwitch Users Group, so let me cover that here.

For DPDK:

rpmbuild -bb --with shared  pkg/dpdk.spec

For OpenVSwitch:

rpmbuild --with-dpdk --without-check --with autoenable -bb openvswitch-fedora.spec

I did not know that you could pass compile flags into rpmbuild. At first, I had hacked up the Makefiles until I learned this. 

Unfortunately, DPDK did not compile, initially. And neither did OpenVSwitch. The reason for both of these, is that the spec file was not being maintained properly, and had to be tweaked and patched. DPDK is big on documentation, it being a development kit, and it does all kinds of doc-related stuff, and some of that wasn't working. I just needed the drivers, not the documentation. I wasn't writing packet sniffers.

DPDK:

• removed inkscape and doxygen dependencies
• removed texlive-collection-latexextra dependency 
• removed %package doc and %description doc
• removed the line make O=%{target} doc 
• removed %files doc and %doc %{doctor}/dpdk

OpenVSwitch

• Changed the BuildRequires for dpdk
• dpdk-devel changed to dpdk-stable-devel
• Commented out some man pages that were causing issues
• ovs-test.8*
• ovs-vlan-test.8*
• ovsdb.5*
• ovsdb.7*
• ovs-db-server.7*

Finally, after all this tweaking of the spec files, we got a couple of successful rpms, and could install those with: 

# yum localinstall <rpm file>