DPDK Hands-On - Part V - Custom Compiling DPDK and OpenVSwitch

So in our last blog, I pointed out that you could "just use yum" to install your DPDK and your OpenVSwitch packages.

There are several problems with this. Let me go through them.

OpenVSwitch is not compiled for DPDK by default

First, and most importantly, when you use yum to install OpenVSwitch, you do NOT get an OpenVSwitch that has been compiled with DPDK support. And this is a very time-consuming and painful lesson to learn.

I just assumed, at first, that the reason nothing seemed to be working was that maybe DPDK wasn't enabled. No. It in fact has to have a special compile flag --with-dpdk in order to support DPDK.

Versions Matter - for BOTH DPDK and OpenVSwitch

At this OpenVSwitch link, http://docs.openvswitch.org/en/latest/faq/releases/ there are two very very important tables you need to examine and consider when choosing your DPDK and OpenVSwitch.

• Kernel version - DPDK version compatibility
• DPDK version - OpenVSwitch compatibility

There are also some specific feature tables as well, so that you don't use the wrong version for a specific feature you want.

So given that yum makes it easy to install a particular version of DPDK and OpenVSwitch, the versions are pinned to those in their repository, and the OVS is not compiled for DPDK. Neither version may line up with the kernel you happen to be running.

For example, I am still on a 3.x kernel on this system. Not a 4.x kernel. I wound up choosing:

1. DPDK 17.11.10
2. OVS 2.10.2

Note: I learned how important these versions are, because I have older Intel e1000e NICs on this little Dell Precision T1700 box. Apparently these NICs were all the rage early on in the development cycles for DPDK and OpenVSwitch. But after a while, these NICs are no longer tested, and in fact may no longer work as new drivers are introduced. So in my case, I was advised to go backwards on DPDK and OVS to ensure that I could find a driver that worked and was tested on the e1000e (more on that in a later post).

Uninstalling OpenVSwitch will break your OpenStack!!!
Before compiling a new from-scratch version of DPDK and OpenVSwitch, you need to remove the DPDK and OpenVSwitch that yum had installed from the default repositories (CentOS 7 in my case). When I did a "yum remove" on DPDK, that went smoothly enough, but when I ran "yum remove openvswitch", there are a plethora of packages that have dependencies on this, and yum removed those as well. All of my Neutron OpenVSwitch packages, for example, were removed. So I saved off the names of these packages, so that I could install them later after I custom compiled my DPDK and OpenVSwitch.

Compiling DPDK

I read the documentation on how to download and compile DPDK and OpenVSwitch. How hard could it be, right? make, make configure, make install. bang bang bang. 

And, within a couple of hours, I had compiled DPDK and OpenVSwitch (for DPDK). I was able to bind a NIC using the vfio driver, initialize OpenVSwitch for DPDK, and add ports to a bridge.

 

NOTE: Later, I will learn that the drivers did not work properly and revert to igb_uio drivers.

Then I realized, that there was no way to start up OpenVSwitch like a typical service. And, as mentioned, my OpenStack was not there anymore because of the fact that I ran a yum remove on openvswitch.

And THAT, is why you want to install packages instead of just compiling stuff.

When we install packages on Linux, we take for granted the rather esoteric and complicated process of compiling, linking, and copying resultant files into a package that can be installed on any POSIX-compliant Linux system. 

 

To make an rpm, the tool called rpmbuilder is required. rpmbuilder parses something called a "spec file" and uses that as the map for creating an rpm.

As I was googling around how to build a spec file for these two hand-compiled packages, I stumbled onto the fact that most "responsible" packages, include a spec file already in them. So that you can just run the rpmbuild command on them.

One issue I had was that I had no experience running rpmbuild. I didn't know what options to use. I got some tutelage from a developer on the OpenVSwitch Users Group, so let me cover that here.

For DPDK:

rpmbuild -bb --with shared  pkg/dpdk.spec

For OpenVSwitch:

rpmbuild --with-dpdk --without-check --with autoenable -bb openvswitch-fedora.spec

I did not know that you could pass compile flags into rpmbuild. At first, I had hacked up the Makefiles until I learned this. 

Unfortunately, DPDK did not compile, initially. And neither did OpenVSwitch. The reason for both of these, is that the spec file was not being maintained properly, and had to be tweaked and patched. DPDK is big on documentation, it being a development kit, and it does all kinds of doc-related stuff, and some of that wasn't working. I just needed the drivers, not the documentation. I wasn't writing packet sniffers.

DPDK:

• removed inkscape and doxygen dependencies
• removed texlive-collection-latexextra dependency 
• removed %package doc and %description doc
• removed the line make O=%{target} doc 
• removed %files doc and %doc %{doctor}/dpdk

OpenVSwitch

• Changed the BuildRequires for dpdk
• dpdk-devel changed to dpdk-stable-devel
• Commented out some man pages that were causing issues
• ovs-test.8*
• ovs-vlan-test.8*
• ovsdb.5*
• ovsdb.7*
• ovs-db-server.7*

Finally, after all this tweaking of the spec files, we got a couple of successful rpms, and could install those with: 

# yum localinstall <rpm file>