DPDK Hands-On - Part V - Custom Compiling DPDK and OpenVSwitch

So in our last blog, I pointed out that you could "just use yum" to install your DPDK and your OpenVSwitch packages.

There are several problems with this. Let me go through them.

OpenVSwitch is not compiled for DPDK by default

First, and most importantly, when you use yum to install OpenVSwitch, you do NOT get an OpenVSwitch that has been compiled with DPDK support. And this is a very time-consuming and painful lesson to learn.

I just assumed, at first, that the reason nothing seemed to be working was that maybe DPDK wasn't enabled. No. It in fact has to have a special compile flag --with-dpdk in order to support DPDK.

Versions Matter - for BOTH DPDK and OpenVSwitch

At this OpenVSwitch link, http://docs.openvswitch.org/en/latest/faq/releases/ there are two very very important tables you need to examine and consider when choosing your DPDK and OpenVSwitch.

• Kernel version - DPDK version compatibility
• DPDK version - OpenVSwitch compatibility

There are also some specific feature tables as well, so that you don't use the wrong version for a specific feature you want.

So given that yum makes it easy to install a particular version of DPDK and OpenVSwitch, the versions are pinned to those in their repository, and the OVS is not compiled for DPDK. Neither version may line up with the kernel you happen to be running.

For example, I am still on a 3.x kernel on this system. Not a 4.x kernel. I wound up choosing:

1. DPDK 17.11.10
2. OVS 2.10.2

Note: I learned how important these versions are, because I have older Intel e1000e NICs on this little Dell Precision T1700 box. Apparently these NICs were all the rage early on in the development cycles for DPDK and OpenVSwitch. But after a while, these NICs are no longer tested, and in fact may no longer work as new drivers are introduced. So in my case, I was advised to go backwards on DPDK and OVS to ensure that I could find a driver that worked and was tested on the e1000e (more on that in a later post).

Uninstalling OpenVSwitch will break your OpenStack!!!
Before compiling a new from-scratch version of DPDK and OpenVSwitch, you need to remove the DPDK and OpenVSwitch that yum had installed from the default repositories (CentOS 7 in my case). When I did a "yum remove" on DPDK, that went smoothly enough, but when I ran "yum remove openvswitch", there are a plethora of packages that have dependencies on this, and yum removed those as well. All of my Neutron OpenVSwitch packages, for example, were removed. So I saved off the names of these packages, so that I could install them later after I custom compiled my DPDK and OpenVSwitch.

Compiling DPDK

I read the documentation on how to download and compile DPDK and OpenVSwitch. How hard could it be, right? make, make configure, make install. bang bang bang. 

And, within a couple of hours, I had compiled DPDK and OpenVSwitch (for DPDK). I was able to bind a NIC using the vfio driver, initialize OpenVSwitch for DPDK, and add ports to a bridge.

 

NOTE: Later, I will learn that the drivers did not work properly and revert to igb_uio drivers.

Then I realized, that there was no way to start up OpenVSwitch like a typical service. And, as mentioned, my OpenStack was not there anymore because of the fact that I ran a yum remove on openvswitch.

And THAT, is why you want to install packages instead of just compiling stuff.

When we install packages on Linux, we take for granted the rather esoteric and complicated process of compiling, linking, and copying resultant files into a package that can be installed on any POSIX-compliant Linux system. 

 

To make an rpm, the tool called rpmbuilder is required. rpmbuilder parses something called a "spec file" and uses that as the map for creating an rpm.

As I was googling around how to build a spec file for these two hand-compiled packages, I stumbled onto the fact that most "responsible" packages, include a spec file already in them. So that you can just run the rpmbuild command on them.

One issue I had was that I had no experience running rpmbuild. I didn't know what options to use. I got some tutelage from a developer on the OpenVSwitch Users Group, so let me cover that here.

For DPDK:

rpmbuild -bb --with shared  pkg/dpdk.spec

For OpenVSwitch:

rpmbuild --with-dpdk --without-check --with autoenable -bb openvswitch-fedora.spec

I did not know that you could pass compile flags into rpmbuild. At first, I had hacked up the Makefiles until I learned this. 

Unfortunately, DPDK did not compile, initially. And neither did OpenVSwitch. The reason for both of these, is that the spec file was not being maintained properly, and had to be tweaked and patched. DPDK is big on documentation, it being a development kit, and it does all kinds of doc-related stuff, and some of that wasn't working. I just needed the drivers, not the documentation. I wasn't writing packet sniffers.

DPDK:

• removed inkscape and doxygen dependencies
• removed texlive-collection-latexextra dependency 
• removed %package doc and %description doc
• removed the line make O=%{target} doc 
• removed %files doc and %doc %{doctor}/dpdk

OpenVSwitch

• Changed the BuildRequires for dpdk
• dpdk-devel changed to dpdk-stable-devel
• Commented out some man pages that were causing issues
• ovs-test.8*
• ovs-vlan-test.8*
• ovsdb.5*
• ovsdb.7*
• ovs-db-server.7*

Finally, after all this tweaking of the spec files, we got a couple of successful rpms, and could install those with: 

# yum localinstall <rpm file>

Linux rpm package management tools with rpmbuilder and rpmreaper

In all of my years of using Linux, I have never created a package with a package manager build tool. I have of course used rpm, all the time. Querying packages, installing packages, removing packages. I just haven't generated, or built, a package myself. Until now.

We use CentOS here, which is a Red Hat Linux distribution. And Red Hat uses the Red Hat Package Manager (rpm) tools to administratively manage software packages on operating systems that are based on Red Hat Linux. Every package on an rpm-based system has a ".rpm" file suffix, and there is a binary called "rpm" that is used to install, uninstall, query, etc any and all packages on a system (that were created with Red Hat Package Management).

I had always heard that working with rpms (generating them) was tedious, painful, and a general pain in the a$$. One reason has to do with package dependencies. You can run into mutual or circular dependencies, nested dependencies, and many other issues. So I probably avoided making packages for these reasons.

One little-known, but very cool tool, is called rpmreaper. It is part of the epel-release repository. If you run this tool, you can visually inspect details about packages, as shown below.

Sample Screenshot of rpmreaper rpm Package Panagement Tool

So while I had no idea what I was doing, I spent a full day making a package and it didn't go too badly.  The rpm I put together parks a couple of kernel drivers and a configuration file on the system. That's it. Sounds simple, huh? Guess again.

First, kernel drivers it turns out, are compressed on Linux systems now. So I needed to use xz to compress the kernel drivers. Which means an uninstall needs to remove the compressed kernel drivers because the .ko files won't be there. And, when plopping new kernel modules onto a system, you do need to run commands like depmod to re-generate dependencies between the modules.

Now this rpm probably goes beyond what a typical rpm would do. I think as a best practice, an rpm will move files to where they need to be, or remove files from where they should be. That's it. And, they may do system things in an effort to support that charter.

Dependencies
I built the kernel drivers outside of the rpm. I could have gotten heavy and sophisticated and had the rpm compile the kernel drivers. This opens up a can of works about chipsets, target architectures, etc. I decided to keep it simple and that was easy to do, fortunately, because my box was an x86_64 architecture and so was the target boxes they wanted to install the rpms on.

So originally, I had dependencies for packages in the group "Development Tools". I commented those out. I instead put JUST the dependencies that the scripting in the rpm needed.

• bash
• xz (for compressing the kernel modules)
• NetworkManager (nmcli)
• ModemManager (mmcli)

Package Success or Failure

There was so much scripting to check for or start/stop services, and or load/unload kernel drivers that I learned that system return codes aside of the normal 0 exit code would cause the package install or package remove to fail outright. 

My solution to this was to provide feedback in the form of echo commands, and use an "|| true" (or true) to ensure that the command didn't cause the rpm itself to bail out. Because, the commands were really for for administrator convenience - not so much related to the deployment/removal of necessary files.

Definitions

Originally I was defining shell variables in the specific shell functions of the rpm specfile. That became redundant and error prone very quickly when I needed access to these same variables in pre/post script of both the install/uninstall sections of the rpm specfile.

Hence, I had to quickly learn and make use of definitions.Which are sort of like global variables. But, definitions are only used on the creation of the rpm itself. They are not referenced when you install or uninstall the package.

Versioning

Rpm specfiles, as you would expect, have nice versioning support, and it is wise to make use of that and document in your specfile what you are doing in each version iteration! 

Ok, in summary, this was interesting to have FINALLY created my own rpm package. I am sure there is a LOT more to learn, and the sophistication can go way beyond what my rpm is doing right now. I have about a 300 line specfile, mainly due to all of the scripting logic. I am only deploying 5 files in this rpm.