DPDK Hands-On - Part III - Huge Pages

The last post discussed IOMMU, which was something I had never heard of. So I had to research it.

Next, in the DPDK Getting Started Guide, http://doc.dpdk.org/spp/setup/getting_started.html, the following is stated about the requirement for HugePages.

Hugepages must be enabled for running DPDK with high performance. Hugepage support is required to reserve large amount size of pages, 2MB or 1GB per page, to less TLB (Translation Lookaside Buffers) and to reduce cache miss. Less TLB means that it reduce the time for translating virtual address to physical.

This SOUNDS like a requirement, but honestly, I am not sure if this is a requirement, or just an optimization. Will DPDK not function at all without HugePages? There is another guide that also discusses Huge Pages.  https://dpdk-guide.gitlab.io/dpdk-guide/setup/hugepages.html .

Why chance it? Let's go ahead and set up HugePages. But - how do you know if your system even supports HugePages?  As it turns out, if the kernel has support for HugePages, you can run this command, and it will allow you to see if not only if it is supported, but the statistics regarding the size and use of your HugePages.

So - we have 16G of memory in this T1700, 4G of which are allocated to HugePages. Hugepages in Linux can be set by passing the parameters into the kernel, as shown below:

1. default_hugepagesz=1G
2. hugepage_sz=1G
3. hugepages=4
4. transparent_hugepage=never 

NOTE: I am not fully abreast of transparent hugepages, but I see that a lot of people recommend turning this off, which is why I have it disabled above.

 

In the file /etc/default/grub, these can be added, and then the following command run to make it stick:

#grub2mkconfig -O /boot/grub2/grub.cfg 

NOTE: This assumes Grub is being used as your bootloader.

 

And, just as with IOMMU kernel command line parameters, after a reboot you can verify that these parameters are set by running:

# cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-1127.el7.x86_64 root=UUID=4102ab69-f71a-4dd0-a14e-8695aa230a0d ro rhgb quiet iommu=pt intel_iommu=on default_hugepagesz=1G hugepagesz=1G hugepages=4 transparent_hugepage=never LANG=en_US.UTF-8gePages_Surp:        0
Hugepagesize:    1048576 kB

After booting the system up with Hugepages, you can check the summary of Hugepages with the following command:

# grep -i HugePages_ /proc/meminfo
AnonHugePages:         0 kB
HugePages_Total:       4
HugePages_Free:        3
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:    1048576 kB

When we run this command, we can see that out of 4 those, 3 are free. So why is one of them in use???

The answer to why 1G (1 Hugepage) is used, has to do with the initialization of OpenVSwitch. The initialization of OpenVSwitch as the following directives:

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem="1024"
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-limit="2048"

This tells OpenVSwitch to grab 1 Hugepage at startup, with a limit of 2 Hugepages. 

NOTE: Calculating the number of Hugepages you need in OVS, is a topic in and of itself - the subject of a separate post.

Another nifty command, is the ability to see where your Hugepages are distributed, across your NUMA nodes!

# numactl --hardware
# echo ""
# numastat -cm | egrep 'Node|Huge'  

available: 1 nodes (0)
node 0 cpus: 0 1 2 3
node 0 size: 15954 MB
node 0 free: 1808 MB
node distances:
node   0
  0:  10

                 Node 0 Total
AnonHugePages         0     0
HugePages_Total    8192  8192
HugePages_Free     7168  7168
HugePages_Surp        0     0

In this example above, we only have a single NUMA node with 4 cores on it. But had this been a 2 x NUMA Node system with 3 cores apiece, you would be able to see how Hugepages are allocated across your NUMA nodes.