IP MAC Discovery on NSX-T

We had a deployment where two customer VMs were deployed as an Active Standby cluster. And the failover wasn't working when they tested it.

I had already deployed a fully working pair of Active-Standby Virtual Machines using KeepaliveD, so I knew that VRRP worked. Now, I am not sure that the customer is using VRRP per se, but the concept of Active Standby failover remains a constant whether both of us were using a strict RFC-compliant VRRP or not. 

 So what was the difference between these customer VMs, and our VMs?

Well, the difference was that I was running my VMs on VLAN-backed network segments that were jacked into (legacy) vCenter / ESXi Distributed Port Groups. The customer's VMs, were jacked into NSX-T virtual switches (overlay segments).

So after re-verifying my VRRP failover (which worked flawlessly in both multicast and unicast peering configurations), the problem seemed to be traced back to NSX-T.

Was it Mac Spoofing? Was it a Firewall? NSX-T does run an Overlay Firewall! And these Firewalls are at the segment level, but also the Transport Zone (Tier 1 router) level.  Sure enough, we realized that the Tier 1 Firewall was dropping packets on failover attempts.

After much testing, it was concluded that it was related to TOFU on the IP Discovery Switching Profile.

From this VMWare link, we get some insight on this:

Understanding IP Discovery Switching Profile

By default, the discovery methods ARP snooping and ND snooping operate in a mode called trust on first use (TOFU). In TOFU mode, when an address is discovered and added to the realized bindings list, that binding remains in the realized list forever. TOFU applies to the first 'n' unique <IP, MAC, VLAN> bindings discovered using ARP/ND snooping, where 'n' is the binding limit that you can configure. You can disable TOFU for ARP/ND snooping. The methods will then operate in trust on every use (TOEU) mode. In TOEU mode, when an address is discovered, it is added to the realized bindings list and when it is deleted or expired, it is removed from the realized bindings list. DHCP snooping and VM Tools always operate in TOEU mode.

So guess what? After disabling this profile, and effectively disabling TOFU mode, TOEU mode kicked in and lo and behold, the customer's failover started working. 

DPDK Testing using TestPMD on a VMWare Virtual Machine

 

Testing and verifying DPDK is NOT easy. And it is even more challenging in VM environments.

After investing in VMWare hypervisors that supposedly run DPDK, we wanted to test and verify that a) it worked, and b) the performance was as advertised.

Below is a list of steps we took to get the host and a DPDK-enabled VM ready:

• Hypervisor(s)
• Enabled the ixgben_ens drivers on the host. There are some ESXI CLI commands you can run to ensure that these are loaded and working. 
  
• VM Settings
• VMXNet3 adaptor types on VM
•  Latency Sensitivity = High (sched.cpu.latencySensitivity=TRUE)
• Hugepages enabled in VM (sched.mem.lpage.enable1GPage TRUE
• Reserve all Guest Memory
• Enable Multiple Cores for High I/O Workloads (ethernetX.ctxPerDev=”1” )
• CPU Reservation
• NUMA Affinity (numaNodeAffinity=X)

After this, I launched the VM. I was smart enough to launch the VM with 3 NICs on it.

1. eth0 - used as a management port, for ssh and such.
2. eth1 - this one to be used for DPDK testing
   
3. eth2 - this one to be used for DPDK testing
   

Launching a VM (i.e. a RHEL Linux VM) with these settings, does NOT mean that you are ready for DPDK!! You still need a DPDK-compiled application on your OS. DPDK applications need to use DPDK-enabled NIC drivers on the VM, and on a Linux VM, these drivers are typically run as kernel modules. There are several different types and kinds of DPDK drivers (kernel modules), such as vfio, uio-pci-generic, igb_uio, et al.

To prepare your VM for testing, we decided to install DPDK, and then run the TestPMD application.

Installing DPDK

To get DPDK, you can go to dpdk.org, and download the drivers, which comes as a tar.gz file that can be unpacked. Or, there is a github site that you can use the clone the directories and files.

# git clone http://dpdk.org/git/dpdk

 It is important to read the instructions when building DPDK, because the old-style "make configure", "make", "make install" process has been replaced by fancier build tools like meson and ninja that you need to install. I chose to install by going to the top of the directory tree, and typing:

# meson -Dexamples=all build

This does not actually compile the code. It sets the table for you to use Ninja to build the code. So the next step was to type:

# ninja

Followed by:

# ninja install

The "ninja install" puts a resultant set of DPDK executables (some ELF, some Python), in /usr/local/bin directory (maybe installs some stuff in other places too).

Right away, I hit a snag. When I tried to run dpdk_setup.py, to bind the VM The kernel module igb_uio.ko was nowhere to be found.

I was completely at a loss about this, until I realized that some other DPDK packages (test load generators) compile DPDK and the igb_uio.ko drivers, either by including them outright, or copying the sources into the build process. Trex, for example, builds the drivers. And so does a package called DTS. After I decided to git clone the DTS package, I stumbled upon some documentation in an archived package called DTS (DPDK Testing Suite). In the DTS package, in the /opt/github/dts/doc/dts_gsg/usr_guide there is a file called igb_uio.rst which describes how to compile the igb_uio.ko drivers for use with DTS. This was the missing link. The section of the file up front, described that the drivers have been moved into a different github repository - and are now separated from DPDK!

Get Source Code - note: assumption is that you are doing this in /opt directory.
---------------

Get igb_uio::

   git clone http://dpdk.org/git/dpdk-kmods
   git clone git://dpdk.org/dpdk-kmods

Get DPDK::

   git clone git://dpdk.org/dpdk
   git clone http://dpdk.org/git/dpdk

The author of this igb_uio.rst file described the process that can be used to fuse DPDK and the drivers back together into a single build - the way it used to be. How convenient. Here is how that is done.

Integrate igb_uio into DPDK
---------------------------

Assume you have cloned the dpdk and dpdk-kmods source code
in opt/dpdk and opt/dpdk-kmods.

Step 1
# Copy dpdk-kmods/linux/igb_uio/ to dpdk/kernel/linux/:

    [root@dts linux]# cp -r /opt/dpdk-kmods/linux/igb_uio /opt/dpdk/kernel/linux/

you should see igb_uio in your output:

    [root@dts linux]# ls /opt/dpdk/kernel/linux/
    igb_uio  kni  meson.build

Step 2:
# enable igb_uio build in meson:

since we have copied the directory over to /opt/dpdk, we will edit the meson.build there.

*   add igb_uio in /opt/dpdk/kernel/linux/meson.build subdirs as below:

     subdirs = ['kni', 'igb_uio']

NOTE: this is an important step not to miss because it will not build if you don't do this.

Step 3:
*   create a file of meson.build in /opt/dpdk/kernel/linux/igb_uio/ as below:

     # SPDX-License-Identifier: BSD-3-Clause
     # Copyright(c) 2017 Intel Corporation

     mkfile = custom_target('igb_uio_makefile',
             output: 'Makefile',
             command: ['touch', '@OUTPUT@'])

     custom_target('igb_uio',
             input: ['igb_uio.c', 'Kbuild'],
             output: 'igb_uio.ko',
             command: ['make', '-C', kernel_dir + '/build',
                     'M=' + meson.current_build_dir(),
                     'src=' + meson.current_source_dir(),
                     'EXTRA_CFLAGS=-I' + meson.current_source_dir() +
                             '/../../../lib/librte_eal/include',
                     'modules'],
             depends: mkfile,
             install: true,
             install_dir: kernel_dir + '/extra/dpdk',
             build_by_default: get_option('enable_kmods'))

How wonderful. To recap, here is what we did:

1. copy the source files from dpdk-kmods into the proper directory of dpdk
2. snap in the proper meson build file (which the author graciously provides)
   
3. uninstall (previous build, assuming you built DPDK before doing all of this) 
4. rebuild
5. reinstall

Step 3:

# cd /opt/dpdk/build

# ninja uninstall

Step 4:
# ninja

Step 5:

# ninja install

A quick find command shows that the kernel module was built.

[root@acdcchndnfvdpk0001 dpdk]# find . -print | grep ko
./build/drivers/net/octeontx/base/libocteontx_base.a.p/octeontx_pkovf.c.o
./build/lib/librte_table.a.p/table_rte_table_hash_cuckoo.c.o
./build/lib/librte_hash.a.p/hash_rte_cuckoo_hash.c.o
./kernel/linux/igb_uio/igb_uio.ko
./kernel/linux/igb_uio/.igb_uio.ko.cmd
./drivers/net/octeontx/base/octeontx_pkovf.c
./drivers/net/octeontx/base/octeontx_pkovf.h
./lib/hash/rte_cuckoo_hash.h
./lib/hash/rte_cuckoo_hash.c
./lib/table/rte_table_hash_cuckoo.h
./lib/table/rte_table_hash_cuckoo.c

Now, we have something we can use to bind our adaptors to the drivers!!! 

You can bind the adaptors to the drivers using a couple of different methods. You can use a utility that is supplied by DPDK to do it (dpdk-devbind.py), or you can also use a nifty Linux utility called driverctl, which I prefer (this needs to be installed typically with a package manager as it generally does not roll onto the OS with a default installation). 

A script I use to do the binding looks like this:

# cat bind-pci.sh
#!/bin/bash

lshw -class network -businfo | grep pci

while :
do
   echo "Linux Interface to override (e.g. p1p1, p1p2, p1p3, p1p4):"
   read iface
   if [ ${iface} == "skip" ]; then
      break
   fi
   lshw -class network -businfo | grep pci | grep ${iface}
   if [ $? -eq 0 ]; then
      pci=`lshw -class network -businfo | grep pci | grep ${iface} | awk '{printf $1}' | cut -f2 -d"@"`
      echo "We will override the kernel driver with igb_uio for PCI address: ${pci}"
      driverctl set-override ${pci} igb_uio
      break
   fi
done 

When you run this script, you can check to see if the binding was successful by running a DPDK command:

# python3 /usr/local/bin/dpdk-devbind.py --status

And this command will show you whether the binding worked or not.

Network devices using DPDK-compatible driver
============================================
0000:0b:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3
0000:13:00.0 'VMXNET3 Ethernet Controller 07b0' drv=igb_uio unused=vmxnet3

Network devices using kernel driver
===================================
0000:03:00.0 'VMXNET3 Ethernet Controller 07b0' if=eth0 drv=vmxnet3 unused=igb_uio *Active*

NOTE: To remove a driver binding, "driverctl unset-override ${pci address}" would be used. In which case, the driver will now become visible to the Linux OS in the Virtual Machine again.

So we now have one adaptor that the Linux Networking Kernel sees (eth0), but the two adaptors Linux saw prior to the binding (eth1, and eth2), have now been "reassigned" to DPDK. And the OS does not see them at all, actually, anymore. 

If we run an ifconfig, or an "ip a" command to see the Linux network interfaces in the VM, this is what it now looks like.

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:b7:83:1a brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.10/24 brd 192.168.2.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:feb7:831a/64 scope link
       valid_lft forever preferred_lft forever

NOTE: No eth1 or eth2 shows up in Linux anymore, as they have been handed over to DPDK, which bypasses the Linux Kernel entirely.

Okay, now we have adaptors set up for DPDK. Now what? In our next step, we will do some simple verification with TestPMD.

ESXi is NOT Linux

ESXi is not built upon the Linux kernel, but uses an own VMware proprietary kernel (the VMkernel) and software, and it misses most of the applications and components that are commonly found in all Linux distributions.

Because ESXi uses "-ix" commands (Unix, Linux, POSIX), it "looks and smells" like Linux, but in fact, these commands are similar to the package CygWin that one can run on a Windows system to get a Linux terminal and command line interpreter. ESXi does not use CygWin, however. They run something called BusyBox.

BusyBox is something used on a lot of small-factor home networking gear. PfSense for example, runs Berkeley Unix (BSD). But many small routers (Ubiquiti EdgeMax comes to mind) use different chipsets, different OS kernels, and then use BusyBox to abstract this kernel away from users by providing a common interface - meaning users don't need to learn a whole slew of new OS commands.

 ESXi has a LOT of things that Linux does NOT have:

1. File systems VMFS6 for example is the newest revision of VMFS.

2. Process Scheduler - and algorithms

3. Kernel hooks that tools like esxtop use (think system activity reporting in Unix and Linux) 

 

This article (the source for this post), discusses some nice facts in comparing ESXi to Linux:

ESXi-is-not-based-on-Linux

I learned some interesting things from this article, such as:

ESXi even uses the same binary format for executables (ELF) than Linux does, so it is really not a big surprise anymore that you can run some Linux binaries in an ESXi shell - provided that they are statically linked or only use libraries that are also available in ESXi! (I exploited this "feature" when describing how to run HP's hpacucli tool in ESXi and when building the ProFTPD package for ESXi).

...You cannot use binary Linux driver modules in ESXi. Lots of Linux device drivers can be adapted to ESXi though by modifying their source code and compiling them specifically for ESXi. That means that the VMkernel of ESXi implements a sub-set of the Linux kernel's driver interfaces, but also extends and adapts them to its own hypervisor-specific needs.

In my opinion this was another very clever move of the VMware ESXi architects and developers, because it makes it relatively easy to port an already existing Linux driver of a hardware device to ESXi. So the partners that produce such devices do not need to develop ESXi drivers from scratch. And it also enables non-commercial community developers to write device drivers for devices that are not supported by ESXi out-of-the-box!

There is a PDF download of the ESXi architecture, which can be downloaded here:

 https://www.vmware.com/techpapers/2007/architecture-of-vmware-esxi-1009.html

VMWare Clustered File Systems - VMFS5 vs VMFS6

 

 A nice table that describes the differences between VMWare's VMFS5 and the new VMFS 6.

Source: http://www.vmwarearena.com/difference-between-vmfs-5-vmfs-6/

For the difference in 512n versus 512e:

VMFSsparse:

VMFSsparse is a virtual disk format used when a VM snapshot is taken or when linked clones are created off the VM. VMFSsparse is implemented on top of VMFS and I/Os issued to a snapshot VM are processed by the VMFSsparse layer. VMFSsparse is essentially a redo-log that grows from empty (immediately after a VM snapshot is taken) to the size of its base VMDK (when the entire VMDK is re-written with new data after the VM snapshotting). This redo-log is just another file in the VMFS namespace and upon snapshot creation the base VMDK attached to the VM is changed to the newly created sparse VMDK.

SEsparse (space efficient):

SEsparse is a new virtual disk format that is similar to VMFSsparse (redo-logs) with some enhancements and new functionality. One of the differences of SEsparse with respect to VMFSsparse is that the block size is 4KB for SEsparse compared to 512 bytes for MFSsparse. Most of the performance aspects of VMFSsparse discussed above—impact of I/O type, snapshot depth, physical location of data, base VMDK type, etc.—applies to the SEsparse format also.

Jinja2 Templating in Ansible

Lately, I have been playing around with Jinja2 Templating in Ansible. Let me explain the context of that.

With the Morpheus CMP solution, it has an Automation Workflow engine that can be used to run Tasks, or whole sets of Workflows, in a variety of different technologies (scripting languages, Ansible, Chef, Puppet, et al).

To access the variables about your Virtual Machine, say after you launch it, you put tags into your script to reference variables. The tags can have subtle differences in syntax, depending on whether it is a bash script, a Python script, or an Ansible playbook.

This post, is related to Ansible specifically.

If you are needing an explicit specific value, the tag in an Ansible playbook would look as follows:
    - name: "set fact hostname"
      set_fact:
        dnsrecord: |
          {{ morpheus["instance"]["hostname"] | trim }}

Really strange, and confusing, syntax. Not to mention, this pipe to a supposed function called trim.

What language is this? I thought it was groovy, or some kind of groovy scripting language - at first. Then, I thought it was a form of Javascript. Finally, after some web research, I have come to learn that this markup is Ansible's Jinja2 scripting language.

First, I had to understand how Morpheus worked. I realized that I could use a Jinja2 tag to dump the entire object (in JSON) about a launched virtual machine (tons of data actually). Once I understood how Morpheus worked, and the JSON it generates, I was able to go to work snagging values that I needed in my scripts.

But - eventually, my needs (use cases) became more complex. I needed to loop through all of the interfaces of a virtual machine! How do you do THAT??

Well, I discovered that to do more sophisticated logic structures (i.e. like loops), the markup and tagging is different, and the distinctions are important. You can wind up pulling your hair out if you don't understand them.

Let's take an example where we loop through a VM's interfaces with Jinja2.

In this example, we loop through all interfaces of a virtual machine. But - we use an if statement to only grab the first interface's ip address. 

Note: To be optimized, we should break after we get that first ip address, but breaking out of loops is not straightforward in Jinja2, and there are only a handful of interfaces, so we will let the loop continue on, albeit wastefully.

    - name: set fact morpheusips
      set_fact:
         morpheusips: |
           {% for interface in morpheus['instance']['container']['server']['interfaces'] %}
             {% if loop.first %}
                {{ interface['ipAddress'] }}
             {% endif %}
           {% endfor %}

Note that an explicit specific value - has NO percent signs in the tag!

But, the "logic", like for loops, if statements, et al, those DO use percent signs in the tag!

This is extremely important to understand!

Now, the variable we get - morpheusips - is a string, which contains leading and trailing spaces, and has newlines - including an annoying newline at the end of the string which wreaked havoc when I needed to convert this string to an array (using the split function).  

I found myself having to write MORE code, to clean this up, and found more useful Jinja2 functions for doing this kind of string manipulation and conversion (to an array).

    - name: "Replace newlines and tabs with commas so we can split easier"
      set_fact:
         commasep: "{{ morpheusips | regex_replace('[\\r\\n\\t]+',',') | trim }}"

    - name: "Remove comma at the end of the string"
      set_fact:
         notrailcomma: "{{ commasep | regex_replace(',$','') | trim }}"

    - name: "convert the ip delimeter string to a list so we can iterate it"
      set_fact:
         morpheusiplst: "{{ notrailcomma.split(',') }}"

    - name: "Loop and Print variable out for morpheusiplst"
      ansible.builtin.debug:
         var: morpheusiplst

I am NOT a guru, or a SME, on Jinja2 Templating. But, this is a blog to share what I have been poking around with as I get used to it to solve some problems.

HAProxy Problem Solved

Okay, an update on my latest post on HAProxy. It turns out that the issue had nothing to do with HAProxy at all, but that the clustering on the back-ends was not working properly.

So, I have reverted the HAProxy back to its original configuration, prior to getting into debug mode. 

Note: The Stats page in HAProxy, by the way, is an invaluable way to help see if your proxy is working correctly. You can see the load distribution, you can see the back-end health checks, etc. Plus the GUI is simple, informative and intuitive.

Below is a quick look at the front and back end configurations. In this configuration, we are using HAProxy as a Reverse Proxy, where we terminate incoming requests (SSL Termination), and rather than forward to our back-ends in the clear, we utilize back-end certificates to proxy to the back-ends using SSL. This is a bit unconventional perhaps in a typical reverse proxy scenario.

#---------------------------------------------------------------------
# ssl frontend
#---------------------------------------------------------------------
frontend sslfront
   mode tcp
   option tcplog
   bind *:443 ssl crt /etc/haproxy/cert/yourcert.pem
   default_backend sslback

#---------------------------------------------------------------------
# sticky load balancing to back-end nodes based on source ip.
#---------------------------------------------------------------------
backend sslback
   # redirects http requests to https which makes site https-only.
   #redirect scheme https if !{ ssl_fc }
   mode http
   balance roundrobin
   option ssl-hello-chk

   option httpchk GET /ping
   http-check expect string TESTPING

   stick-table type ip size 30k expire 30m
   stick on src
   #stick-table type binary len 32 size 30k expire 30m
   #stick on ssl_fc_session_id
   default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions

   server servername01 192.168.20.10:443 ssl check id 1 maxconn 1024
   server servername02 192.168.20.11:443 ssl check id 2 maxconn 1024
   server servername03 192.168.20.12:443 ssl check id 3 maxconn 1024

HAProxy - Aggravating Problem I Have Not Solved

I have not ever really blogged on proxies. I don't have a lot of proxy experience and don't consider myself a guru with proxies, load balancers, etc.

But more and more often, solutions have come in that require load distribution to an N+1 (Active Active) cluster. And, HAProxy is supposed to be a rather lightweight and simple approach, especially in situations where the mission is not totally critical, or the load is not seriously high.

I originally set HAProxy up to distribute load to a Cloudify cluster. And Cloudify provided the configuration for HAProxy that they had tested in their lab, and that they knew worked well. Later, I set HAProxy up to load balance our Morpheus cluster. Initially it was working fine. 

Or, so it seemed. Later, I noticed errors. The first thing you generally do when you see errors, is to tell HAProxy to use one node (and not 2 or 3), so that you can reduce troubleshooting complexity and examine the logs on just one back-end node.  So in doing this, I managed to rather quickly figure out that if I told HAProxy to use one back-end node, things worked fine. When I told HAProxy to use two or more back-end nodes, things didn't work.  

So that's where it all started.

The Problem
Below is a picture of what we are doing with HAProxy, and based on the picture below, web access comes in on the northbound side of the picture, and web access is not the problem we are having.  The problem, is that VMs that are deployed onto various internal networks by Morpheus "phone home" and they phone home on a different network interface. 

This works fine with a single back-end enabled. But if you enable more than one back-end in HAProxy, Morpheus fails to fully transition the state of the VM to "running".

In testing this out a bit and dumping traffic, we initially noticed something interesting. The Source IP coming into each Morpheus node, was not the HAProxy VIP - it was the interface IP address. We wound up solving this, by telling KeepAliveD to delete and re-create the routes with the VIP to be used as the Source IP - but only when it had control of the VIP. But in the end, while this made traffic analysis (tcpdump on the Morpheus nodes) a bit more clear about the traffic flow, it did not solve the actual issue.

I STILL don't know why it works with one back-end, and not two or more. I have had Proxy experts in our organization come in and look, and they seem to think HAProxy is doing its job properly, and that the issue is on the back-end clustering. The vendor, however, is telling us the issue is with HAProxy.

Our next step may be to configure a different load balancer. That should definitely rule things out. I know Squid Proxy is quite popular, but these tools do have a Learning Curve, and I have zero zilch experience with Squid Proxy. I think we may use a Netscaler Load Balancer if we wind up going with another one.

I should mention that the HAProxy configuration is not the simplest. And as a result of configuring this, I have increased my general knowledge on Load Balancing.