VMware Deficiency - DRS Not Considering Network Throughput

We had a customer complaining about dropped packets.

So, we looked. Yup, dropped packets.

Why did  we not know? Well, apparently Aria Operations was not alerting on any dropped packets thresholds. But hey - there are a myriad of reasons that packets can be dropped, and some packets NEED to be dropped! So when you look at dropped packets, you really need to get deep into the statistics after an initial look at the situation. Are they sent (Tx) packets? Receive (Rx)? On all vmnics or just one? What protocols? And so on.

In this case, the packets being dropped were receive (Rx) packets.  From the uplink switch.

We then noticed that in this heterogeneous cluster we had,  DRS (VMware's Distributed Resource Scheduler) wanted to stack up a large number of VMs on one host. I think there were 44 VMs on that one host, and 24 on another, and then 3 hosts that had no workloads at all on them.

We had to examine host rules, affinity and anti-affinity rules, et al. We finally called in VMware. Turns out, "this is just the way DRS works". DRS - which I mention in past posts - no longer does a "balanced water level" approach to placement. It will pick the host it likes and as long as it meets a criteria, it will continue placing workloads on that host until no more meet the criteria in which case it will go to the next best host. This supposedly reduces the DRS overhead, reduces live migrations, et al.

But - DRS is ONLY LOOKING AT MEMORY AND CPU!!! IT IS NOT LOOKING AT NETWORK THROUGHPUT. 

 SO IF YOU HAVE 45 VMS TRYING TO INGEST PACKETS, THE NUMBER OF INTERRUPTS GENERATED OVERWHELMS THE RING BUFFER AND THE PACKETS GET DROPPED.

Now, having poll mode drivers might help this situation. We are not using poll mode drivers. 

So, the only way to fix this, is with rules to ensure too many of these VMs are not sitting on a single hypervisor on a single vmnic.

 

High Packet Loss in the Tx of TAP Interfaces

I was seeing some bond interfaces that had high dropped counts, but these were all Rx drops.

I noticed that the tap interfaces on OpenStack compute hosts - which were hooked to OpenContrail's vRouter - had drops on the Tx.

So, in trying to understand why we would be dropping packets on Tap interfaces, I did some poking around and found this link.

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/ovs-dpdk_end_to_end_troubleshooting_guide/high_packet_loss_in_the_tx_queue_of_the_instance_s_tap_interface

From this, article, an excerpt:
"TX drops occur because of interference between the instance’s vCPU and other processes on the hypervisor. The TX queue of the tap interface is a buffer that can store packets for a short while in case that the instance cannot pick up the packets. This would happen if the instance’s CPU is prevented from running (or freezes) for a long enough time."

The article goes on and elaborates on diagnosis, and how to fix by adjusting the Tx Queue Length.