High Packet Loss in the Tx of TAP Interfaces

I was seeing some bond interfaces that had high dropped counts, but these were all Rx drops.

I noticed that the tap interfaces on OpenStack compute hosts - which were hooked to OpenContrail's vRouter - had drops on the Tx.

So, in trying to understand why we would be dropping packets on Tap interfaces, I did some poking around and found this link.

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/ovs-dpdk_end_to_end_troubleshooting_guide/high_packet_loss_in_the_tx_queue_of_the_instance_s_tap_interface

From this, article, an excerpt:
"TX drops occur because of interference between the instance’s vCPU and other processes on the hypervisor. The TX queue of the tap interface is a buffer that can store packets for a short while in case that the instance cannot pick up the packets. This would happen if the instance’s CPU is prevented from running (or freezes) for a long enough time."

The article goes on and elaborates on diagnosis, and how to fix by adjusting the Tx Queue Length.

Beware: Swapping a NIC Card Out Changes the MAC Address

I observed an issue this week where a flapping NIC was replaced by a Dell technician.

When the Dell technician swapped out the NIC card (and left), the interfaces on the card would not come up and go into operation.

They were ABOUT to call Dell back in and swap out the motherboard, when I decided to wander over and take a look and get involved.

It is always important to remember, that when you change out a NIC card, the mac address CHANGES!

And you never know, where that previous mac address might have been used! Here are just a few things a mac address might be used:

• an upstream DHCP server might be assigning an IP address based on mac address
• firewalls might be using the mac address in certain rules and policies
• interfaces in the OS (Linux in particular - especially CentOS) might not come up with a new mac address
• CentOS7 has HWADDR directive in the interface configuration files
• scripts in rc.local or udev may be using the mac address to do certain things
• configure certain interfaces to bridges or bonds

In this particular case, a udev script was renaming a specific interface - based on mac address - and assigning it to a nic teaming configuration (bond).

Don't just swap NICs out! Figure out who might be paying attention to mac addresses before swapping! It can pay dividends.

Graphical Network Simulator-3 (GNS3) - Part II Installation on a Linux Server

Okay for Part II of GNS3, I came in today looking to install GNS3 on a Linux Server.

I noticed that GNS3 is designed to run on Ubuntu Linux, and as I tend to run in a CentOS7 shop, I am now faced with the hump of putting an Ubuntu server in here, or trying to get this to run on CentOS7. It should run on CentOS7, right? After all, this is a Linux world, right? 😏

I decided to take one of my 32Gb RAM servers, an HP box, that runs CentOS7, and follow a cookbook for installing GNS3 on it.

I followed this link:
https://gns3.com/discussions/how-to-install-gns3-on-centos-7-

I chose this box because it runs X Windows. It didn't have Python 3.6 on it, or the pip36 used for installing and managing python 3.6 packages.

A lot of steps in this thing.

Some questions I have about this cookbook that I need to look into:

1. Why does the cookbook use VirtualBox on Linux? I have KVM installed. Surely I can use that instead of VirtualBox. I only use VirtualBox on my Win10 laptop. So I have, for now, skipped that section.

2. What is IOU support? I will need to google that.

UPDATE: IOU (also called IOL, which stands for IOS on Linux, is basically an IOS simulator) that can run on an i386 chipset.  You would need and want that if you run any Cisco elements on the GNS3 simulator.

What on Earth is Canonical Doing with Ubuntu? Sheez

I have been using CentOS almost exclusively since I have been working here, first with CentOS 6, and then (now) CentOS7. I have watched the kernels move up from 3.x to 4.x, I have fought with (and still fight with) NetworkManager, etc.

You get used to what you use.

I have also used Ubuntu in the past, 14.04, and 16.04, but it has been a while.

So, I needed to install Ubuntu in order to run OSM, because OSM (on their website at least) wants to use Ubuntu. I think Ubuntu is probably bigger in Europe, is my guess.

So - for two straight days now, I have been trying to install a Ubuntu Cloud image and get it working on a KVM system. Something that SHOULD be simple, right? Wrong.

Here is a rundown of all of the headaches I have run into thus far, which has pissed me off about Ubuntu.

1. On 16.04, the root file system is only 2G for the cloud image you download off the web.

I ran out of disk space in no time flat. Trying to install X Windows and a Display Manager, which by default are not enabled on the cloud image. 

Trying to increase the root file system? Damn near impossible. I tried using qemu-img --resize, and that only created a /vdb file system. The ./dev./sda1 was STILL, and REMAINED, 2G. I could not install X Windows, I couldn't do squat. I am sure if I rolled up my sleeves, and got to work using advanced partitioning tools and whatnot, I could have made this happen. Or, maybe not. Point is, this was a hassle. An unnecessary hassle in my opinion.

2. I realized that the 18.04 Ubuntu uses a qcow2 format - which you CAN resize. Again, why Ubuntu is using 2G as a root file system size is beyond me, and this is ANNOYING. This is the year 2019.

So, I resized the image, and put a password on the image (cloud images are not set up to log in with prompt, only certificates, which of course is a good practice, albeit a hassle for what I needed).

3. I launched 18.04 and guess what? NO NETWORKING!!!! @^%$

I realize no networking was set up. At all! WHAT???

4. Let's go set up networking. Yikes! You CAN'T!!!!!! WHY? Because the iproute2 packages and legacy packages that everyone in the WORLD uses, are not on the machine!

They want you to use this newfangled tool called NetPlan to set up your networking!?!?

Fine. You can google this and set it up, which I did.

BUT WHY ARE ALL OF THESE LINUX DISTRIBUTIONS BECOMING SO DIFFERENT?

THAT IS NOT, I SAY...NOT...WHAT LINUX IS ALL ABOUT?????

I remember when Gentoo came out, and how different a beast it was. Now, the distinction between CentOS and Ubuntu is becoming a very wide chasm.