Kubernetes Networking - Multus

It's been a while since I have posted on here. What have I been up to?

Interestingly enough, I have had to reverse-engineer a Kubernetes project. I was initially involved with this, but got pulled off of it, and the project had grown immensely in its layers, complexity and sophistication in my absence.  The chief developer on it left, so I had to work with a colleague to try and get the solution working, deployable and tested.

Right off the bat, the issue was related to Kubernetes Networking. That was easy to see.

The project uses Multus to create multi-homed pods (pods with multiple network interface adaptors).

By default, a Kubernetes pod only allows a single NIC (i.e. eth0). If you need two interfaces or more, there is a project call Multus (Intel sponsors this) that accomodates this requirement.

Multus is not a simple thing to understand. Think about it. You have Linux running on baremetal hosts. You have KVM virtual machines running on the VMs (virtualized networking). You have Kubernetes, and its Container Networking Interface plugins that supply a networking fabric amongst pods (Flannel, Weave, Calico, et al). And, now, on top of that, you have - Multus.

Multus is not a CNI itself. It does not "replace" Flannel, or Weave, but instead inserts itself between Kubernetes and Flannel or Weave much like a proxy or a broker would.

This article here has some good diagrams and exhibits that show this:

https://neuvector.com/network-security/advanced-kubernetes-networking/

[ I am skipping some important information about the Multus Daemonset here - and how that all works. But may come back to it. ]

One issue we ran into, is that we had two macvlan (Layer 2) configurations.

One used static host networking configuration:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "eth1",
      "mode": "bridge",
      "ipam": {
        "type": "host-local",
        "subnet": "10.10.20.0/24",
        "rangeStart": "10.10.20.1",
        "rangeEnd": "10.10.20.254",
        "routes": [
          { "dst": "0.0.0.0/0" }
        ],
        "gateway": ""
      }
    }

while the other used DHCP.

{
   "cniVersion": "0.3.1",
   "name": "macvlan-conf-2",
   "type": "macvlan",
   "master": "eth1",
   "mode": "bridge",
   "ipam": {
             "type": "dhcp",
             "routes": [ { "dst": "192.168.0.0/16", "gw": "192.168.0.1" } ]
           },
   "dns": {
             "nameservers": [ "4.4.4.4", "8.8.8.8" ]
          }
}

The DHCP directive is interesting, because it will NOT work unless you have ANOTHER cni plugin called cni-dhcp deployed into Kubernetes so that it is installed on each Kubernetes node that is receiving containers that use this. This took me a WHILE to understand. I didn't even know about the plugin, its existence, or anything like that.

We were running into an issue where the DHCP Multus pods (those that used this macvlan-conf-2) where stuck in an Initializing state. After considerable debugging, I figured out the issue was with DHCP.

Once I realized the plugin existed, I knew the issue had to either be with the plugin (which requests leases), or the upstream DHCP server (which responds). In the end, it turned out to be that the upstream DHCP server was returning routes that the dhcp plugin could not handle. By removing these routes, and letting the upstream DHCP server just worry about ip assignment, the pods came up successfully.

Kubernetes Part VI - Helm Package Manager

This past week, my colleague has introduced me to something called Helm, which is sort of like a "pip" for Python. It manages Kubernetes packages (it is a Kubernetes Package Manager).

The reason this was introduced:
We found SEVERAL github repos with Prometheus Metrics in them, and they were not at all consistent.

• Kubernetes had one
• There was another one at a stefanprod project
• There was yet a third called "incubator"

My colleague, through relentless research, figured out (or decided) that the one installed through Helm was the best one.

This meant I had to understand what Helm is. Helm is divided into a client (Helm), a server (Tiller), and you install packages (Charts). I guess it's a maritime themed concept, although I don't know why they can't call a package a package (copyright reasons maybe?).

So - I installed Helm, and that went smoothly enough. I installed it on my Kubernetes Master.  I also downloaded a bunch of Charts off the stable release in GitHub (Prometheus is one of these). These all sit in a /stable directory (after the git clone, ./charts/stable).

When I came back in, and wanted to pick back up, I wasn't sure if I had installed Prometheus or not. So I ran a "helm list", and got the following error:

Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list configmaps in the namespace "kube-system"

Yikes. For a newbie, this looked scary. Fortunately Google had a fix for this on a StackOverflow page.

I had to run these commands:

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'      
helm init --service-account tiller --upgrade

These seemed to work. The "helm list" command showed no results, though, so I guess I need to install the prometheus package (sorry...chart) after all with helm now.

But, more importantly, I really need to take some time to understand what we just ran above, with regards to this stuff above; the cluster role bindings, et al.

Jumping into Kubernetes without understanding Docker and Containers

Clearly, it's best to understand Containers and Docker before you jump right into the deep water of Kubernetes.

I went in and started following cookbooks and how-to guides on Kubernetes without "stepping back" and trying to learn Docker first, and this has bit me and caused me to now go back to learn a bit about Docker.

As it turns out, the Containers I was launching with Kubernetes kept failing with a CrashLoopBackOff error.

It took me a while to learn how to debug this effectively in Kubernetes. I finally learned how to use kubectl to look at the logs, show the container information and events and so forth. I finally came to realize that the container we were pulling from jFrog was running a python script that was failing because someone had hard-coded IP addresses into it that weren't in use.

I decided to build a new container and fix this problem. I had to go to jFrog to learn that Docker containers are built in "Layers". Then, I decided I had to build a new container from scratch to fix this problem.

Doing all of this with no Docker knowledge means that I am essentially going ground up with Docker 101.

Thankfully we have this new guy we hired who is teaching me Docker (and some Kubernetes). Which is cool. It's a good two way exchange. I teach him about Networking and SD-WAN, and OpenStack/OpenBaton, and he teaches me about Docker and Kubernetes.