What on Earth is Canonical Doing with Ubuntu? Sheez

I have been using CentOS almost exclusively since I have been working here, first with CentOS 6, and then (now) CentOS7. I have watched the kernels move up from 3.x to 4.x, I have fought with (and still fight with) NetworkManager, etc.

You get used to what you use.

I have also used Ubuntu in the past, 14.04, and 16.04, but it has been a while.

So, I needed to install Ubuntu in order to run OSM, because OSM (on their website at least) wants to use Ubuntu. I think Ubuntu is probably bigger in Europe, is my guess.

So - for two straight days now, I have been trying to install a Ubuntu Cloud image and get it working on a KVM system. Something that SHOULD be simple, right? Wrong.

Here is a rundown of all of the headaches I have run into thus far, which has pissed me off about Ubuntu.

1. On 16.04, the root file system is only 2G for the cloud image you download off the web.

I ran out of disk space in no time flat. Trying to install X Windows and a Display Manager, which by default are not enabled on the cloud image. 

Trying to increase the root file system? Damn near impossible. I tried using qemu-img --resize, and that only created a /vdb file system. The ./dev./sda1 was STILL, and REMAINED, 2G. I could not install X Windows, I couldn't do squat. I am sure if I rolled up my sleeves, and got to work using advanced partitioning tools and whatnot, I could have made this happen. Or, maybe not. Point is, this was a hassle. An unnecessary hassle in my opinion.

2. I realized that the 18.04 Ubuntu uses a qcow2 format - which you CAN resize. Again, why Ubuntu is using 2G as a root file system size is beyond me, and this is ANNOYING. This is the year 2019.

So, I resized the image, and put a password on the image (cloud images are not set up to log in with prompt, only certificates, which of course is a good practice, albeit a hassle for what I needed).

3. I launched 18.04 and guess what? NO NETWORKING!!!! @^%$

I realize no networking was set up. At all! WHAT???

4. Let's go set up networking. Yikes! You CAN'T!!!!!! WHY? Because the iproute2 packages and legacy packages that everyone in the WORLD uses, are not on the machine!

They want you to use this newfangled tool called NetPlan to set up your networking!?!?

Fine. You can google this and set it up, which I did.

BUT WHY ARE ALL OF THESE LINUX DISTRIBUTIONS BECOMING SO DIFFERENT?

THAT IS NOT, I SAY...NOT...WHAT LINUX IS ALL ABOUT?????

I remember when Gentoo came out, and how different a beast it was. Now, the distinction between CentOS and Ubuntu is becoming a very wide chasm.

ONAP - Just Too Big and Heavy?

I have been working with Service Orchestrators for a while now. Here are three of them I have had experience with:

• Heat - which is an OpenStack Project, so while OpenStack can be considered the VIM (Virtual Infrastructure Manager), Heat is an Orchestrator that runs on top of OpenStack and allows you to deploy and manage services 
• Open Baton - this was the ORIGINAL Reference Implementation for the ETSI MANO standards, out of a Think Tank in Germany (Frauenhofer Fokus).  
• ADVA Ensemble - which is an ETSI-based Orchestrator that is not in the public domain. It is the IPR of ADVA Optical Networks, based out of Germany.

There are a few new Open Source initiatives that have surpassed Open Baton for sure, and probably Heat also. Here are a few of the more popular open source ones:

• ONAP - a Tier 1 carrier solution, backed by the likes of at&t. 
• OSM - I have not examined this one fully. TODO: Update this entry when I do.
• Cloudify - a private commercial implementation that bills itself as being more lightweight than the ONAP solution.

I looked at ONAP today. Some initial YouTube presentations were completely inadequate for allowing me to "get started". One was a presentation by an at&t Vice President. Another was done by some architect who didn't show a single slide on the video (the camera was trained on the speaker the whole time).

This led me to do some digging around. I found THIS site: Setting up ONAP

Well, if you scroll down to the bottom of this, here is your "footprint" - meaning, your System Requirements, to install this.

ONAP System Requirements

Okay. This is for a Full Installation, I guess. The 3Tb of Disk is not that bad. You can put a NAS out there and achieve that, no problem.  But 148 VCPU????  THREE HUNDRED THIRTY SIX  Gig of RAM? OMG - That is a deal killer in terms of being able to install this in a lab here. 

I can go through and see if I can pare this down, but I have a feeling that I cannot install ONAP. This is a toy for big boys, who have huge servers and lots of dinero.

I might have to go over and look at OSM to see if that is more my size.

I will say that the underlying technologies include Ubuntu, OpenStack, Docker and Mysql - which are pretty mainline mainstream.

RUST Programming Language - Part II

I spent a few hours with RUST again yesterday.

There's definitely some things to learn with this language.

One thing I noticed, was that the Compiler is really good. Very very intelligent. It can make a lot of intelligent inferences about what you are doing (or trying to do). It will also bark out all kinds of warnings about things.

Cargo
In my previous post, I may have made light mention of something called Cargo. Cargo is a "Configuration Management" facility for RUST programmers. It can build (cargo build), check (cargo check), or build AND execute (cargo run) your program.

It also manages packages, and dependencies. So I think it probably is somewhat synonymous with pip in Python. If you are familiar with yum or some equivalent package manager on a Linux distribution, you can get the gist of what Cargo does from the perspective of pulling in packages you need for your project.

This link is a book on Cargo:  The Cargo Book

So yesterday, I wrote some code from the book I have been using, but deviated the code a little bit, and pulled in a package called strum, which allows you to iterate over an "Enum" object. My Enum object has Coins and Coin values (penny, nickel, dime, quarter) and I would use strum to iterator over this and print out the monetary value of each coin. Nothing super sophisticated, but in this RUST language, you definitely need to learn how to do the basics first.

Match Expression
Another interesting thing is that you can use simple "if / then" logic, but you can also use this more sophisticated "match" expression, or construct. So this is the "higher IQ" way to do matching, for the more advanced or off the beaten path cases (i.e. regular expression searches, etc).

Here is a link on that, which is actually a relative link to a more comprehensive book on RUST that has a lot more good stuff in it than just the Match expression.

https://doc.rust-lang.org/reference/expressions/match-expr.html

RUST Programming Language - Part I

In hearing about this "new" language, I have spent some time this week going through a book called "The Rust Programming Language", which can be found at this link:

The Rust Programming Language

I will have to come back and edit this post as I go through the book, but so far, I have been trying to learn "enough" about the language to understand WHY the language has even emerged - in other words, what it's selling point is, what deficiencies in other languages it addresses, etc.

What do I have to say right now?

Well, it's a Compiled language. It's been a long time coming that I have seen a new compiled language emerge. We have had nothing but runtime interpreted languages for many years now.

It has some interesting resemblances to C.

It has no advanced traditional object oriented capabilities, like Inheritance and Polymorphism. That said, it does support Encapsulation, and this concept of Traits (which up to now, seem to me to resemble Templates in C++ a bit - but I may revise this statement after I get more familiar with it).

The language is designed to be more "safe" than traditional C or C++, the latter of which, due to direct memory access and manipulation, can cause Segmentation Violations, etc. One example of course is References where one thread might de-reference a pointer that another thread might be using, accessing, manipulating, etc.

Q-In-Q Tunneling and VLAN Translation

I have been working on this Customer Premise Edge (CPE) project, in which a Service Orchestrator deploys Virtual Network Functions (VNFs) to a "piece of hardware". In the specific implementation I have been working with, the CPE runs a FastPath DataSwitch, and 3-4 Docker containers that in turn run:

• OpenStack (Compute Node in one container, Controller Node in another)
• VRFs
• a FastPath Data Switch

The architecture looks, essentially, as shown below:

Two Customer Premise Edge Devices Connecting over a Secure Transport Overlay Network (SD-WAN)

This architecture relies on Layer 2 (Ethernet Frame) Forwarding. So what happens, essentially, is that when a virtual network is created, a "service" is generated at runtime, which connects two "interface ports" (can be L3, L2, et al). But because an interface is a physical (i.e. Operating System managed) device, traffic is run through a virtual concept called a "service port" as it comes off the wire. Depending on what kind of service it is, there are different topologies, and rulesets that can (and must) be configured and applied to make traffic handling and flows work properly.

I was not altogether very interested in this concept of "match sets" that were required to configure these services - initially. I just keyed in what I was told (which was an asterisk to allow all traffic).

But, finally, I became more interested in a deep-dive on these rules. I noticed that there was a rule to configure "inner VLAN" and "outer VLAN" settings. Huh? What does that mean? A VLAN is a VLAN, right? Well, sort of. Kind of. Not exactly.

As it turns out, in order to handle multi-tenant traffic (a service provider managing multiple customers), VLANs can overlap. And you cannot have Customer A's traffic going to Customer B, or you will find yourself out of business pretty quickly (possibly with a liability lawsuit).

So - they came up with concepts like Q-in-Q Tunneling and VLAN Translation to deal with these problems. Now, you can have a Customer VLAN (C-VLAN), and a Service Provider VLAN (S-VLAN), and you can map and manage the packets based on S-VLANS without manipulating and or disturbing the original customer VLAN that is set on the frame.

So NOW - I understand why these match set rules have fields for an "inner" and an "outer" VLAN. 

Just to be thorough, the outer VLAN, by the way, is the S-VLAN (therefore the inner VLAN is the C-VLAN).

Credit for this explanation and understanding goes to this link (although there are probably numerous sources for this concept available on the world wide web):

Q-in-Q Tunneling and VLAN Translation

Palo Alto Firewall VM Series - Integration and Evaluation - Part IV

Okay this is probably a final post on the Palo Alto Firewall VM Series Integration project.

This project was aimed at doing a proof of concept reference implementation of a fully integrated and provisioned SD-WAN, using a Palo-Alto Virtual Firewall to protect the edges of two campus networks (e.g. East and West).

It uses a Service Orchestration software platform (Service Orchestrator) to model and specify services (networks) that the Orchestrator will deploy, out to the field, to the CPE, at the push of a button.

What are we deploying?

We are deploying overlay networks (SD-WAN) to Customer Premise Edge (CPE) devices. These devices are typically deployed smack on the edge of a premise network, and for that reason have optical SFP termination ports, and Gigabit (or 10 Gig in some cases) ports.

Now to make these overlay networks work in these modern times, we rely on Virtualization. Virtual Machines, Containers (i.e. Docker), OpenStack, et al. (depending on the CPE and software chosen).

Network Function Virtualization (NFV) - meaning non-iron-based network functions like Routing, Switching, Network Stacks in Namespaces, and Overlay networks (i.e. secure overlay) - ties all of this together. NFV is a fairly recent technology and has made networking (as if it weren't complicated enough already) a bit MORE complicated. But virtualizing applications and processes without also virtualizing the networking the rely on to intercommunicate doesn't achieve the objectives that everyone is after by using virtualization in the first place. It is like going halfway and stopping.

So - we have been deploying this new VM-Series Palo Alto Firewall in a network "topology" in an automated fashion, using the Palo Alto API to provision all of the "things that need to be provisioned", a few of which are mentioned below.

• Interfaces
• Zones
• Security Policies
• Static Routes
• Policy-Based Forwarding Rules

This is done, by using the Palo-Alto XML API.

So to do this, at least in this proof of concept, we provision:

1. A Palo-Alto Firewall
2. A virtual machine (temporary) that provisions the Palo-Alto Firewall
3. A SD-WAN Gateway behind the Palo-Alto Firewall
4. A client or server behind the Gateway.

Topology Provisioned on a CPE device - Site West
screenshot: ADVA Ensemble Orchestrator

Sounds easy, right? It's not. Here are just a few of the challenges you face:

• Timing and Order
• The Firewall needs to be deployed first, naturally. Otherwise you cannot provision it.
• The Firewall Provisioner configures all of the interfaces, routes, zones, policies and rules. This is done on a management network - separate from the service plane, or traffic plane as it is often referred to.
• With a properly configured firewall in place, the Gateway can make outbound calls to the Cloud.
• Once the Gateway is up and cloud-connected, the server can begin listening for and responding to requests.
• Routing
• Dynamic routing stacks are difficult to understand, set up and provision; especially BGP. Trying to do all of this "auto-magically" at the push of a button is even more challenging.
• In our case, we used static routes, and it is important to know how these routes need to be established. 
• Rules
• Indeed, it is sometimes difficult to understand why traffic is not flowing properly, or at all. Often is due to routing. It can also be due to not having proper rules configured.
• Programmatic APIs
• Making API calls is software development. So the code has to be written. It has to be tested. And re-tested. And re-tested.

When you deploy a network push-button, the visual ease at which it all happens starts to make people take it for granted, and NOT fully appreciate all of the complexity and under-the-hood inner workings that make it all come to light.

Most of the network troubleshooting (or all of it perhaps) had to do with missing or incorrect routes, or missing rules. It can be a chicken or egg problem trying to figure out which one is a culprit.

In this particular CPE, it runs:

• an OS (Linux)
• two docker containers - that contain OpenStack nodes (Controller node and Compute node)
• a FastPath Data Switch - that map physical ports to virtual ports 

The "magic" is in the automation and integration. The data switch is pre-provisioned and custom developed for performance. The integration with OpenStack allows virtual ports and virtual switch services to be created / removed as new Virtualized Network Functions (VNFs) are initialized or terminated on demand.

Then, on top of this, you have the VNFs themselves; in this case here, the Palo Alto Firewall, the Edge Gateway behind this firewall, and a Server on a trusted network behind the gateway.

Traffic from the server will run through the firewall for inspection before it gets to the Gateway so that the firewall can inspect that traffic before the gateway encrypts it for outbound transport.

Then, the Gateway will encrypt and send the traffic back through the firewall, and out the door, on a virtual circuit (or multiple virtual circuits if multi-pathing is used), to the complementary gateway on the other side (think East to West across two corporate campus networks here).