Palo Alto Firewall VM Series - Integration and Evaluation - Part IV

Okay this is probably a final post on the Palo Alto Firewall VM Series Integration project.

This project was aimed at doing a proof of concept reference implementation of a fully integrated and provisioned SD-WAN, using a Palo-Alto Virtual Firewall to protect the edges of two campus networks (e.g. East and West).

It uses a Service Orchestration software platform (Service Orchestrator) to model and specify services (networks) that the Orchestrator will deploy, out to the field, to the CPE, at the push of a button.

What are we deploying?

We are deploying overlay networks (SD-WAN) to Customer Premise Edge (CPE) devices. These devices are typically deployed smack on the edge of a premise network, and for that reason have optical SFP termination ports, and Gigabit (or 10 Gig in some cases) ports.

Now to make these overlay networks work in these modern times, we rely on Virtualization. Virtual Machines, Containers (i.e. Docker), OpenStack, et al. (depending on the CPE and software chosen).

Network Function Virtualization (NFV) - meaning non-iron-based network functions like Routing, Switching, Network Stacks in Namespaces, and Overlay networks (i.e. secure overlay) - ties all of this together. NFV is a fairly recent technology and has made networking (as if it weren't complicated enough already) a bit MORE complicated. But virtualizing applications and processes without also virtualizing the networking the rely on to intercommunicate doesn't achieve the objectives that everyone is after by using virtualization in the first place. It is like going halfway and stopping.

So - we have been deploying this new VM-Series Palo Alto Firewall in a network "topology" in an automated fashion, using the Palo Alto API to provision all of the "things that need to be provisioned", a few of which are mentioned below.

• Interfaces
• Zones
• Security Policies
• Static Routes
• Policy-Based Forwarding Rules

This is done, by using the Palo-Alto XML API.

So to do this, at least in this proof of concept, we provision:

1. A Palo-Alto Firewall
2. A virtual machine (temporary) that provisions the Palo-Alto Firewall
3. A SD-WAN Gateway behind the Palo-Alto Firewall
4. A client or server behind the Gateway.

Topology Provisioned on a CPE device - Site West
screenshot: ADVA Ensemble Orchestrator

Sounds easy, right? It's not. Here are just a few of the challenges you face:

• Timing and Order
• The Firewall needs to be deployed first, naturally. Otherwise you cannot provision it.
• The Firewall Provisioner configures all of the interfaces, routes, zones, policies and rules. This is done on a management network - separate from the service plane, or traffic plane as it is often referred to.
• With a properly configured firewall in place, the Gateway can make outbound calls to the Cloud.
• Once the Gateway is up and cloud-connected, the server can begin listening for and responding to requests.
• Routing
• Dynamic routing stacks are difficult to understand, set up and provision; especially BGP. Trying to do all of this "auto-magically" at the push of a button is even more challenging.
• In our case, we used static routes, and it is important to know how these routes need to be established. 
• Rules
• Indeed, it is sometimes difficult to understand why traffic is not flowing properly, or at all. Often is due to routing. It can also be due to not having proper rules configured.
• Programmatic APIs
• Making API calls is software development. So the code has to be written. It has to be tested. And re-tested. And re-tested.

When you deploy a network push-button, the visual ease at which it all happens starts to make people take it for granted, and NOT fully appreciate all of the complexity and under-the-hood inner workings that make it all come to light.

Most of the network troubleshooting (or all of it perhaps) had to do with missing or incorrect routes, or missing rules. It can be a chicken or egg problem trying to figure out which one is a culprit.

In this particular CPE, it runs:

• an OS (Linux)
• two docker containers - that contain OpenStack nodes (Controller node and Compute node)
• a FastPath Data Switch - that map physical ports to virtual ports 

The "magic" is in the automation and integration. The data switch is pre-provisioned and custom developed for performance. The integration with OpenStack allows virtual ports and virtual switch services to be created / removed as new Virtualized Network Functions (VNFs) are initialized or terminated on demand.

Then, on top of this, you have the VNFs themselves; in this case here, the Palo Alto Firewall, the Edge Gateway behind this firewall, and a Server on a trusted network behind the gateway.

Traffic from the server will run through the firewall for inspection before it gets to the Gateway so that the firewall can inspect that traffic before the gateway encrypts it for outbound transport.

Then, the Gateway will encrypt and send the traffic back through the firewall, and out the door, on a virtual circuit (or multiple virtual circuits if multi-pathing is used), to the complementary gateway on the other side (think East to West across two corporate campus networks here).

Palo Alto Firewall VM Series - Integration and Evaluation - Part III

Okay, this is just a short post to discuss where we are in the integration process.

1. I have a Python script that generates XML. In this script, you pass in parameters, and then I use the ETree library in Python to generate the XML.
2. I have some bash scripts that take the XML files, and invoke the Python XML API Wrapper, which in turn does the legwork to send the data to the API Server on the Firewall.

Normally one might create the Management Profile, Zones and Security Policies first. And THEN add or assign interfaces, routers, routes on those routers, etc.

This is the basic process I am following thus far:

1. Create Management Profile
2. Load Interface(s) - the management profile in #1 is included.
3. Create Zone(s)
4. Create Security Policies - the interfaces included
5. Assign interface to default router
6. Load Static Route on default router - include interface

Seems to be working okay although the process needs to be tightened up a bit so that you are not using one Python program to generate the xml, and another to call the API. 

But it's good enough to load and test and see if I can get a firewall operational.

Palo Alto Firewall VM Series - Integration and Evaluation - Part II

After a couple of days of playing around with the Palo Alto VM-Series Firewall (running the VM on a KVM / LibvirtD virtualization platform on a CentOS7 host), I felt I was comfortable enough with it to explore the API.

I asked a Palo Alto engineer how they bootstrap these things. He told me they use CloudInit and use a boot.xml file to change the default password. From there, they use their management platform, Panorama, to push configurations to the devices.

I don't happen to have Panorama anywhere. And I presume like everything else, it needs licenses. So, I started looking at the facilities to interface/integrate with the device; meaning APIs.

There are actually several APIs:

• Command Line Interface (CLI)
• WildFire API
• AutoFocus API
• PAN-OS Licensing API
• Panorama XML API (requires Panorama of course)
• Pan-OS XML API

I located, downloaded and glanced through the XML API Guide. Which actually does do a nice job of getting you acquainted with the API. There is nothing really unusual. You need to authenticate, get a token (they call it a key), and with that key you can go to work (I won't cover details of the API here).

Next it was time to examine the API firsthand. Is it running? Do I need a license? I used Postman for this. I don't know if there are other better tools for picking at APIs, but I think Postman is definitely one of those most popular tools. Making add/modify changes is always risky when you are learning a new API, so it always makes sense to start with some "get" calls so you can understand the structure of the data. So, I was able to hit the VM on standard SSL port 443, and get back a key, and with the key, run a few get commands based on examples in the API Guide. The API works, it appears!

One noteworthy comment is that the API would not work without turning off certificate validation in the settings!

Next, I considered starting to write some Python code as a client, but as Palo Alto is a pretty popular firewall from a large company, there had to be some folks who have broken ground on that already, right? A quick google search for a Python API client turned up a project from a guy named Kevin Steves, who has clients for ALL of the APIs in Python. It is on GitHub with a free use license.

https://github.com/PaloAltoNetworks/pandevice/

After cloning this, I noticed you can run setup. I elected not to run setup, and just try to invoke the API directly. I had to use the panxapi.py python file. Examining the source code, you can supply an exhaustive list of options to the main() module of the Python file, which will parse those and invoke accordingly.

Immediately, however, I ran into the same certificate validation error I experienced with PostMan. But in PostMan I could just go into settings and disable certificate validation. Figuring out how to do this with the API was more difficult. Eventually, I found an issue recorded on the project that discusses this same problem, which can be found at this link:  Certificate Validation Issue

The issue discusses versions of Python on CentOS that do certificate checking. Rather than fool with upgrading Python, one poster pointed out that you can, in fact, disable certificate checking in Python by setting an environment variable: "export PYTHONHTTPSVERIFY=0". Bingo. That's all I need right now to experiment with the API.

Palo Alto Firewall VM Series - Integration and Evaluation - Part I

This week, I am evaluating the Palo Alto VM-Series Firewall.

I will ramble a bit about what I am doing, learning, etc.

First off, this VM-Series Firewall can run as a qcow2 image on KVM, it can load as an OVF onto the VMWare vSphere ESXi platform, and I have seen some evidence of people loading it on VirtualBox also. I am using it on a KVM (libvirtd) host.

The main thing about a virtual firewall appliance is how it plumbs into the virtual networking, using virtual adaptors, or host NICs.

One the VM I just installed, I set up 4 Adaptors.

4 NICs on the Palo Alto VM

If we assume 4 NICs on the virtual machine, the very first NIC is designated as the management NIC.  What is confusing about this, is that you might expect this NIC to show up in the list of interfaces. It doesn't. You have to "know" that the first NIC is a management NIC that does NOT show up in the list of Interfaces.

If we look at the screenshot below, you will see a Management IP Address on a 172.22.0.0/24 network. This is shown on the "Dashboard" tab of the Palo Alto user interface.

The Management Interface connects to a VM NIC that does not show up in the list of Interfaces

Yet, if we look at the list of Interfaces (Network tab), we will see that this Management Interface (first NIC in the KVM list of adaptors) does not exist.

I would have to go back and see how well this is documented, for I admittedly dive in without reading documentation sometimes. But it was NOT very intuitive. I would prefer for the interfaces to "line up" with the VM adaptors, and if one is a Management Interface, perhaps it is greyed out, or managed in a unique way.

I understand why Palo Alto did this, for the Management Interface is generally considered a unique pipe into the product - used for administration of the device itself and generally is not considered part of a traffic plane interface. But it did make it difficult at first for me because I did not know which bridge the first Interface was indeed connected to - the br0 (which has a 172.22.0.0/24 network on it), or br1 (which has a 172.21.0.0/24 network on it).

Palo Alto, like FirewallD, is a zone-based firewall. So while you may have a tendency to get fixated on your Interfaces (trying to get them to light up green) initially, the first thing you really SHOULD do is a bit of forethought and planning, and create your Zones.

I created two zones (currently):

• Trusted L3
• UnTrusted L3

Two Zones Initially Created - L3 Trusted, and L3 Untrusted

The Untrusted Zone contains the interface Ethernet1/1, which is connected to a host adaptor via a bridge. For this reason I considered this interface to be untrusted, as per my thinking, it would connect up to my Router much like a Firewall at the edge of a premise might connect to an ISP.

The Trusted Zone contains two interfaces Ethernet1/2 and Ethernet1/3. 

Ethernet1/2 is mapped to an adaptor on the virtual machine that is connected to the "default" KVM network, which has CIDR of 192.168.122.0/24. But - this is a NAT network! Packets that leave the KVM host are Source NAT'ed. How this works with the Firewall? I don't know yet - I have not tested extensively with this "type" of network.

Ethernet1/3 is mapped to an adaptor on the virtual machine that is connected to an Internal KVM network which has a CIDR of 192.168.124.0/24. This network, though is NOT a NAT network. It is a Routed Network. A Routed Network is routed between KVM Internal networks, but generally is not reachable outside the KVM host, because KVM creates iptables rules that drop inbound packets coming from any host aside of the KVM host itself (I tested this - pings from another host cannot reach the 192.168.122.0/24 network because they get dropped on a FORWARD chain rule). I suppose theoretically, if you hacked the iptables rules properly on the KVM host, these KVM internal networks could be reachable. Maybe there is a facility for this designed to accommodate strange circumventions like these, but messing with the iptables rules, and especially the order of such rules, are prone to issues.

So in summary, it does appear that the Palo Alto Firewall VM-Series, on KVM, will work with KVM Internal networks in a respectful way. You would just want to classify these as "Trusted" networks when it comes to zones and security policies.

Hairpin NAT

A lot of folks don't understand Hairpin NAT, meaning what it is, why it exists, or the specific use cases in which it applies.

This is an awesome site that explains it nicely - although you have to read the very very last paragraph to get to the bottom of it:

Hairpin NAT Explained