HAProxy Problem Solved

Okay, an update on my latest post on HAProxy. It turns out that the issue had nothing to do with HAProxy at all, but that the clustering on the back-ends was not working properly.

So, I have reverted the HAProxy back to its original configuration, prior to getting into debug mode. 

Note: The Stats page in HAProxy, by the way, is an invaluable way to help see if your proxy is working correctly. You can see the load distribution, you can see the back-end health checks, etc. Plus the GUI is simple, informative and intuitive.

Below is a quick look at the front and back end configurations. In this configuration, we are using HAProxy as a Reverse Proxy, where we terminate incoming requests (SSL Termination), and rather than forward to our back-ends in the clear, we utilize back-end certificates to proxy to the back-ends using SSL. This is a bit unconventional perhaps in a typical reverse proxy scenario.

#---------------------------------------------------------------------
# ssl frontend
#---------------------------------------------------------------------
frontend sslfront
   mode tcp
   option tcplog
   bind *:443 ssl crt /etc/haproxy/cert/yourcert.pem
   default_backend sslback

#---------------------------------------------------------------------
# sticky load balancing to back-end nodes based on source ip.
#---------------------------------------------------------------------
backend sslback
   # redirects http requests to https which makes site https-only.
   #redirect scheme https if !{ ssl_fc }
   mode http
   balance roundrobin
   option ssl-hello-chk

   option httpchk GET /ping
   http-check expect string TESTPING

   stick-table type ip size 30k expire 30m
   stick on src
   #stick-table type binary len 32 size 30k expire 30m
   #stick on ssl_fc_session_id
   default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions

   server servername01 192.168.20.10:443 ssl check id 1 maxconn 1024
   server servername02 192.168.20.11:443 ssl check id 2 maxconn 1024
   server servername03 192.168.20.12:443 ssl check id 3 maxconn 1024

HAProxy - Aggravating Problem I Have Not Solved

I have not ever really blogged on proxies. I don't have a lot of proxy experience and don't consider myself a guru with proxies, load balancers, etc.

But more and more often, solutions have come in that require load distribution to an N+1 (Active Active) cluster. And, HAProxy is supposed to be a rather lightweight and simple approach, especially in situations where the mission is not totally critical, or the load is not seriously high.

I originally set HAProxy up to distribute load to a Cloudify cluster. And Cloudify provided the configuration for HAProxy that they had tested in their lab, and that they knew worked well. Later, I set HAProxy up to load balance our Morpheus cluster. Initially it was working fine. 

Or, so it seemed. Later, I noticed errors. The first thing you generally do when you see errors, is to tell HAProxy to use one node (and not 2 or 3), so that you can reduce troubleshooting complexity and examine the logs on just one back-end node.  So in doing this, I managed to rather quickly figure out that if I told HAProxy to use one back-end node, things worked fine. When I told HAProxy to use two or more back-end nodes, things didn't work.  

So that's where it all started.

The Problem
Below is a picture of what we are doing with HAProxy, and based on the picture below, web access comes in on the northbound side of the picture, and web access is not the problem we are having.  The problem, is that VMs that are deployed onto various internal networks by Morpheus "phone home" and they phone home on a different network interface. 

This works fine with a single back-end enabled. But if you enable more than one back-end in HAProxy, Morpheus fails to fully transition the state of the VM to "running".

In testing this out a bit and dumping traffic, we initially noticed something interesting. The Source IP coming into each Morpheus node, was not the HAProxy VIP - it was the interface IP address. We wound up solving this, by telling KeepAliveD to delete and re-create the routes with the VIP to be used as the Source IP - but only when it had control of the VIP. But in the end, while this made traffic analysis (tcpdump on the Morpheus nodes) a bit more clear about the traffic flow, it did not solve the actual issue.

I STILL don't know why it works with one back-end, and not two or more. I have had Proxy experts in our organization come in and look, and they seem to think HAProxy is doing its job properly, and that the issue is on the back-end clustering. The vendor, however, is telling us the issue is with HAProxy.

Our next step may be to configure a different load balancer. That should definitely rule things out. I know Squid Proxy is quite popular, but these tools do have a Learning Curve, and I have zero zilch experience with Squid Proxy. I think we may use a Netscaler Load Balancer if we wind up going with another one.

I should mention that the HAProxy configuration is not the simplest. And as a result of configuring this, I have increased my general knowledge on Load Balancing.