VRRP Testing Round 3

Today, tested the concept of a Synch Group.

Used the configuration (essentially) from this ubuntu manpage.
http://manpages.ubuntu.com/manpages/precise/man5/keepalived.conf.5.html

Worked very well on VirtualBox, where we connected and disconnected virtual cables and watched the IPs move, in tandem.

Using just two instances if you disconnect one interface on each, VRRP considers itself in a complete fault state and neither box gets a virtual IP on either interface - which is what you want if it's a synch group.

Pretty happy with this testing.

BTW...another site I found had some decent comments about some options:
https://linux.die.net/man/5/keepalived.conf

Multicast - Low Level How it Works

http://www.firewall.cx/networking-topics/general-networking/107-network-multicast.html

VRRP Testing Note 2

In doing some extensive testing with VRRP with Keepalived, I was able to put it into Split Brain mode.

I have documented how I did this, and I will need to see if it is repeatable.

Here is one link that discusses troubleshooting for this, although I will need to find others.

https://www.atlantic.net/community/howto/vrrp-keepalived-configuration/

UPDATE: Here is another.

http://serverfault.com/questions/512153/both-servers-running-keepalived-become-master-and-have-a-same-virtual-ip

In his case on this link, here is the solution.

The problem has been resolved.

The problem was from the switch setting. When multicast filter mode is filter-all, the problem happened. The Keepalived runs O.K. when multicast filter mode is forward-all.

In his specific case, unicast could be the answer.

I realize that I do not have that mcast src ip in my file (don't think that is needed but I can add it).

I could also try to use PASS instead of AH. So I can experiment with that as well.

VRRP Testing - Note 1

First problem I ran into was with SeLinux. Apparently when the notify script wanted to write to an output file, SeLinux complained and prevented this. This made it impossible to tell if the script was even working, and therefore made it very hard to tell if VRRP / Keepalived was working.

I got around this by changing SeLinux to permissive in the /etc/selinux/conf file.

With permissive, it still complains, but does not prevent.


Next, I came to realize that the configuration parameters that were being used to start keepalived were in the /etc/sysconfig/keepalived folder.

I changed these parms to use -P -D -d -S 7.

The -P is for VRRP Only option.
-D is for Debug
-d is for dump config
-S is for log level, which is set to 7 for maximum logging.


With these changes, I noticed that BOTH virtual machines had a virtual IP, instead of just one. This surprised me. I expected only one to use the virtual IP and that it would switch back and forth.

But, doing some initial testing by disconnecting adaptors, I could see the state change being logged out to the log file.

VRRP with Unicast

Some good examples of how to do keepalived with unicast:

http://www.linux-admins.net/2015/02/keepalived-using-unicast-track-and.html

VRRP with Keepalived

First, the website for keepalived:

http://www.keepalived.org/

Originally, I read the documentation on their website, which is dated June 13th, 2002.

http://www.keepalived.org/pdf/UserGuide.pdf

Unable to believe that the documentation could not be changed after this many years, I went digging for better more updated documentation, and indeed, I found that here:

https://media.readthedocs.org/pdf/keepalived/latest/keepalived.pdf

WHY DOES THIS NOT LIVE ON THE WEBSITE????

I started reading. This is not simple / trivial.

Keepalived is essentially used mainly for load balancing, and when you read up on it, it is inferred that one needs to have LVS (Linux Virtual Server). There are quite a number of sites discussing how to build a Load Balanced with LVS, like this one below.

http://www.linux-admins.net/2013/01/building-load-balancer-with-lvs-linux.html

But there did (and still does not) seem to be much documentation discussing what you need to do, exactly, to install and run LVS. For instances, does it come by default on systems? Do you need to install it? Compile it?

One article, albeit from 2004, scared me when I started reading about patching the kernel to use it.

http://www.ultramonkey.org/papers/lvs_tutorial/html/

Later, I found this link, which, FINALLY, makes it clear that you need the package ipvsadm (on CentOS, which is the OS I am interested in).

NOTE: Maybe these are different services and packages on other Linux distributions.

https://www.server-world.info/en/note?os=CentOS_7&p=lvs

Next, I started reading further, and realized that, in order to use VRRP, you don't even need to use LVS!!! WHY? Because VRRP is only used to maintain state between Master and Slaves on Load Balancers, so that a) someone is always on top of the load balancing task and b) the Load Balancers know who the master is, and who the slave is. It is also used to prevent Split-Brain mode (multiple masters, etc).

So - why would you be interested in VRRP, without the Load Balancing? That does not seem to make much sense? And that is why these concepts are so intertwined. But in our case, admittedly a rare one, that is our requirement. We just want VRRP for the purpose of making sure a node is always up; we don't need the nodes running VRRP to load balance anything behind them.

So I found these sites below that, rather than address LVS (and using KeepaliveD for LVS and Load Balancing), and instead focus on the VRRP aspects of KeepaliveD.

https://tobrunet.ch/2013/07/keepalived-check-and-notify-scripts/

http://packetpushers.net/vrrp-linux-using-keepalived-2/

This site from Oracle helps provide some guidance on the tracking features of keepalived (scripts, interfaces, et al). It finally explained what interface tracking is for / does.

https://docs.oracle.com/cd/E37670_01/E41138/html/section_hxz_zdw_pr.html

VirtualBox Networking Configuration

VirtualBox has its own way of Networking that is a bit different than other virtualization platforms.

It's important to understand the distinctions between the types and kinds of networks that are out there and available.

VirtualBox Networking Chapter of Manual:
https://www.virtualbox.org/manual/ch06.html

VirtualBox Manual Itself:
http://www.virtualbox.org/manual/

Note: There is a pesky NIC that shows up in "ifconfig", called virbr0. This is a virtual network driver that shows up if you are running libvirtd. You will need to stop and disable libvirtd if this interface is resident, and often times you will need to reboot the box or VM after you do this for the interface to completely disappear from the ifconfig interface listing.