Version Control Systems

I used to use Clearcase and CVS for version control "back in the day" (way back in the day actually).

Some shops use Subversion now, which is referred to these days in the era of shortcuts and short attention spans, as SVN.

This link here - an analysis - shows some differences between CVS and SVN compared:
http://www.pushok.com/software/svn-vscvs.html


Moving on to GIT, here is a link where fundamental differences between GIT and SVN are discussed.

http://boxysystems.com/index.php/5-fundamental-differences-between-git-svn/

It's been a long time since I have done version control, or had a need to. So this is some information that can be helpful to get back "up to speed".

The TTY Demystified - great article

Great background article. Interesting for anyone who wants to know the history.

http://www.linusakesson.net/programming/tty/

Some more interesting discussions.

https://news.ycombinator.com/item?id=10631513

Strict, Moderate, Open NAT

This came up yesterday. We'll take a note on the distinctions.

http://serverfault.com/questions/208522/what-is-strict-moderate-and-open-nat

DD-WRT

I spent some time today dissecting DD-WRT, which is an open source operating system used on many routers and embedded devices (that use Wireless technologies, especially 802.x).

Looks like it uses BusyBox for most of the binaries. BusyBox appears to be a single binary that can run many of the GNU / POSIX unix/linux-descended binaries. It's supposed to be more compact. For example, options have been stripped away or replaced on many of the binaries in the name of compact simplified code (lower footprint).  It is open source, so one can download, inspect, compile, etc.

For ssh, it uses something called DropBear. Again, a compact ssh server.

The webserver, httpd, looks to be an in-house developed web server (meaning, written by developers of dd-wrt). Chillispot is the access point controller.  ttraff is used as the traffic counter. dnsmasq is used as the DHCP server on the box.

All in all, the box looked very tight - with 40M used of 256M,

In some ways interesting, in other ways, uninteresting - which is a good thing for embedded devices.

Windows 10 Chatter

I wanted to take some time to do some forensic analysis on my router (this is another blog entirely).

One of the things I noticed, was how busy the router was.

Well, hey. It's a busy house. And the whole family is here this weekend, all streaming NetFlix on phones, laptops, et al.

But - I noticed an inordinate amount of traffic from one PC.

Upon doing some further investigation, using packet sniffing tools, Task Manager, et al. I realized that Microsoft was calling home for a number of reasons.

I should have looked into this more when I bought this PC, but got busy. Shame on me.  I saw all kinds of things, and starting shutting down services, etc. Finally, I after shutting down so many things, and continuing to see packets flowing, I went out to the web and found this link - which is reasonable recent.

http://arstechnica.com/information-technology/2015/08/even-when-told-not-to-windows-10-just-cant-stop-talking-to-microsoft/?comments=1

I also read the comments, and wound up installing DisableWinTracking. I downloaded the source code first (as a tar file), and looked it over. Deciding it looked clean, I went ahead and ran the program, which is a series of powershell scripts that disable the tracking. There's a considerable amount of it. I un-installed all of the Microsoft apps listed, and disabled the services (I did not delete them - just disabled them). You have to be a little worried about shutting off your security updates on Windows 10, which is something I will need to monitor.  But a lot of that crap, I agreed with and decided was not really in my interests at all, and in fact was sucking up bandwidth and tying up an already-busy router.

Now, I'll go back and start looking at traffic again.

Firewall HA with conntrackd and keepalived

As we are doing some keepalived work, and looking at groups of IPs, I came across a couple of links worth mentioning:

This link below shows an approach of using a vrrp_sync group for an internal and external network. This is very much along the lines of what we were thinking we wanted to achieve and it was nice to see reassurance that we might be using a sync group as it is intended to be used (given the dirth of use-case documentation on this).

http://manpages.ubuntu.com/manpages/precise/man5/keepalived.conf.5.html

Another link is this one below.

Up to this point I have only been testing the VIP assignment and reassignment by pulling cables and making firewall adjustments (to make sure the VRRP could get through the firewall).

This link below has me concerned that true client testing might (will?) fail. Because indeed, Firewalld and iptables on Linux are, indeed, stateful firewalls. So I may need this link to bail me out.

http://backreference.org/2013/04/03/firewall-ha-with-conntrackd-and-keepalived/

VRRP Testing Round 3

Today, tested the concept of a Synch Group.

Used the configuration (essentially) from this ubuntu manpage.
http://manpages.ubuntu.com/manpages/precise/man5/keepalived.conf.5.html

Worked very well on VirtualBox, where we connected and disconnected virtual cables and watched the IPs move, in tandem.

Using just two instances if you disconnect one interface on each, VRRP considers itself in a complete fault state and neither box gets a virtual IP on either interface - which is what you want if it's a synch group.

Pretty happy with this testing.

BTW...another site I found had some decent comments about some options:
https://linux.die.net/man/5/keepalived.conf

Multicast - Low Level How it Works

http://www.firewall.cx/networking-topics/general-networking/107-network-multicast.html

VRRP Testing Note 2

In doing some extensive testing with VRRP with Keepalived, I was able to put it into Split Brain mode.

I have documented how I did this, and I will need to see if it is repeatable.

Here is one link that discusses troubleshooting for this, although I will need to find others.

https://www.atlantic.net/community/howto/vrrp-keepalived-configuration/

UPDATE: Here is another.

http://serverfault.com/questions/512153/both-servers-running-keepalived-become-master-and-have-a-same-virtual-ip

In his case on this link, here is the solution.

The problem has been resolved.

The problem was from the switch setting. When multicast filter mode is filter-all, the problem happened. The Keepalived runs O.K. when multicast filter mode is forward-all.

In his specific case, unicast could be the answer.

I realize that I do not have that mcast src ip in my file (don't think that is needed but I can add it).

I could also try to use PASS instead of AH. So I can experiment with that as well.

VRRP Testing - Note 1

First problem I ran into was with SeLinux. Apparently when the notify script wanted to write to an output file, SeLinux complained and prevented this. This made it impossible to tell if the script was even working, and therefore made it very hard to tell if VRRP / Keepalived was working.

I got around this by changing SeLinux to permissive in the /etc/selinux/conf file.

With permissive, it still complains, but does not prevent.


Next, I came to realize that the configuration parameters that were being used to start keepalived were in the /etc/sysconfig/keepalived folder.

I changed these parms to use -P -D -d -S 7.

The -P is for VRRP Only option.
-D is for Debug
-d is for dump config
-S is for log level, which is set to 7 for maximum logging.


With these changes, I noticed that BOTH virtual machines had a virtual IP, instead of just one. This surprised me. I expected only one to use the virtual IP and that it would switch back and forth.

But, doing some initial testing by disconnecting adaptors, I could see the state change being logged out to the log file.

VRRP with Unicast

Some good examples of how to do keepalived with unicast:

http://www.linux-admins.net/2015/02/keepalived-using-unicast-track-and.html

VRRP with Keepalived

First, the website for keepalived:

http://www.keepalived.org/

Originally, I read the documentation on their website, which is dated June 13th, 2002.

http://www.keepalived.org/pdf/UserGuide.pdf

Unable to believe that the documentation could not be changed after this many years, I went digging for better more updated documentation, and indeed, I found that here:

https://media.readthedocs.org/pdf/keepalived/latest/keepalived.pdf

WHY DOES THIS NOT LIVE ON THE WEBSITE????

I started reading. This is not simple / trivial.

Keepalived is essentially used mainly for load balancing, and when you read up on it, it is inferred that one needs to have LVS (Linux Virtual Server). There are quite a number of sites discussing how to build a Load Balanced with LVS, like this one below.

http://www.linux-admins.net/2013/01/building-load-balancer-with-lvs-linux.html

But there did (and still does not) seem to be much documentation discussing what you need to do, exactly, to install and run LVS. For instances, does it come by default on systems? Do you need to install it? Compile it?

One article, albeit from 2004, scared me when I started reading about patching the kernel to use it.

http://www.ultramonkey.org/papers/lvs_tutorial/html/

Later, I found this link, which, FINALLY, makes it clear that you need the package ipvsadm (on CentOS, which is the OS I am interested in).

NOTE: Maybe these are different services and packages on other Linux distributions.

https://www.server-world.info/en/note?os=CentOS_7&p=lvs

Next, I started reading further, and realized that, in order to use VRRP, you don't even need to use LVS!!! WHY? Because VRRP is only used to maintain state between Master and Slaves on Load Balancers, so that a) someone is always on top of the load balancing task and b) the Load Balancers know who the master is, and who the slave is. It is also used to prevent Split-Brain mode (multiple masters, etc).

So - why would you be interested in VRRP, without the Load Balancing? That does not seem to make much sense? And that is why these concepts are so intertwined. But in our case, admittedly a rare one, that is our requirement. We just want VRRP for the purpose of making sure a node is always up; we don't need the nodes running VRRP to load balance anything behind them.

So I found these sites below that, rather than address LVS (and using KeepaliveD for LVS and Load Balancing), and instead focus on the VRRP aspects of KeepaliveD.

https://tobrunet.ch/2013/07/keepalived-check-and-notify-scripts/

http://packetpushers.net/vrrp-linux-using-keepalived-2/

This site from Oracle helps provide some guidance on the tracking features of keepalived (scripts, interfaces, et al). It finally explained what interface tracking is for / does.

https://docs.oracle.com/cd/E37670_01/E41138/html/section_hxz_zdw_pr.html

VirtualBox Networking Configuration

VirtualBox has its own way of Networking that is a bit different than other virtualization platforms.

It's important to understand the distinctions between the types and kinds of networks that are out there and available.

VirtualBox Networking Chapter of Manual:
https://www.virtualbox.org/manual/ch06.html

VirtualBox Manual Itself:
http://www.virtualbox.org/manual/

Note: There is a pesky NIC that shows up in "ifconfig", called virbr0. This is a virtual network driver that shows up if you are running libvirtd. You will need to stop and disable libvirtd if this interface is resident, and often times you will need to reboot the box or VM after you do this for the interface to completely disappear from the ifconfig interface listing.

Generating UUID for Network Interfaces

Ever wonder where these came from? How these got generated?

Ever wanted to copy an interface file, and wonder about what happens if you use the same UUID, or - perhaps worse - what happens if you blow it away, or just change it willy nilly?

Well, I came across this and tested it on VirtualBox - because VirtualBox did not generate files or UUIDs for new NAT interfaces I created. I generated new UUIDs, and nothing complained or barked at all. Yay.

I'm sure I will remember this, but I'll post it just in case.

NOTE: The uuidgen utility seemed to be on the box. I did not need to install anything. I don't know what package this utility is a part of, and have not researched. It just seemed to work - for me at least.

http://www.itechlounge.net/2014/03/linux-how-to-generate-uuid-for-network-interface-on-rhelcentos/

Source NAT on Linux

Had a request come in to try and do a source-based NAT.

The reason for this is that the customer had a ISP router that they presumably could not log in and configure to do NAT with. Behind this router, was a Switch - apparently an L3 Switch that had some intelligence, but apparently it could not do NAT. It could have actually been related to change control, also.

I found this website here, which discussed how to do SNAT.

http://www.netfilter.org/documentation/HOWTO/NAT-HOWTO-6.html

Since websites tend to die an move, I will clip the excerpt of text I was interested in:
----------------------------------------------------------------------------------------------------------
You want to do Source NAT; change the source address of connections to something different. This is done in the POSTROUTING chain, just before it is finally sent out; this is an important detail, since it means that anything else on the Linux box itself (routing, packet filtering) will see the packet unchanged. It also means that the `-o' (outgoing interface) option can be used.
Source NAT is specified using `-j SNAT', and the `--to-source' option specifies an IP address, a range of IP addresses, and an optional port or range of ports (for UDP and TCP protocols only).

## Change source addresses to 1.2.3.4.
# iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 1.2.3.4

## Change source addresses to 1.2.3.4, 1.2.3.5 or 1.2.3.6
# iptables -t nat -A POSTROUTING -o eth0 -j SNAT --to 1.2.3.4-1.2.3.6

## Change source addresses to 1.2.3.4, ports 1-1023
# iptables -t nat -A POSTROUTING -p tcp -o eth0 -j SNAT --to 1.2.3.4:1-1023

Masquerading

There is a specialized case of Source NAT called masquerading: it should only be used for dynamically-assigned IP addresses, such as standard dialups (for static IP addresses, use SNAT above).
You don't need to put in the source address explicitly with masquerading: it will use the source address of the interface the packet is going out from. But more importantly, if the link goes down, the connections (which are now lost anyway) are forgotten, meaning fewer glitches when connection comes back up with a new IP address.

## Masquerade everything out ppp0.
# iptables -t nat -A POSTROUTING -o ppp0 -j MASQUERADE

----------------------------------------------------------------------------------------------------------
Since I used FirewallD, I had to put the NAT rule into the direct.xml rule of FirewallD (I considered trying to put it in a zone-based rule, but decided to use direct.xml).

In testing this, I found some very interesting things. I did not have an IP to NAT "to", since I don't run the networks here, and everything is set up for DHCP. But - I did have two NICs on two separate networks, so I decided to SNAT the IPs of NIC A (10.1.x.y) to NIC B (172.31.x.y).

I used tcpdump to examine packets that typically came out of NIC A as 10.1.x.y, to see if they would come out as 172.31.x.y: tcpdump -A -n -i NICA grep "172.31.x.y"

This seemed to work as long as I had ONE rule for ONE Nic. But if I tried to use TWO rules for the TWO Nics, nothing seemed to NAT at all.

VLC Streaming

Cool article on how to stream video using VLC.

http://www.howtogeek.com/118075/how-to-stream-videos-and-music-over-the-network-using-vlc/

Resetting Network Interface Statistics

Resetting the statistics on ifconfig (interfaces).

I have wanted to do that numerous times.

Here is how (which involves unloading and reloading the kernel module).

This won't work if the kernel is monolothic (compiled without modules).

http://www.ducea.com/2006/09/08/resetting-ifconfig-counters/

LSB Init Standard

This is a good-to-know for anyone working with Linux.

There's a standard to try and normalize / rationalize the init scripts across Linux distributions.

Here's a little blog on that topic:

http://www.thegeekstuff.com/2012/03/lsbinit-script/

More Fun with VirtualBox

Created a VirtualBox without enough disk space (compiling Boost / ASIO - another topic for another day).

Had to resize the drive.

Used this resource here, which worked like a champ.

http://www.howtogeek.com/124622/how-to-enlarge-a-virtual-machines-disk-in-virtualbox-or-vmware/

I did create a Primary partition at the end of the disk, and made it a Swap Partition (with Label Swap and formatted as linux-swap).

Installing VirtualBox GuestAdditions

Good blog on how to do this.

https://naveensnayak.wordpress.com/2016/01/19/installing-virtualbox-guest-addition-on-centos-7-server-no-gui/

I did not realize that GuestAdditions plugs into the kernels with a bunch of kernel modules, which requires you to have the kernel-devel packages in order to build those modules.

Upgrading the Linux Kernel

You could upgrade the kernel via elrepo.

rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm (external link)
yum install --enablerepo=elrepo-kernel kernel-ml

You can also install the updated firmware and headers

yum install --enablerepo=elrepo-kernel kernel-ml-{firmware,headers,devel}

You'll probably need to remove the kernel-firmware first:

yum remove kernel-{firmware,headers,devel}

EtherApe Network Visualization Tool

EtherApe.

What is it? It's a very cool network visualization tool. I first encountered the tool when a colleague of mine used it in a Marketing Showcase for a large networking company that we worked for.

Later, I rejoined that same colleague, and he had unleashed the same weapon for marketing demo systems. Interestingly, despite the fact that it was 10 years or so later, the people at this state-of-the-art networking company had not heard of the tool.

So - EtherApe was such a cool tool, that our trade shows and showcases were dependent on it once again. Which meant, that it not only did people want to run it, but inevitably, people wanted to customize it. Myself, I had never ventured into the bowels of EtherApe (we had other engineers doing this, so why duplicate effort?).

Eventually, though, there WAS a need to - a project - that came up, and we did want to customize EtherApe, so I rolled up my sleeves and downloaded EtherApe, and customized it using the Glade software. Unfortunately, I sent the server I did this on out into the field (to the customer), and therefore, although documenting what I did, did not have the actual customizations I had made.

I learned that I got lucky on that initial project. Compiling EtherApe can be difficult enough on a modern system. Customizing it leaves a number of places where someone can go wrong.

So, the objective of this blog is to "recapture the steps" to:
a. Download and Run EtherApe
b. Have the ability to customize EtherApe.

On the Etherape page, it lists the following dependencies:

• The libpcap packet capture library, available from the Lawrence Berkeley National Laboratory.
• GTK+, available from the GTK+ site. Version 2.12 or above is needed.
• Libglade 2, available from ftp://ftp.gnome.org/pub/GNOME/sources/libglade
• Gnome, available from the Gnome site. Version 2 is needed.
• The standard resolver library (exact name varies with OS and distribution)

libpcap
I checked, and the libpcap package was already installed, thanks to prior installations of things like Wireshark. In fact, not only libpcap was installed, but the libpcap development package is also installed. This is good, because it provides headers and additional packages.
My versions are:
libpcap-1.5.3-8.el7.x86_64
libpcap-devel-1.5.3-8.el7.x86_64

gtk+

Next, I looked at gtk+. The Etherape page states a minimum version of 2.12, which indicates to me that this version was probably tested by the developers.  When I go to the website for gtk+, I notice that there is a "gtk+" version 3 branch, and a version 2 branch (see below image). The version 2 has a 2.24 - and when you click on this, the latest version on 2.24 sources is 2.24.31. So - in summary, this is how I wound up installing this particular version of gtk+. I downloaded these sources, compiled them and installed them. This did not install an rpm, by the way - so an rpm -qa would not report this particular package as being installed on the system. One would have to "know", or "remember" that this was installed in a custom way, as is the case when you install sources like this.

NOTE:
Looking around, I do find this 2.12 version sitting on a site called:
https://launchpad.net/gtk/+milestone/2.12.0

This version is hosted as a bzip2 file, which means you must use the bzip utility to download it! And bzip is not like tar - in that it does not, by default, save the original archive. You must use the -dk options to do this, such as:
bzip2 -dk gtk+-2.12.0.tar.bz2

I did go ahead and download the 2.12 version of gtk+, and bzip - figuring it could be good to have, "just in case". You never know when these sites might disappear.


Next, from the gtk+ site, I see that gtk+ also has its list of dependencies:

Now keep in mind, these are dependencies for version 3.22 of GTK+. It doesn't mean an earlier version of GTK+ uses all of these, although it would be reasonable to assume so.

So let's do a quick check on the command line, with rpm -q (query), and see what we get:
rpm -qa | grep glib; reports version 2.46.2-4.el7.x86_64. glib-devel is NOT installed.
rpm -qa | grep pango; reports version 1.36.8-2 of both pango and pango-devel.
rpm -qa | grep pixbuf; reports version 2.31.6-3 of both gdk-pixbuf2 and gdk-pixbuf2-devel.
rpm -qa | grep atk; reports version 2.14.0-1.el7.x86_64 of both atk and atk-devel.
rpm -qa | grep gobj; reports version 1.42.0-1.el7.x86_64 of gobject-introspection.

libglade 2
With glade, we downloaded the 2.6.4 and did a configure/make/install on that package.
Doing a quick check to see what we have on the system:
rpm -qa | grep glade; reports libglade2-2.6.4-11.el7.x86_64 and also the devel package.

Gnome
It is not clear what we need to install for Gnome. Obviously, you need to have the Gnome desktop installed (which we do).

I decided to take a guess, and install libgnome (not sure if this is even a real package or not). So I typed "yum install libgnome", and sure enough, six packages installed.  The versions of libgnome and libgnome-devel include libgnome-2.32.1-9.el7.x86_64 and libgnome-devel-2.32.1-9.el7.x86_64.


On to Etherape

So we have all packages necessary, it appears, for Etherape to run.
Do we have the right versions of these packages? Not sure.

Let's check and see if we have Etherape.
rpm -qa | grep Etherape; returns nothing.

What version of Etherape should we download?
You can download binaries, or you can download and compile the source code, from the Sourceforge site. It could be interesting to inspect the source, so let's download that.

It looks like .9.1.4 is the current version (we are not at version 1.0 of this software, which has been around now for years? that seems weird to me). At this version level, it appears they have binary (precompiled) installs for several distribution flavors and versions of Linux, including CentOS 6 and 7.

But we want the source, and there is a source file there as well. We will download the file: etherape-0.9.14.tar.gz . This will download the file into the default browser directory for the user you are running as, and you will probably want to unpack it in another directory (/opt, /usr/local, etc) to compile it.

NOTE: If you get an error about a missing package called ScrollKeeper, install the package rarian. ScrollKeeper is now part of the rarian package, to my understanding.

From here, you can - and should - invoke Etherape to make sure it runs. That's certainly important.

Once you have verified that it runs, the next step is to run glade, and make sure glade launches. Problem is, how to you actually start glade? I wasn't sure - and it wasn't easy to figure out in two seconds. This site here has an example of how to start glade:

https://www-uxsup.csx.cam.ac.uk/courses/moved.Glade/course.pdf

But glade-2 did not start it. Instead, I found a glade-3 binary in /usr/local/bin. It turns out, that I guess Linux installed glade 3.85 on the system.

I did invoke this glade (3.85) and it did read in an Etherape glade file.

To be continued....