This is a good article that explains the VPN Filter hack.
https://arstechnica.com/information-technology/2018/05/hackers-infect-500000-consumer-routers-all-over-the-world-with-malware/
Friday, June 1, 2018
Router Hack: VPN Filter
This is a good article that explains the VPN Filter hack.
https://arstechnica.com/information-technology/2018/05/hackers-infect-500000-consumer-routers-all-over-the-world-with-malware/
Building a Cross Compilation Toolchain for MIPS32 - Part II
I spent about 3 days trying to get the cross compilation toolchain for MIPS 32 working. Maybe this wasn't enough time, but it was all of the time I had before I had to stick a knife in the effort and call it done - unsuccessfully.
Compiling the toolchain is essentially done in 3 steps:
1. Compile binutils for the target architecture
2. Compile a lightweight C compiler for the target architecture
3. Compile a standard C Library (e.g. glibc) for the target architecture using the compiler from step 2.
I could never make this work. I could usually get #1 to work. I could usually get #2 to work. I could never get #3 to work.
At this point, I went back and reverse-engineered the toolchain that OpenWRT was using for MIPS32. The CMakefiles, scripts, and generated Makefiles were extremely complex. And I saw patches being used in several places. Patches to scripts. Patches to Makefiles. Patches to header files.
At that point, I decided I didn't have the time or bandwidth to complete the project of building a cross-compilation toolchain successfully.
I decided instead to try and use the one from OpenWRT that was already built. More on that in a subsequent post.
Building a Cross Compilation Toolchain for MIPS32 - Part I
It seems there are a lot of people out there who do cross-compiling for the ARM processor (i.e. Raspberry Pi uses an ARM if I'm not mistaken).
But there aren't many people out there that have done this for a MIPS32 processor.
I noticed that when I built OpenWRT and used the make menuconfig utility to specify the target parameters, it has its own cross-compilation framework and engine in it.
To compile some ADDITIONAL software of your own, you would need to reverse-engineer (and possible recycle and reuse) this framework. Or, you could look into building your own cross-compilation platform.
I looked into the former, and then did some quick web searches to attempt the latter.
In doing the web searches, I began to learn the "methodology" for doing cross compilation in general. The steps seem more or less the same, no matter what kind of platform you are compiling for. It involves building a "toolchain". A Toolchain is essentially defined as "the tools used for cross compilation".
I found a guy who built such a toolchain for MIPS, and began the process using his site as a guide:
http://www.lara.prd.fr/users/oliviermehani/2007ie/mipsellinuxtoolchain
I like the fact that he lays out the process (or, at least, HIS process):
a) Download and Build binutils
b) Download and build gcc
c) Download and build gcc-lib
Another link I came across reaffirmed these steps, but with some additional insight and information.
http://cowlark.com/2009-07-04-building-gcc/
In this link, he explains, for example, why it is important to do your builds from a different directory than where the source is initially downloaded and extracted/spooled.
BTW...in his article he suggests building binutils from an outside directory but the first link does not do this. I was following the first set of steps from Olivier Mehani before I came across this additional link.
So What Went Wrong Initially
I was following the instructions on Oliver Mehani's site "to the letter". Which means that I decided to use the same versions of binutils, gcc and gcclib that he used in the article.
I failed to realize how old these versions were - some of which were going back to 2004 (it is now 2018). I also failed to realize that the true target, which is OpenWRT running on a former Ubiquity EdgeRouterX device, is a MUCH MUCH newer device and Linux operating system.
In fact, the OS is now called LEDE rather than OpenWRT (OpenWRT split with LEDE and the two have just recently merged back into a single entity and they now use the term LEDE for the OS).
When I downloaded the 2.14 version of binutils, it configured and built (and installed) just fine.
Building gcc also went fine although time consuming (at least on a single or dual core Virtual Machine on VirtualBox running on a Windows 10 laptop).
When I compiled gcclib, I got an error about it being incompatible with the binutils version I was using.
At this point, I began trying some different versions of both binutils as well as the compiler. I first downloaded the most recent versions of these. And got a series of nasty errors on the compile and make processes of gcc.
Later the light bulb went off and I realized that the best practice would be to use the same versions of binutils, gcc and gcclib that were on the target device (again, OpenWRT MIPS32 image I installed on the Ubiquity EdgeRouterX device).
So the versions on that device are:
GCC Prequisite Dependencies
When I compiled the original 2.13 of binutils and then compiled the 3.3.4 version of gcc on Olivier's page, binutils compiled fine but the gcc produced an error (a data type I believe it was).
So this led to me trying new versions and combinations/permutations of binutils and gcc trying to find a magic combination that would work (configure, make and make install).
Later more recent versions of gcc started to complain about some extraneous packages that it cold not find, telling me I could specify these libraries as compiler options:
configure: error: Building GCC requires GMP 4.2+, MPFR 2.4.0+ and MPC 0.8.0+. Try the --with-gmp, --with-mpfr and.or --with/mpc options to specify their locations.
I did not get these errors on gcc 3.3.4, so I started to get frustrated about the circuitous loop of package dependencies and considered abandoning my effort.
But - via a web search , I learned that that in version 5.4.0 of gcc, a script was packaged in the /contrib directory called "download_prerequisites" and the rather sophisticated script (I did not study it) downloads and positions these packages so that they can be built and incorporated into gcc seamlessly.
NOTE: I noticed this script was NOT included in version 4.3.3 of gcc, which I later used, but I found that I could port the script from version 5.4.0 to 4.3.3 and it appeared to work.
GCC Version
After initially trying to get 3.3.4 to work, I downloaded version 5.4.0 of the gcc compiler and after pulling down the threads and running the pre-req script to install GMP, MPFR and MPC, I tried to run the configure script which bailed with the error:
"@itemx must follow @item"
TexInfo Error on GCC Configure
Some web searches showed that the issue was the version of texinfo (I was using a recent 5.x version of it on my CentOS7 Linux VM). I downloaded the tool rpmreaper to see if it was safe to uninstall it (it was), and installed a much earlier version of texinfo (4.13) that someone on the web suggested might be valid to use in order to avoid this error. I compiled it and installed it, and tried to run the configure script again on version 5.4.0 of gcc.
The "configure" script in gcc is a VERY long script indeed to run (>30 min) on my laptop. You cannot specify -j X with the configure script, so it is serial, much like a linking process.
Link tests are not allowed after GCC_NO_EXECUTABLES
Finally - after waiting for what seemed like the better part of an hour, the configure process on gcc 5.4.0 bailed out with the following error:
checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.
Web searches appear inconclusive, although I do see some people complaining about this issue. Most of the people on these threads suggest that an earlier version of gcc may fix the issue.
So - we will attempt version 4.3.3 of the gcc compiler, which someone on the web suggests may work. First we will do a rm -rf on the gcc-build directory, then download this version 4.3.3 and re-run our configure script again to see if it solves this issue.
Version 4.3.3 of gcc
And off we go. Again. Actually, the configure script on this came back lickety split.
So now we will run, from our gcc-build directory, the "make -j 2" command to try and leverage two cores that we have provisioned on this virtual machine.
And off we go. Again. Waiting. Again.
But there aren't many people out there that have done this for a MIPS32 processor.
I noticed that when I built OpenWRT and used the make menuconfig utility to specify the target parameters, it has its own cross-compilation framework and engine in it.
To compile some ADDITIONAL software of your own, you would need to reverse-engineer (and possible recycle and reuse) this framework. Or, you could look into building your own cross-compilation platform.
I looked into the former, and then did some quick web searches to attempt the latter.
In doing the web searches, I began to learn the "methodology" for doing cross compilation in general. The steps seem more or less the same, no matter what kind of platform you are compiling for. It involves building a "toolchain". A Toolchain is essentially defined as "the tools used for cross compilation".
I found a guy who built such a toolchain for MIPS, and began the process using his site as a guide:
http://www.lara.prd.fr/users/oliviermehani/2007ie/mipsellinuxtoolchain
I like the fact that he lays out the process (or, at least, HIS process):
a) Download and Build binutils
b) Download and build gcc
c) Download and build gcc-lib
Another link I came across reaffirmed these steps, but with some additional insight and information.
http://cowlark.com/2009-07-04-building-gcc/
In this link, he explains, for example, why it is important to do your builds from a different directory than where the source is initially downloaded and extracted/spooled.
BTW...in his article he suggests building binutils from an outside directory but the first link does not do this. I was following the first set of steps from Olivier Mehani before I came across this additional link.
So What Went Wrong Initially
I was following the instructions on Oliver Mehani's site "to the letter". Which means that I decided to use the same versions of binutils, gcc and gcclib that he used in the article.
I failed to realize how old these versions were - some of which were going back to 2004 (it is now 2018). I also failed to realize that the true target, which is OpenWRT running on a former Ubiquity EdgeRouterX device, is a MUCH MUCH newer device and Linux operating system.
In fact, the OS is now called LEDE rather than OpenWRT (OpenWRT split with LEDE and the two have just recently merged back into a single entity and they now use the term LEDE for the OS).
When I downloaded the 2.14 version of binutils, it configured and built (and installed) just fine.
Building gcc also went fine although time consuming (at least on a single or dual core Virtual Machine on VirtualBox running on a Windows 10 laptop).
When I compiled gcclib, I got an error about it being incompatible with the binutils version I was using.
At this point, I began trying some different versions of both binutils as well as the compiler. I first downloaded the most recent versions of these. And got a series of nasty errors on the compile and make processes of gcc.
Later the light bulb went off and I realized that the best practice would be to use the same versions of binutils, gcc and gcclib that were on the target device (again, OpenWRT MIPS32 image I installed on the Ubiquity EdgeRouterX device).
So the versions on that device are:
- binutils v2.27
- gcc v5.4.1
- glibc v5.4.1
The closest ones I could find on the mirrors were:
- binutils v2.27
- gcc v5.4.0
- glibc v5.4.0
GCC Prequisite Dependencies
When I compiled the original 2.13 of binutils and then compiled the 3.3.4 version of gcc on Olivier's page, binutils compiled fine but the gcc produced an error (a data type I believe it was).
So this led to me trying new versions and combinations/permutations of binutils and gcc trying to find a magic combination that would work (configure, make and make install).
Later more recent versions of gcc started to complain about some extraneous packages that it cold not find, telling me I could specify these libraries as compiler options:
configure: error: Building GCC requires GMP 4.2+, MPFR 2.4.0+ and MPC 0.8.0+. Try the --with-gmp, --with-mpfr and.or --with/mpc options to specify their locations.
I did not get these errors on gcc 3.3.4, so I started to get frustrated about the circuitous loop of package dependencies and considered abandoning my effort.
But - via a web search , I learned that that in version 5.4.0 of gcc, a script was packaged in the /contrib directory called "download_prerequisites" and the rather sophisticated script (I did not study it) downloads and positions these packages so that they can be built and incorporated into gcc seamlessly.
NOTE: I noticed this script was NOT included in version 4.3.3 of gcc, which I later used, but I found that I could port the script from version 5.4.0 to 4.3.3 and it appeared to work.
GCC Version
After initially trying to get 3.3.4 to work, I downloaded version 5.4.0 of the gcc compiler and after pulling down the threads and running the pre-req script to install GMP, MPFR and MPC, I tried to run the configure script which bailed with the error:
"@itemx must follow @item"
TexInfo Error on GCC Configure
Some web searches showed that the issue was the version of texinfo (I was using a recent 5.x version of it on my CentOS7 Linux VM). I downloaded the tool rpmreaper to see if it was safe to uninstall it (it was), and installed a much earlier version of texinfo (4.13) that someone on the web suggested might be valid to use in order to avoid this error. I compiled it and installed it, and tried to run the configure script again on version 5.4.0 of gcc.
The "configure" script in gcc is a VERY long script indeed to run (>30 min) on my laptop. You cannot specify -j X with the configure script, so it is serial, much like a linking process.
Link tests are not allowed after GCC_NO_EXECUTABLES
Finally - after waiting for what seemed like the better part of an hour, the configure process on gcc 5.4.0 bailed out with the following error:
checking dynamic linker characteristics... configure: error: Link tests are not allowed after GCC_NO_EXECUTABLES.
Web searches appear inconclusive, although I do see some people complaining about this issue. Most of the people on these threads suggest that an earlier version of gcc may fix the issue.
So - we will attempt version 4.3.3 of the gcc compiler, which someone on the web suggests may work. First we will do a rm -rf on the gcc-build directory, then download this version 4.3.3 and re-run our configure script again to see if it solves this issue.
Version 4.3.3 of gcc
And off we go. Again. Actually, the configure script on this came back lickety split.
So now we will run, from our gcc-build directory, the "make -j 2" command to try and leverage two cores that we have provisioned on this virtual machine.
And off we go. Again. Waiting. Again.
Thursday, May 10, 2018
OpenWRT on a Ubiquiti EdgeRouterX - Part II
So in Part I, I built the source from scratch and generated an image that I could load onto a Ubiquiti EdgeRouterX device.
This worked - in the sense that I was able to boot that image and get a functional and working OpenWRT (LEDE) image.
The router handed me an IP address like it should have. The router was passing my packets out to the web and back, performing its SNAT functions like a good little router should.
The problems started when I tried to use the package manager.
The first problem I ran into using the package manager was the fact that the urls in the feeds.conf file were not resolving. They were all using openwrt.git.org. This may have been because the git sites for openwrt were down (temporarily). I fixed these urls by looking out on the web for the actual github urls, and commenting out the git.org sites and using the github sites instead.
Once I fixed this issue, I noticed that I kept getting an error from the opkg utility about having zero bytes in the overlay file system.
In looking, I did not have an /overlay file system. I had a directory off the root directory called /overlay.
In doing a google search on this issue I found a number of forum posts but when you clicked them, they links timed out.
Clearly, something went wrong when I built the image.
I typed the mount command, and saw a / mounted on a rootfs file system. I saw a tmpfs on /tmp. But no overlay. I didn't really understand the overlay file system anyway, so I was in trouble there. I knew enough to believe I might be able to manually mount this file system on a device, but I didn't see the device and things looked quite intimidating.
I began to realize that I was quite screwed. I had a router that I could not administratively update or manage because the file system was not "correct". I considered paper clipping (hard reset) the router, but I wasn't sure what that would do since I had replaced the original flash image from Ubiquiti with the one from OpenWRT. It seemed risky to do that.
I finally found a page that allowed me to take a more sensible risk. I found a page that had hardware / firmware images, and I decided to "upgrade" my router with the new image listed on this page. I wasn't sure if that would actually fix the file system layout on the flash or not, and I suspected it wouldn't, but I gave it a try nonetheless.
So after downloading the new flash image, I discovered another issue. This image I had built did not include the Luci GUI, so I needed to do anything and everything via command line. I found a way to upgrade the router via command line with this link:
https://openwrt.org/docs/guide-user/installation/sysupgrade.cli
Using ssh, I copied the downloaded image into a temporary directory and ran the command:
Now we have a /dev/root on /rom of type squashfs - something we did not see before.
Most importantly, we see the overlayfs mounted on the / folder in a directory called /overlay.
And some others we did not see before, either.
Things look better. We can launch a GUI and we can update our packages now. I installed OpenVPN, for example, which added itself to the "Services" menu.
This worked - in the sense that I was able to boot that image and get a functional and working OpenWRT (LEDE) image.
The router handed me an IP address like it should have. The router was passing my packets out to the web and back, performing its SNAT functions like a good little router should.
The problems started when I tried to use the package manager.
The first problem I ran into using the package manager was the fact that the urls in the feeds.conf file were not resolving. They were all using openwrt.git.org. This may have been because the git sites for openwrt were down (temporarily). I fixed these urls by looking out on the web for the actual github urls, and commenting out the git.org sites and using the github sites instead.
Once I fixed this issue, I noticed that I kept getting an error from the opkg utility about having zero bytes in the overlay file system.
In looking, I did not have an /overlay file system. I had a directory off the root directory called /overlay.
In doing a google search on this issue I found a number of forum posts but when you clicked them, they links timed out.
Clearly, something went wrong when I built the image.
I typed the mount command, and saw a / mounted on a rootfs file system. I saw a tmpfs on /tmp. But no overlay. I didn't really understand the overlay file system anyway, so I was in trouble there. I knew enough to believe I might be able to manually mount this file system on a device, but I didn't see the device and things looked quite intimidating.
I began to realize that I was quite screwed. I had a router that I could not administratively update or manage because the file system was not "correct". I considered paper clipping (hard reset) the router, but I wasn't sure what that would do since I had replaced the original flash image from Ubiquiti with the one from OpenWRT. It seemed risky to do that.
I finally found a page that allowed me to take a more sensible risk. I found a page that had hardware / firmware images, and I decided to "upgrade" my router with the new image listed on this page. I wasn't sure if that would actually fix the file system layout on the flash or not, and I suspected it wouldn't, but I gave it a try nonetheless.
So after downloading the new flash image, I discovered another issue. This image I had built did not include the Luci GUI, so I needed to do anything and everything via command line. I found a way to upgrade the router via command line with this link:
https://openwrt.org/docs/guide-user/installation/sysupgrade.cli
Using ssh, I copied the downloaded image into a temporary directory and ran the command:
sysupgrade -v /tmp/*.binAnd - lo and behold - after rebooting, I got a new, much superior image and experience. I had the Luci GUI. The file system looked proper.
Now we have a /dev/root on /rom of type squashfs - something we did not see before.
Most importantly, we see the overlayfs mounted on the / folder in a directory called /overlay.
And some others we did not see before, either.
Things look better. We can launch a GUI and we can update our packages now. I installed OpenVPN, for example, which added itself to the "Services" menu.
Tuesday, May 8, 2018
OpenWRT on a Ubiquity EdgeRouterX - Part I
We have been using these Ubiquity EdgeRouter X devices for a while now in here (about a year). They're incredible devices; can't say enough about them.
We were curious to see if you could run OpenWRT on these.
The first thing we had to do is to get some intelligence on the device.
We could have cracked open the case and studied the circuit board, but we didn't do that. By using search engines, we were able to ascertain that it was a 32 bit MIPS processor (MIPS Technologies); MIPS32 1004K. They make a couple of renditions of these chips (i.e. MIPS321004Kc or MIPS321003Kf), the c standing for "coherent" and the f standing for "floating point".
After logging onto the device and looking in /proc/cpuinfo (issuing the command "show hardware cpu" also shows this information), we were able to ascertain that the board (system type?) is MediaTek MT6721 ver:1 eco:3. The processor was confirmed as MIPS 1004Kc V2.15.
So the next step was to figure out how to download and compile OpenWRT on this chipset. This can get hairy and complicated, depending on how much support for the chip there is out in the communities.
As it turns out, if you download OpenWRT on a Linux machine, there is some built-in cross compilation support for MIPS32 right within OpenWRT itself. Apparently the gnu compiler gcc supports the MIPS32 target architecture also, which is nice [ NOTE: validate this ].
If there wasn't, you would need to try to find a toolchain (which has compiler, linker and so forth), and [try to] build OpenWRT from scratch, and this would also include building any and all libraries that the code referenced - a potentially huge job. Just doing the Makefiles could take a week or two.
So to compile it, you need to download the OpenWRT source code (I created a directory called /opt/crosscompile and untarred to that, which creates a openwrt directory) and from the top level directory run "make config", or, in my case, I ran "make menuconfig". Running menuconfig pulls up a curses-based menu that allows you to configure your build - including the options to include in your OpenWRT and target platform specifics for a cross-compile.
Build Options
Originally when I ran this, I went to select the Target System and chose "MIPS Malta CoreLV board (qemu)". There was also an option called "MIPS pistachio". I was confused as to which was the right one. And - YOU BETTER CHOOSE THE RIGHT ONE! If you don't, you can risk bricking (ruining) the device when you try to run the code.
It turns out, that neither is the right one. The correct choice for "Target System" was further down below, called "MediaTek Ralink MIPS" (other than the Mediatek and MIPS, there wouldn't be any easy way to figure this out).
The Subtarget is MT6721, which now starts to make some sense based on the output of what showed up in /proc/cpuinfo on the Ubiquity device. But this subtarget would never have shown up as an option if you didn't get the proper Target System right to begin with.
If you select the proper Target and Subtarget, you get an option for "Ubiquity EdgeRouter X" as the Target Profile.
And finally, for "Target Image", you need to make sure that "ramdisk" is selected because this generates a "img" file that is tarred into a tarball that you will eventually boot into the Ubiquity image loader to boot the OpenWRT image you compile.
I made no changes to the Global Build Settings. Initially, I did select "Build the OpenWRT Image Builder" and "Build the OpenWRT SDK" options in the make menuconfig menu listing, but I had to back these out because the compile was spewing out warnings about library dependencies.
Libraries
Running the build without fully reading the README is almost always a mistake, and it bit me here as well. There is a script called "feeds" that will update and install the dependent libraries. This needs to be run.
This script failed when I ran it, and this COULD be because OpenWRT had an issue with their Github site that day (the mirrors).
I went on the web and did some checking, and the packages I needed in the build seemed to be on https://github.com/XXXXX as opposed to https://openwrt.git.org which is what the scripts were all primed with. So what I did was to fix these urls one at a time, and iteratively kick off the build until it launched with no dependent library warnings.
Build
After the build completed (which took a while - about 2 hours - in a 2 core VirtualBox virtual machine running CentOS7 Linux), I had to figure out where the resultant images were written. They are actually written to the /openwrt/bin/targets folder. The full path is:
/opt/crosscompile/openwrt/bin/targets/ramips/mt7621/openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar.
There is also a bin file in this directory as well, but it is the tar you need to load into the device.
Transfer
The next step is secure copy (scp) the tar file into a temporary directory on the Ubiquity EdgeRouterX device. We elected to put it into /tmp.
Then - once copied over, you can log into the UbiquityEdgeRouterX via ssh or through the web console, and go into the temporary directory and type:
"add system image openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar"
If this add system command worked you should get a confirmation (i.e. image added). It will place the image on a list of images stored on the device, and mark that image as bootable (apparently it works much like a LIFO queue, where the last image added is the bootable one).
It is now time to reboot the device, by typing "reboot". This is the moment of truth.
But...there is an important step!
Change Ethernet Cable to eth1 on the device!!!!
On the Ubiquity device, eth0 is the management interface, and eth1 is actually the WAN link (eth2-4 are local links).
On OpenWRT, it uses eth0 as a WAN link, and eth1 is the management link.
If you attempt to reboot the device and your cable is [ still ] plugged into the wrong NIC, you will not be able to log in and access the device under OpenWRT.
So switching the cable from eth0 to eth1 is the next step...
Image Re-load
If there any major issues with the image I think adding the image would have flagged that. But the real proof is when you reboot the device.
In our case, the image booted with a "OpenWRT" ASCII MOTD banner, letting you know it booted.
Explore
From here, you can dance around the OpenWRT file systems to your hearts content, as this review stops here.
The only thing I will mention is that I did install the graphical menu for OpenWRT, called Luci.
I will log into that web interface to see what it looks like as I explore OpenWRT.
We were curious to see if you could run OpenWRT on these.
The first thing we had to do is to get some intelligence on the device.
We could have cracked open the case and studied the circuit board, but we didn't do that. By using search engines, we were able to ascertain that it was a 32 bit MIPS processor (MIPS Technologies); MIPS32 1004K. They make a couple of renditions of these chips (i.e. MIPS321004Kc or MIPS321003Kf), the c standing for "coherent" and the f standing for "floating point".
After logging onto the device and looking in /proc/cpuinfo (issuing the command "show hardware cpu" also shows this information), we were able to ascertain that the board (system type?) is MediaTek MT6721 ver:1 eco:3. The processor was confirmed as MIPS 1004Kc V2.15.
So the next step was to figure out how to download and compile OpenWRT on this chipset. This can get hairy and complicated, depending on how much support for the chip there is out in the communities.
As it turns out, if you download OpenWRT on a Linux machine, there is some built-in cross compilation support for MIPS32 right within OpenWRT itself. Apparently the gnu compiler gcc supports the MIPS32 target architecture also, which is nice [ NOTE: validate this ].
If there wasn't, you would need to try to find a toolchain (which has compiler, linker and so forth), and [try to] build OpenWRT from scratch, and this would also include building any and all libraries that the code referenced - a potentially huge job. Just doing the Makefiles could take a week or two.
So to compile it, you need to download the OpenWRT source code (I created a directory called /opt/crosscompile and untarred to that, which creates a openwrt directory) and from the top level directory run "make config", or, in my case, I ran "make menuconfig". Running menuconfig pulls up a curses-based menu that allows you to configure your build - including the options to include in your OpenWRT and target platform specifics for a cross-compile.
Build Options
Originally when I ran this, I went to select the Target System and chose "MIPS Malta CoreLV board (qemu)". There was also an option called "MIPS pistachio". I was confused as to which was the right one. And - YOU BETTER CHOOSE THE RIGHT ONE! If you don't, you can risk bricking (ruining) the device when you try to run the code.
It turns out, that neither is the right one. The correct choice for "Target System" was further down below, called "MediaTek Ralink MIPS" (other than the Mediatek and MIPS, there wouldn't be any easy way to figure this out).
The Subtarget is MT6721, which now starts to make some sense based on the output of what showed up in /proc/cpuinfo on the Ubiquity device. But this subtarget would never have shown up as an option if you didn't get the proper Target System right to begin with.
If you select the proper Target and Subtarget, you get an option for "Ubiquity EdgeRouter X" as the Target Profile.
And finally, for "Target Image", you need to make sure that "ramdisk" is selected because this generates a "img" file that is tarred into a tarball that you will eventually boot into the Ubiquity image loader to boot the OpenWRT image you compile.
I made no changes to the Global Build Settings. Initially, I did select "Build the OpenWRT Image Builder" and "Build the OpenWRT SDK" options in the make menuconfig menu listing, but I had to back these out because the compile was spewing out warnings about library dependencies.
Libraries
Running the build without fully reading the README is almost always a mistake, and it bit me here as well. There is a script called "feeds" that will update and install the dependent libraries. This needs to be run.
This script failed when I ran it, and this COULD be because OpenWRT had an issue with their Github site that day (the mirrors).
I went on the web and did some checking, and the packages I needed in the build seemed to be on https://github.com/XXXXX as opposed to https://openwrt.git.org which is what the scripts were all primed with. So what I did was to fix these urls one at a time, and iteratively kick off the build until it launched with no dependent library warnings.
Build
After the build completed (which took a while - about 2 hours - in a 2 core VirtualBox virtual machine running CentOS7 Linux), I had to figure out where the resultant images were written. They are actually written to the /openwrt/bin/targets folder. The full path is:
/opt/crosscompile/openwrt/bin/targets/ramips/mt7621/openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar.
There is also a bin file in this directory as well, but it is the tar you need to load into the device.
Transfer
The next step is secure copy (scp) the tar file into a temporary directory on the Ubiquity EdgeRouterX device. We elected to put it into /tmp.
Then - once copied over, you can log into the UbiquityEdgeRouterX via ssh or through the web console, and go into the temporary directory and type:
"add system image openwrt-ramips-mt7621-ubnt-erx-initramfs-factory.tar"
If this add system command worked you should get a confirmation (i.e. image added). It will place the image on a list of images stored on the device, and mark that image as bootable (apparently it works much like a LIFO queue, where the last image added is the bootable one).
It is now time to reboot the device, by typing "reboot". This is the moment of truth.
But...there is an important step!
Change Ethernet Cable to eth1 on the device!!!!
On the Ubiquity device, eth0 is the management interface, and eth1 is actually the WAN link (eth2-4 are local links).
On OpenWRT, it uses eth0 as a WAN link, and eth1 is the management link.
If you attempt to reboot the device and your cable is [ still ] plugged into the wrong NIC, you will not be able to log in and access the device under OpenWRT.
So switching the cable from eth0 to eth1 is the next step...
Image Re-load
If there any major issues with the image I think adding the image would have flagged that. But the real proof is when you reboot the device.
In our case, the image booted with a "OpenWRT" ASCII MOTD banner, letting you know it booted.
Explore
From here, you can dance around the OpenWRT file systems to your hearts content, as this review stops here.
The only thing I will mention is that I did install the graphical menu for OpenWRT, called Luci.
I will log into that web interface to see what it looks like as I explore OpenWRT.
Thursday, April 26, 2018
SDN Solutions
Note to self: Look into Aryaka Networks
I am told they have a very robust and mature orchestration solution.
Look into Thousand Eyes
Not sure what this is all about.
TCPKali Part II
I've spent about a week testing with this tcpkali tool now.
This is the test case I initially put together, which is to test connections between two CentOS 7 VMs via an Ubuntu VM (IP Forwarder / Router).
The tool spawns worker threads that test connection establishment and bandwidth on each of the connections, both based on an individual thread and collectively.
One thing I did run into with using Lighttpd, is that at high volume levels I kept getting invalid request headers. I wasn't sure if this was tcpkali messing up the headers, or if it was Lighttpd.
So I switched the architecture a bit (based on the suggestion of one of the other engineers in here). Instead of using Lighttd as the web server, I put tcpkali on both sides (left and right) - in client and server mode respectively.
This worked much better than using Lighttpd as a web server. You can send messages and run the server in active or silent mode, returning responses or discarding them.
The tcpkali has some great features. One HUGE feature is the ability to send requests per second with the -r option. The tool attempts to ramp up to the specified number of connections, and if it cannot do this, it will bail without examining throughput measurements. I did notice that it sends data with the -r option DURING the ramp-up period, in addition to after. The tool also has other features like the connect rate, so that if you are establishing 100K connections you can speed up that process by doing 10K or 20K a clip rather than smaller default limits.
What I discovered in the testing I was doing, was that the connection rate had a profound effect on the number of connections you could establish. The default connection rate is 100. And this works fine for up to about 10,000 connections. But if you try to go to higher than that, the rate starts to inhibit your ability to scale the connections and you need to tweak it higher - just not too much higher.
The sweet spot I discovered was a 1,000 : 1 ratio. So 10,000 connections would use a rate of 100 connections per second, 20,000 would use a rate of 200 connections per second, and so forth.
This is the test case I initially put together, which is to test connections between two CentOS 7 VMs via an Ubuntu VM (IP Forwarder / Router).
The tool spawns worker threads that test connection establishment and bandwidth on each of the connections, both based on an individual thread and collectively.
One thing I did run into with using Lighttpd, is that at high volume levels I kept getting invalid request headers. I wasn't sure if this was tcpkali messing up the headers, or if it was Lighttpd.
So I switched the architecture a bit (based on the suggestion of one of the other engineers in here). Instead of using Lighttd as the web server, I put tcpkali on both sides (left and right) - in client and server mode respectively.
This worked much better than using Lighttpd as a web server. You can send messages and run the server in active or silent mode, returning responses or discarding them.
The tcpkali has some great features. One HUGE feature is the ability to send requests per second with the -r option. The tool attempts to ramp up to the specified number of connections, and if it cannot do this, it will bail without examining throughput measurements. I did notice that it sends data with the -r option DURING the ramp-up period, in addition to after. The tool also has other features like the connect rate, so that if you are establishing 100K connections you can speed up that process by doing 10K or 20K a clip rather than smaller default limits.
What I discovered in the testing I was doing, was that the connection rate had a profound effect on the number of connections you could establish. The default connection rate is 100. And this works fine for up to about 10,000 connections. But if you try to go to higher than that, the rate starts to inhibit your ability to scale the connections and you need to tweak it higher - just not too much higher.
The sweet spot I discovered was a 1,000 : 1 ratio. So 10,000 connections would use a rate of 100 connections per second, 20,000 would use a rate of 200 connections per second, and so forth.
Subscribe to:
Posts (Atom)
SLAs using Zabbix in a VMware Environment
Zabbix 7 introduced some better support for SLAs. It also had better support for VMware. VMware, of course now owned by BroadSoft, has prio...
