Pinephone Pro (with Tow-Boot) - Installing a new OS on the eMMC

In my previous Pinephone Pro, I was describing how I was coming up to speed on the different storage mechanisms on the Pinephone Pro: SPI vs eMMC vs microSD.  

Contextually, we are talking about block storage, and there is a well known command that one can run to see block storage on a Linux device: lsblk.  Running this command on your Pinephone Pro - in a Terminal - can help you understand "what is what and where". And it's important to understand this.

One concern I had, was installing a new OS to the eMMC, and blowing away the boot process. I had installed Tow-Boot on the phone, but had to make sure it was in its own spot (it was, SPI) - away from where a new OS was going to go - especially if you plan to clean or format the eMMC before installing a new OS. So my previous post discusses how I had to figure all of this out, and learned that Tow-Boot was installed on the SPI - making it safe to install a new OS.

Here was my process for installing this new OS, with comments:

1. Download the image
• Figure out what display manager you want. 
• Phosh? Plasma? Gnome? Xfce? There is not shortage of choices.
• I went with Plasma - because it worked well when I ran the OS on the microSD
•  I went with 20250206
• Check the md5 hash - which is always wise to verify the integrity of the image.
• Unpack/Uncompress the "xz" file.
• NOTE: In Mac Pro, xz did not work, and neither did tar xvfz. Using the GUI and clicking the file in File Manager invoked the Archive Utility to unpack the file. But in Linux, xz or tar should work fine.
  
2. Make sure your Pinephone Pro is sufficiently charged. 
• At least 50%, preferably 75%, and optimally fully charged at 90% or greater. 
• I should note that with Manjaro, the power charge percentage was not always accurate.
3. Power off your Pinephone Pro.
4. Connect the Pinephone Pro using USB-C connector, into a USB-C connector of your laptop.
5. Power the phone up, and after first vibration, hit the Volume-Up button
• You are looking for a blue light on your Pinephone Pro, signifying that you are in USB Mode.
6. Make sure the laptop/computer sees the Pinephone Pro as a device.
• In my case, on a Mac Pro, I used File Manager.
• Examine the /dev devices and this is IMPORTANT! Because if you install your OS onto your laptop partition, you have a big big problem.
• I quickly noticed that /dev/disk4 had the BOOT_MJRO volume name on it, ensuring that disk4 was the disk I wanted to install the new OS to.
7. Unmount the disk
• because you cannot format or do an image copy on a disk that's already mounted. 
• on a Mac Pro, diskutil was used for this: sudo diskutil unmountDisk /dev/disk4 
  
8. Clean the partition
• sudo dd if=/dev/zero of=/dev/disk4 bs=1M count=100
9. Copy the image to the eMMC 
• Tools like Balena Etcher can be used for this.
• The "dd" tool is a Linux tried-true way to do this and this is what I chose:
• sudo dd if=20250206-0046-postmarketOS-v24.12-plasma-mobile-5-pine64-pinephonepro.img of=/dev/dis
  k4 bs=1M status=progress
10. Watch the progress, and once finished, eject the phone. 
• sudo diskutil eject /dev/disk4
11. Power the Pinephone Pro down
12. Unplug the USB-C connector that is connected between Pinephone Pro and the laptop/computer.
    
13. Power the Pinephone Pro back up.
• You will see a terminal show up on the screen - don't mess with it - just wait.
• Eventually the Plasma Display Manager will (or should) light up.
• The OS will take some minutes to initialize, and to be responsive to user input.
14. Log into the phone

Pinephone Pro - Booting an OS off SPI vs eMMC

I finally got a chance to pick the Pinephone Pro back up and play with it some more.

I was able to charge up the battery, and boot the phone and verify that Tow-Boot was installed on it properly. That was my first step. I believe I verified this by holding the volume down button, and waiting for the light to turn aqua (note, it may have been volume up, I should check this for correctness).

Next, I rebooted the phone,  and it booted into the Manjaro OS which is installed on the eMMC drive of the phone.

Next, I put the PostMarketOS into the microSD card slot, and booted the phone. Apparently Tow-Boot uses the following boot order:

1. SPI - more on this in a bit, I had to learn what this is
   
2. microSD Card
3. eMMC (which has Manjaro on it)

I didn't get a Boot Menu - but maybe a key sequence (volume up?) would give me such a menu. It booted straight into the PostMarket OS. 

I proceeded to experiment with PostMarket OS, and did a complete update of all of the packages on it.

Next, I wondered how I could "replace" the default Manjaro with the PostMarket OS, which was newer than Manjaro, such that it would boot PostMarket OS on the eMMC, allowing me recycle the microSD card for perhaps another OS distribution I could take a look at later. 

It turns out, that there is a PostMarketOS "on-disk installer".  It is called pmbootstrap.

THIS is where I had to learn about SPI. Because there is a warning about over-writing your Tow-Boot installation, if Tow-Boot was not installed on SPI. 

so...what is SPI? (more search required)

SPI Flash is a type of non-volatile memory that uses the Serial Peripheral Interface (SPI) protocol for communication. It is commonly used in embedded systems for data storage and transfer, allowing devices to retain information even when powered off. 

Apparently it is a newer (or improved, perhaps) concept, found on phones with System-On-A-Chip (SOC) architectures. 

so...how do you know if you even have SPI?

Answer: I had to figure out which version of Pinephone Pro I have. 

I finally learned that there is a Developer Edition of the Pinephone Pro, and there is a Explorer Edition. The Explorer Edition supposedly has the SPI. 

But what confused me, is that it said the phone supporting SPI had the Rockchip RK3399S SoC. And when I went into the terminal on the phone and ran "lscpu", it said I had an ARM Cortex A-53 chip. 

so...now I am thoroughly confused.

Well, I finally learned, that the Rockchip RK3399S SoC combines four Cortex-A53 cores with two Cortex-A72 cores.

hmmm, I did not see the 72 in the lscpu command I ran - but, it does look like I have the SPI.

but, how do I know that Tow-Boot was installed on the SPI, versus the eMMC? Because if I have this wrong, I can't boot an OS as there would be no bootloader partition.

I think the SPI is mmcblk1 device. And /boot is on mmcblk1p1 partition of that device.

The Manjaro (previous installation) is definitely on the eMMC, which is on mmcblk2 device, which has two partitions on it, one of them being /root.

NUMA PreferHT VM setting on a Hyperthread-Enabled ESXi Hypervisor

This could be a long post, because things like NUMA can get complicated.

For background, we are running servers - hypervisors - that have 24 cores. There are two chips - wafers as I like to refer to them - each with 12 cores, giving a total of 24 physical cores.

When you enable hyperthreading, you get 48 cores, and this is what is presented to the operating system and cpu scheduler (somewhat - more on this later).  But - you don't get an effective doubling of cores when you enable hyperthreading. What is really happening, is that the 24 cores are "cut in half" so that another 24 cores can be "fit in", giving you 48 logical cores.  

Worth mentioning also, is that each (now half) core, has a "sibling" - and this also matters from a scheduling perspective when you see things like cpu pinning used - because if you pin something to a specific core, then that "sibling" cannot be used for something else.  For example, if you enabled hyperthreading, the cores would look like:

0 | 1

2 | 3

4 | 5

... and so on. So if someone pinned to core 4, core 5 is also "off the table" now from a scheduling perspective because pinning is a physical core concept, not a logical core concept.

So with this background, we had a tenant who wanted to enable a "preferHT" setting. This setting can be applied to an entire hypervisor by setting numa.PreferHT=1, affecting all VMs deployed on it.

Or, one can selectively add this setting to a particular or specific virtual machine by going into the Advanced Settings and configuring numa.vcpu.preferHT=TRUE.  

In our case, it was the VM setting being requested - not the hypervisor setting.  Now, this tenant is the "anchor tenant" on the platform, and their workloads are very latency sensitive. So it was important to jump through this hoop when it was requested. First, we tested the setting by powering a VM off and adding the setting, then powering the VM back on. No problems with this. We then migrated the VM to another hypervisor, and had no issues with that either. Aside of that, though, how do you know that the VM setting "took" - meaning that it was picked up and recognized?

It turns out, that there are a couple of ways to do this:

1. esxtop

When you load esxtop, it is going to show you cpu by default. But if you hit the "m" key, it goes into a "memory view". If you go into memory view by hitting "m" and then hit the "f" key, a list of fields will show up. One of them, is NUMA Statistics. So by selecting this, you get a ton of interesting information about NUMA. The settings you are most interested in, are going to be:

NHN - Current home node for the virtual machine or resource pool - in our case, this was 0 or 1 (we had two numa nodes, as there is usually one per physical cpu socket).

NMIG - Number of NUMA migrations between two snapshot samples

NRMEM - (NUMA Remote Memory): Amount of remote memory allocated to the virtual machine, in MB

NLMEM (NUMA Local Memory) - Amount of local memory allocated to the virtual machine, in MB

L%D - this shows the amount of memory that is Localized. You want this number to be 100% but seeing the number in the 90s is probably okay also because it is showing that the memory access is not traversing a NUMA bus, which adds latency.

GST_NDx (Guest Node x): Guest memory being allocated for the VM on NUMA node x, where x is the node number

MEMSZ (Memory Size): Total amount of physical memory allocated to a virtual machine

2. vmdumper command

I found this command on a blog post - which I will list in my sources at the end of this blog post. This useful command, can show you a lot of interesting information about how NUMA is working "under the hood" (in practice). It can show you a Logical Processor to NUMA Node Map, it can show you how many home nodes are utilized for a given VM, and show you the assignment of NUMA clients to the respective NUMA nodes.

One of the examples covered in this blog post refers to the situation where a VM has 12 vCPUs on a 10 core system, and then goes down and shows what it would look like if the VM had 10 vCPU instead.

Sources:

http://www.staroceans.org/ESXi_VMkernel_NUMA_Constructs.htm

https://frankdenneman.nl/2010/02/03/sizing-vms-and-numa-nodes/

https://frankdenneman.nl/2010/10/07/numa-hyperthreading-and-numa-preferht/

https://docs.pexip.com/server_design/vmware_numa_affinity.htm

https://docs.pexip.com/server_design/numa_best_practices.htm#hyperthreading

https://knowledge.broadcom.com/external/article?legacyId=2003582

 

MySQL Max Allowed Packet

I recently conducted an upgrade, and for the life of me I couldn't figure out why the application wouldn't initialize.

I checked MySQL - it seemed to be running fine. I logged into the database, checked the Percona cluster status, it looked fine.

I checked RabbitMQ, and it also seemed to be running fine.

In checking the application logs, I saw an exception about a query and the packet size being too big, and I thought this was strange - mainly because of the huge size of the packet.

Sure enough, after calling support, I was informed that I needed to change the MySQL configuration in my.cnf and add a directive in the [mysqld] section.

max_allowed_packet=128M

In terms of what this value should 'really' be, I was told that this is a normal setting on most installations.

Who knew? It's unusual to be adding new parameters on the fly like this to a clustered database. 

But, sure enough, after restarting the database (well, the whole VM actually because I had done updates), it came up just fine.

Cisco UCS M5 Server Monitoring with Zabbix

I got a request from my manager recently, about using Zabbix to monitor Cisco servers.  

Specifically, someone had asked about whether it was possible to monitor the CRC errors on an adaptor.

Right now, the monitoring we are doing is coming from the operating systems and not at the hardware level. But we do use Zabbix to montor vCenter resources (hypervisors), using VMware templates, and we use Zabbix to "target monitor" certain virtual machines at the Linux OS level (Linux template) and at Layer 7 (app-specific templates).

Up to this point, our Zabbix monitoring has been, essentially, "load and forget" where we load the template, point Zabbix to a media webhook (i.e. Slack) and just monitor what comes in. We haven't really even done much extension of the templates, using everything "out of the box". Recently, we did add some new triggers on VMware monitoring, for CPU and Memory usage thresholds. We were considering adding some for CPU Ready as well.

But...this ask was to monitor Cisco servers, with our Zabbix monitoring system.

The first thing I did, was to check and see what templates for Cisco came "out of the box". I found two:

1. Cisco UCS by SNMP
2. Cisco UCS Manager by SNMP

I - incorrectly - assumed that #2, the Cisco UCS Manager by SNMP, was a template to interface with a Cisco UCS Manager. I learned a bit later, that it is actually a template to let Zabbix "be" or "emulate" a Cisco UCS Manager (as an alternative or replacement). 

First, I loaded the Cisco UCS by SNMP template. The template worked fine from what I could tell, but it didn't have any "network" related items (i.e. network adaptors).

As mentioned, after reading that Cisco UCS Manager was an extension or superset of Cisco UCS by SNMP, I went ahead and loaded that template on some selected hosts. We were pleased to start getting data flowing in from those hosts, and this time the items included in the template were adaptor metrics, but very basic metrics such as these shown below.

Adaptor/Ethernet metrics in Cisco UCS Manager Template

This was great. But we needed some esoteric statistics, such as crc errors on an adaptor. How do we find these? Are they available?

Well, it turns out that they indeed are available...in a MIB called:CISCO-UNIFIED-COMPUTING-ADAPTOR-MIB

Unfortunately, this MIB is not included in the CISCO-UCS-Manager template. So what to do now? Well, there are a couple of strategies...

1. Add a new Discovery Rule to the (cloned) Cisco UCS Manager template. 
2.  Create a new template for the adaptor mib using a tool called mib2zabbix.

I tried to do #1 first, but had issues because the discover rule needed an LLD Macro and I wasn't sure how, syntactically, to create the Discovery Rule properly. My attempts at doing so, failed to produce any results when I tested the rule.

 

I went to pursue #2, which led me down an interesting road. First, the mib2zabbix tool requires the net-snmp package to be installed. And on CentOS, this package alone will not work - you also have to install net-snmp-util package to get the utilities like snmptranslate that you need.

The first time I ran mib2zabbix, it produced a template that I "knew" was not correct. I didn't see any of the crc objects in the template at all.  I did some additional research, and found that for mib2zabbix to work correctly, there has to be a correct "mib search path". 

To create the search path, you create a ".snmp" folder in your home directory, and in that directory, you create an snmp.conf file. This file looked as follows for me to be able to run snmptranslate and mib2zabbix "properly".

 

mibdirs +/usr/share/snmp/mibs/cisco/v2
mibdirs +/usr/share/snmp/mibs/cisco/ucs-C-Series-mibs
mibdirs +/usr/share/snmp/mibs/cisco/ucs-mibs

Zabbix to BigPanda Webhook Integration

Background
BigPanda has made its way into the organization. I wasn't sure at first why, given that there's no shortage of Network Monitoring OSS / EMS systems in play. 

Many vendors use their own EMS. VMware for example, uses VROPS (vRealize Operations Suite - now known as Aria Operations). So there is and has been a use case for consolidating this information from these disparate monitoring systems into a "Northbound" system. 

So that's what BigPanda is, I guess. It was pitched as a Northbound system. It does not seem to be very mature, and it is simpler to use than most of them (based on limited inspection and reading). But the business case pitch is that it has an Artificial Intelligence rules engine that provides superior correlation, and if this is true, it could certainly make it a northbound system worthy of consideration.

So - that is why we stepped in to integrate Zabbix with BigPanda. We already have VROPS as our "authoritative" monitoring system for all things VMWare. Our team, which does use this VROPS, does not own and manage that platform (another group does). I believe they use it to monitor the vCenters, the hypervisors, and datastores.  I don't think they're using it to monitor tenant workloads (virtual machines running on the hypervisors).

Our Zabbix platform, which we manage ourselves, is a "second layer of monitoring" behind VROPS. It manages only the VMWare Hypervisors along with some targeted specific virtual machines we run (load balancers, cloud management platform VMs, et al).  The BigPanda team wanted to showcase the ability to correlate information from Zabbix and VROPS, so we volunteered to integrate the two systems. 

Note: It is critical when integrating, that these integration steps be done in precisely this order!!!

Integration Steps

Setting up the Media Type

First, you need to "create" a Media Type - and this means Importing one, not creating one. There are two buttons when you click Media Type, "Create" and "Import". Because the Media Type has already been crafted, we will use "Import". The BigPanda Media Type, which is classified as a Webhook media type, is available for download, and you can find this (json) file at the following link: https://docs.bigpanda.io/docs/zabbix

When you import this webhook media type,  you have the option to "Update Existing" or "Create New". The first time, of course, requires "Create New" but any subsequent updates to the webhook would utilize the "Update Existing" button. 

After the media type has been created, everything will auto-populate. The Media Type tab will have a name (BigPanda in this case), a Type (Webhook), and a set of parameters. Most of these can be left alone, but four of them will need to be changed to literal values of macros (literal values for initial testing is recommended): BP_app_key, BP_endpoint, BP_token - and the zabbix url (which is at the bottom and out of view in the screenshot example below).

Setting Up the User Group

Next, you will create a User Group. The main reason for creating a (new) Big Panda user group, is that you can restrict the access of the Hosts that Big Panda has access to. If you wanted to allow Big Panda to have free roam access to all monitored hosts, then you probably could use one of the other host groups available. We wanted Big Panda to only receive alerts for specific hosts (hypervisors, test VMs, etc) so this was the justification for creating a new and separate Big Panda group. In the Host Permissions, we give this new user group Read access to these host groups.

Below is an example of what this group looks like.

Now, one thing worth looking at in this example, is the fact that the newly created User Group has Debug disabled. But there is a separate Debug Enabled group which does have Debug enabled, and any groups that we want to be debugged, can simply be slipped into this group.  There will be more on debugging later. Another thing worth mentioning, is that we did NOT enable FrontEnd access for this user group. This is an integration outbound, and we don't expect a Big Panda user / group to be logging into the UI.

Setting Up the User

Next, we create the User. Users need to have a Media Type, and are placed in User Groups which is why the Media Type and User Groups were created BEFORE the user.  Below, is an example of how the user is defined:

Notice that the user is mapped into the bigpandaservice User Group that we created in the previous step, which is why the User Group was pre-created in a previous step.

Now, after we establish the user fields, it is critically important to attach the User to the Media Type. Without this mapping, the alerts from Zabbix WILL NOT SEND!!!

After this Update button is hit, it is wise to verify and double-verify that this Media Type sticks - in our case, it did not and we had to remove the user and re-create it for some reason.

The final step in configuration is to create a Trigger Action on the Media Type. This is how that looks:

Next, you can click on Media Type, and select the "Test" button next to BigPanda. If you don't fill in the umpteen fields, and leave them as macros, with just the 4 fields we configured in the Media Type (endpoint, api key, token and zabbix url), the Test button "should" produce a 201 result, but you may get a json parse error because no actual data was sent. This is okay.

If the 201 exists, the Big Panda should receive the test alert. But this does not mean that the trigger is firing!!! The step to be taken after the Media Type "Test" button, is to generate an alert condition on the hosts that the Big Panda host group has access to, and make sure that Big Panda receives it!

Debugging & Troubleshooting

Troubleshooting requires making sure that all of these configuration steps were taken properly. This Webhook integration is all about mappings - users to user groups, users to media types, trigger definitions, host groups that are correct, etc.

When it comes to debugging, the debugging for a Webhook occurs within the Webhook!!!

The BigPanda Webhook, meaning the json file you imported, if you click on the Webhook you can see this json! In the screenshot below, notice the field called "script"...

If you were to click the "pencil" icon to the right, it will open up the entire webhook source code, which in this case is written in JavaScript.  

Now, you will notice that the BigPanda Webhook is sending messages to the Zabbix log at Level 4. The problem is, most people shouldn't be using Level 4 in their Zabbix logging (in zabbix_server.cfg file). It is too voluminous and makes debugging impossible if you are watching or tailing the log looking for webhook-specific messages. 

What I did, for testing and debugging, was to use a level that allows me to see the Webhook information without having to comb through a mountain of Zabbix debug information that you would normally see at Level 4 (Debug level). You will see in the screenshot below, that I commented out the "level 4" and replaced it with "level 2" - temporarily of course, until I could make sure that the Webhook was working properly. This example below, of course is just that: an example of how you can more simply debug the webhook. There are more lines in this code that I made these kinds of changes to, but the screenshot gives you an example of how it's done.

So hopefully that helps anyone wanting to get the BigPanda Webhook working in Zabbix, or for that matter, these steps should be helpful for any Webhook integration (i.e. Slack, Discord, et al).

Fixing Clustering and Disk Issues on an N+1 Morpheus CMP Cluster

I had performed an upgrade on Morpheus which I thought was fairly successful. I had some issues doing this upgrade on CentOS 7 because it was designated EOL and the repositories were archived, but I worked through that and it seemed everyone was using the system just fine.

Today, however, I had someone contact me to tell me that they provisioned a virtual machine, but it was stuck in an incomplete "Provisioning" state (a state that has a blue icon with a rocketship in it). The VM was provisioned on vCenter and working, but the state in Morpheus never set to "Finalized".

I couldn't figure this out, so I went to the Morpheus help site and I discovered that I myself had logged a ticket on this issue quite a while back. It turned out that the reason the state never flipped in that case, was because the clustering wasn't working properly.

So I checked RabbitMQ. It looked fine.

I checked MySQL and Percona, and I suspected that perhaps the clustering wasn't working properly. In the process of restarting the VMs, one of the virtual machines wouldn't start. I had to do a bunch of Percona advanced troubleshooting to figure out that I needed to do a wsrep recover commit before I could start the system and have it properly join the cluster. 

The NEXT problem was that Zabbix was screeching about these Morpheus VMs using too much disk space. It turned out that the /var file system was 100% full - because of ElasticSearch. Fortunately I had an oversized /home directory, and was able to do an rsync of the elasticsearch directory over to /home and re-link it.

But this gets to the topic of system administration with respect to disks.

First let's start with some KEY commands you MUST know:

>df -Th 

This command (disk free = df) shows how much space is used in human readable format, but with the mountpoint and file system type. This tells you NOTHING about the physical disks though!

>lsblk -f

This command (list block device) will give you the physical disk, the mountpoint, the uuid and any labels. It is a device specific command and doesn't show you space consumption.

>fdisk -l

I don't really like this command that much because of the output formatting. But it does list disk partitions and related statistics.

Some other commands you can use are:

>sudo file -sL /dev/sda3

the -s flag enables reading of block or character files and -L enables following of symlinks:

>blkid /dev/sda3

Similar command to lsblk -f above.