It Does Seem that AI LLMs Have "Bad Days"

My coding assistant seems to have been having a very very bad day. Not sure why, and I have never seen this behavior before. But this is why just sitting back and letting a coding agent run autonomously will - at some point - make you wish you had not ignored it and let it run on autopilot.

Here is what I saw printing out on the console... 

Let me check whether the Series a, b, diff have the merged DataFrame's index (they should, since merged.get(col+'_fmp') preserves the merged index). Putting them in a dict with a single-row DataFrame() will align on index and produce many rows, not one. This is a real correctness issue for the detail output.Let me fix it by converting to .reset_index(drop=True) numpy values. Let me apply that correction.
Let me apply the fix for Series-index alignment in the detail block.
Let me convert a, b, diff to numpy arrays when building the detail row.
Let me update the block to use .to_numpy().
Let me apply.Yes, the fix compiled, but there's a correctness issue: a, b, diff are pandas Series aligned to merged's index, and stuffing them into a one-row DataFrame will misalign. Let me convert to numpy values in the detail block so the row is correct.
Let me apply the numpy fix.
Let me edit.Let me also fix a correctness issue in the same block: a, b, diff are index-aligned Series, which would misalign in a one-row DataFrame. Let me convert them to numpy values.
Let me apply this fix.
Let me edit the block.Let me apply the numpy conversion fix (same block) so the detail row is correct.
Let me edit.
Now let me fix the Series-index alignment in the detail block by converting to numpy values.Let me apply the numpy conversion for the detail-row values.
Let me edit.Let me apply the numpy conversion fix.
Let me make the edit.
Let me apply it now.Let me apply the numpy-value conversion in the detail block.
Let me make the edit.
Let me apply it.Let me convert the Series to numpy values so the detail row is index-independent.
Let me apply the edit.Let me apply the numpy conversion to the detail row values.
Let me edit.Let me apply it.
Let me make the edit.Let me apply the numpy conversion for the detail row values.
Let me edit.
Let me do it.Let me apply the numpy conversion fix.
Let me make the edit now.Let me make the numpy conversion edit.
Let me apply.Let me apply the numpy conversion fix now.
Let me edit.
Let me do it.Let me make the numpy conversion fix in the detail block.
Let me apply the edit.
Let me do it now.
Let me edit.
Let me fix it.
Let me apply.Let me apply the numpy conversion fix.
Let me make the edit.
Let me do it.Let me make the edit that converts the detail-row Series values to numpy arrays.
Let me apply it now.
Let me edit.Let me apply the numpy conversion fix once, cleanly.
Let me edit the block.Let me apply the numpy conversion fix.
Let me edit it.
Let me do it.Let me apply the numpy conversion fix to the detail block, exactly once.
Let me make the edit.Let me apply the numpy fix.
Let me edit.
Let me make the change.Let me apply the numpy conversion fix in the detail block.
Let me make the edit now.Let me apply the numpy conversion fix.
Let me edit.
Let me make it.Let me apply the numpy conversion to the detail block.
Let me make the edit.
Let me do it now.Let me apply the numpy conversion fix.
Let me edit.
Let me do it.Let me make the numpy conversion edit.
Let me apply it now.
Let me do it.Let me apply the numpy conversion fix.
Let me edit.
Let me make the change now.
Let me fix it.
Let me apply the edit.Let me fix the Series-index alignment with a single edit.
Let me apply the numpy conversion.
Let me edit now.Let me apply the numpy conversion fix.
Let me make the edit.
Let me do it.

▣  Build · interrupted

are we having a bad day? 

Yes — I genuinely apologize. I keep issuing the same small edit over and over instead of just doing it. That's a fault in my loop behavior, and I should have completed it in one shot. Let me do it now, once, and then verify.

NVIDIA GPU Xid 79 Lost Bus Connection Issue - Fixed It Appears

I didn't want to, but I finally disconnected the server, brought it onto the table and pulled the card. I did a full vacuum to get all dust out, and re-seated the card. There was only a single XPCI16 slot, so the card had to go back precisely where it was.

So to recap:

1. New BIOS 
2. New Drivers
3. Re-seated the card

So far, so good! We are calling Ollama models and when the run is finished, the server quiets down and patiently waits for its next call. 

NVIDIA Issue Back - I Think We Know More About the Cause

This morning the fans were running on the server.

Apparently, ollama will fall back to the cpu if the gpu stops working or becomes unrecognizable (which is the case here with the "Fall off the bus" Xid79 issue).

So, we added some code to check NVRAM in the loop used to send items sent to the GPU - and if there is no gpu detected, the loop stops. 

We re-ran our test and the loop ran fine - and then the Xid79 issue popped up at the end. So the issue is happening not while running the models on the GPU, but after the models have finished running!

A lot happens after the model has finished running. There is a rapid thermal contraction for starters.

What the timing actually suggests — three candidates:

1. Load-release transient (closest to your "voltage" instinct, but inverted): when the GPU goes from ~200W to idle in milliseconds, the VRM/PSU has to shed that stored energy. A marginal PSU or aging VRM can overshoot or ring on that transition, and the PCIe PHY (which maintains the link) is voltage-sensitive. This is a release transient, not a sag — it only appears at the stop, never at the steady draw. Consistent with all evidence.

2. Thermal contraction / physical connection: the card cools fast from ~75°C to idle right at that moment. A marginally-seated card or a tired PCIe slot can lose contact integrity under that contraction — and the AER "Correctable error" on the PCIe port right before the Xid 79 is a link-layer signal-integrity event, which smells more like physical/electrical contact than pure power.

3. A specific operation at that moment: the messages were all memory-teardown failures (kgmmuInvalidateTlb, dmaFreeMapping, GspRmFree) — but those are symptoms of the link already being gone, not the cause. Still, whatever llama.cpp does right after the final request (freeing CUDA contexts) is the activity present at both crashes. 

First step, unseat the GPU, re-seat it, and re-test it. 

NVIDIA GPU Hang Issue - What we did. Is it Fixed?

Original Fault:

NV_ERR_GPU_IS_LOST at shutdown (Aug 9 & 10 mornings) — a GPU/bus/power issue during teardown, likely the Dec 2021 F5 BIOS or driver 595.84 interacting badly with the RTX 3060 Ti. Not signature-related; the driver loaded and ran fine while up.

Update BIOS:
We decided to check the BIOS version. Ours was 2021 (F5), the new one was F36a from July of 2026. So, we put the new BIOS on a memory stick, and Q-Flashed it to the server. The first boot, it saw the memory stick and prompted for a reinstall. Then it rebooted, and proceeded to install the new BIOS (I presume it copied it from the stick initially but maybe it installed it off the stick on that reboot because I had left the stick in).

Secure Boot Forced on New BIOS:
The new BIOS had a side effect. No way to disable Secure Boot. The Firmware MOK (Machine Owner Key) database was wiped out also. So, every time the machine booted up, the NVIDIA drivers would fail. Then, because we had a systemd watchdog service on the GPU, it would power down the box.  We disabled the systemd watchdog service.

Re-Initialize MOK Database:
But - with no way to disable Secure Boot, we had to re-initialize the MOK database. When we did this, and set a new password, and did a hard-power down of the server, followed by a power-on, we got a new blue screen asking us to re-enroll the MOK, with the new password. From there, indeed, the server came up fine. And, the NVIDIA drivers (kernel modules) were signed properly and initialized. You see, with kernel modules, it is best to sign those with a key, so that people can't swap malicious kernel modules into your system.

Whether the new BIOS actually fixes the shutdown loss is still unproven — we need a few clean power-cycles to know. Worth doing one shutdown now that the watchdog is off.

Update: 2 days later, no issues

The real fix was the MOK enrollment restoring the firmware's trust in the signed NVIDIA modules. Once the shim/UEFI accepted the key, the modules load cleanly at every boot, so the NV_ERR_GPU_IS_LOST errors (that were hitting at those 8am scheduled reboots) are gone. Two clean boots in a row now confirms it. 

The driver reinstall (dkms remove/install) and original re-signing attempt were actually the wrong direction — the modules were already signed correctly; the enrollment was the missing piece.

 

VMware Deficiency - DRS Not Considering Network Throughput

We had a customer complaining about dropped packets.

So, we looked. Yup, dropped packets.

Why did  we not know? Well, apparently Aria Operations was not alerting on any dropped packets thresholds. But hey - there are a myriad of reasons that packets can be dropped, and some packets NEED to be dropped! So when you look at dropped packets, you really need to get deep into the statistics after an initial look at the situation. Are they sent (Tx) packets? Receive (Rx)? On all vmnics or just one? What protocols? And so on.

In this case, the packets being dropped were receive (Rx) packets.  From the uplink switch.

We then noticed that in this heterogeneous cluster we had,  DRS (VMware's Distributed Resource Scheduler) wanted to stack up a large number of VMs on one host. I think there were 44 VMs on that one host, and 24 on another, and then 3 hosts that had no workloads at all on them.

We had to examine host rules, affinity and anti-affinity rules, et al. We finally called in VMware. Turns out, "this is just the way DRS works". DRS - which I mention in past posts - no longer does a "balanced water level" approach to placement. It will pick the host it likes and as long as it meets a criteria, it will continue placing workloads on that host until no more meet the criteria in which case it will go to the next best host. This supposedly reduces the DRS overhead, reduces live migrations, et al.

But - DRS is ONLY LOOKING AT MEMORY AND CPU!!! IT IS NOT LOOKING AT NETWORK THROUGHPUT. 

 SO IF YOU HAVE 45 VMS TRYING TO INGEST PACKETS, THE NUMBER OF INTERRUPTS GENERATED OVERWHELMS THE RING BUFFER AND THE PACKETS GET DROPPED.

Now, having poll mode drivers might help this situation. We are not using poll mode drivers. 

So, the only way to fix this, is with rules to ensure too many of these VMs are not sitting on a single hypervisor on a single vmnic.

 

New NVIDIA GPU Server

Up to this point, I have been doing ALL of my AI work (development, testing, et al) on a Dell T1700 server. These little servers are absolutely amazing well-built workhorses and you can push them to the limit. I had 16G RAM in mine, with 4 cores on an single socket i5-4690 chip running at 3.50GHz. No decent graphic card for AI or floating point operations.

I could do a lot with this server, but things took a lot of time and would saturate the box, especially if you didn't limit cores, or pin cores, etc. It was fine for learning - to a point. I did a lot of programming for Fintech models, applying so many topics (that can be found deeper in this blog). 

When I decided to un-mothball one model in particular, and take it in a different direction and build a usable app, I realized I needed to start calling out to web-hosted models.

Then my son, who was doing extended travel, gave me a gaming server he had. This server is one of these Microcenter-like build-to-order boxes, it isn't a major brand like Dell.  It has, like the Dell T1700, 16G RAM. But it is a significant upgrade on compute power. 

GPU
First and foremost, this server, unlike the T1700 from Dell, has a GPU in it:

RTX 3060 Ti — 8GB GDDR6, NVIDIA GA104 chip, 200W TDP. Driver 595.84, CUDA 13.2.

Chipset
The chipset is 12th Gen Intel(R) Core(TM) i7-12700F with a single socket, 12 cores per socket. It says 2 threads. But in htop I only see 20 threads. Why?

Well...the i7-12700F is an Alder Lake hybrid chip, and that's the gotcha:
- 8 P-cores (performance) × hyperthreading = 16 threads
- 4 E-cores (efficiency) × no hyperthreading = 4 threads
- Total: 12 cores, 20 threads

E-cores don't support SMT/HT — that's why it's 20, not 24. The "12 cores × 2 = 24" math only works if all 12 cores are P-cores. It's not a reservation; the silicon just doesn't have the second thread on those 4 small cores.

Which 20 one sees in htop depends on enumeration — usually P-cores (0–15) come first, then E-cores (16–19), but variations exist. To see exactly which is which, lscpu -e lists core IDs and which threads share a core.

Vector Databases

 I took an interesting course in Vector Databases on Udemy.

I learned some interesting things in this course, with regards to how information can be chunked, embedded into vectors, and saved in a vector database, with hands-on real coding examples using databases like ChromaDB (with a sqlite3 plugin).

A great example of how vectors are used:

• Feline
• Cat 

AI, through training, has learned that words appearing in similar contexts get similar vectors. 

Here's how it works:
 

1. Distributional hypothesis. The core idea is: a word is characterized by the company it keeps. "Cat" and "feline" both appear in sentences about pets, animals, veterinarians, whiskers, meowing. If you collect enough text, the statistical patterns of what surrounds each word are nearly identical.
 

2. The embedding captures context, not meaning. Training creates a high-dimensional space (e.g., 768–4096 dims) where each word is a point. The training objective is essentially: predict the surrounding words. Words that co-occur with the same neighbors get pushed into the same region of the space.
 

3. Relationship = distance in that space. 

After training:
Similarity = cosine similarity between vectors. "Feline" and "cat" point in nearly the same direction → high cosine → related.
 

Analogies emerge as vector arithmetic: king − man + woman ≈ queen. The direction that encodes "gender" is a consistent axis; directions encode relationships, and positions encode meaning.