NVIDIA Issue Back - I Think We Know More About the Cause

This morning the fans were running on the server.

Apparently, ollama will fall back to the cpu if the gpu stops working or becomes unrecognizable (which is the case here with the "Fall off the bus" Xid79 issue).

So, we added some code to check NVRAM in the loop used to send items sent to the GPU - and if there is no gpu detected, the loop stops. 

We re-ran our test and the loop ran fine - and then the Xid79 issue popped up at the end. So the issue is happening not while running the models on the GPU, but after the models have finished running!

A lot happens after the model has finished running. There is a rapid thermal contraction for starters.

What the timing actually suggests — three candidates:

1. Load-release transient (closest to your "voltage" instinct, but inverted): when the GPU goes from ~200W to idle in milliseconds, the VRM/PSU has to shed that stored energy. A marginal PSU or aging VRM can overshoot or ring on that transition, and the PCIe PHY (which maintains the link) is voltage-sensitive. This is a release transient, not a sag — it only appears at the stop, never at the steady draw. Consistent with all evidence.

2. Thermal contraction / physical connection: the card cools fast from ~75°C to idle right at that moment. A marginally-seated card or a tired PCIe slot can lose contact integrity under that contraction — and the AER "Correctable error" on the PCIe port right before the Xid 79 is a link-layer signal-integrity event, which smells more like physical/electrical contact than pure power.

3. A specific operation at that moment: the messages were all memory-teardown failures (kgmmuInvalidateTlb, dmaFreeMapping, GspRmFree) — but those are symptoms of the link already being gone, not the cause. Still, whatever llama.cpp does right after the final request (freeing CUDA contexts) is the activity present at both crashes. 

First step, unseat the GPU, re-seat it, and re-test it. 

New NVIDIA GPU Server

Up to this point, I have been doing ALL of my AI work (development, testing, et al) on a Dell T1700 server. These little servers are absolutely amazing well-built workhorses and you can push them to the limit. I had 16G RAM in mine, with 4 cores on an single socket i5-4690 chip running at 3.50GHz. No decent graphic card for AI or floating point operations.

I could do a lot with this server, but things took a lot of time and would saturate the box, especially if you didn't limit cores, or pin cores, etc. It was fine for learning - to a point. I did a lot of programming for Fintech models, applying so many topics (that can be found deeper in this blog). 

When I decided to un-mothball one model in particular, and take it in a different direction and build a usable app, I realized I needed to start calling out to web-hosted models.

Then my son, who was doing extended travel, gave me a gaming server he had. This server is one of these Microcenter-like build-to-order boxes, it isn't a major brand like Dell.  It has, like the Dell T1700, 16G RAM. But it is a significant upgrade on compute power. 

GPU
First and foremost, this server, unlike the T1700 from Dell, has a GPU in it:

RTX 3060 Ti — 8GB GDDR6, NVIDIA GA104 chip, 200W TDP. Driver 595.84, CUDA 13.2.

Chipset
The chipset is 12th Gen Intel(R) Core(TM) i7-12700F with a single socket, 12 cores per socket. It says 2 threads. But in htop I only see 20 threads. Why?

Well...the i7-12700F is an Alder Lake hybrid chip, and that's the gotcha:
- 8 P-cores (performance) × hyperthreading = 16 threads
- 4 E-cores (efficiency) × no hyperthreading = 4 threads
- Total: 12 cores, 20 threads

E-cores don't support SMT/HT — that's why it's 20, not 24. The "12 cores × 2 = 24" math only works if all 12 cores are P-cores. It's not a reservation; the silicon just doesn't have the second thread on those 4 small cores.

Which 20 one sees in htop depends on enumeration — usually P-cores (0–15) come first, then E-cores (16–19), but variations exist. To see exactly which is which, lscpu -e lists core IDs and which threads share a core.