I didn't want to, but I finally disconnected the server, brought it onto the table and pulled the card. I did a full vacuum to get all dust out, and re-seated the card. There was only a single XPCI16 slot, so the card had to go back precisely where it was.
So to recap:
So far, so good! We are calling Ollama models and when the run is finished, the server quiets down and patiently waits for its next call.
This morning the fans were running on the server.
Apparently, ollama will fall back to the cpu if the gpu stops working or becomes unrecognizable (which is the case here with the "Fall off the bus" Xid79 issue).
So, we added some code to check NVRAM in the loop used to send items sent to the GPU - and if there is no gpu detected, the loop stops.
We re-ran our test and the loop ran fine - and then the Xid79 issue popped up at the end. So the issue is happening not while running the models on the GPU, but after the models have finished running!
A lot happens after the model has finished running. There is a rapid thermal contraction for starters.
What the timing actually suggests — three candidates:
1. Load-release transient (closest to your "voltage" instinct, but inverted): when the GPU goes from ~200W to idle in milliseconds, the VRM/PSU has to shed that stored energy. A marginal PSU or aging VRM can overshoot or ring on that transition, and the PCIe PHY (which maintains the link) is voltage-sensitive. This is a release transient, not a sag — it only appears at the stop, never at the steady draw. Consistent with all evidence.
2. Thermal contraction / physical connection: the card cools fast from ~75°C to idle right at that moment. A marginally-seated card or a tired PCIe slot can lose contact integrity under that contraction — and the AER "Correctable error" on the PCIe port right before the Xid 79 is a link-layer signal-integrity event, which smells more like physical/electrical contact than pure power.
3. A specific operation at that moment: the messages were all memory-teardown failures (kgmmuInvalidateTlb, dmaFreeMapping, GspRmFree) — but those are symptoms of the link already being gone, not the cause. Still, whatever llama.cpp does right after the final request (freeing CUDA contexts) is the activity present at both crashes.
First step, unseat the GPU, re-seat it, and re-test it.
Original Fault:
NV_ERR_GPU_IS_LOST at shutdown (Aug 9 & 10 mornings) — a GPU/bus/power issue during teardown, likely the Dec 2021 F5 BIOS or driver 595.84 interacting badly with the RTX 3060 Ti. Not signature-related; the driver loaded and ran fine while up.
Update BIOS:
We decided to check the BIOS version. Ours was 2021 (F5), the new one was F36a from July of 2026. So, we put the new BIOS on a memory stick, and Q-Flashed it to the server. The first boot, it saw the memory stick and prompted for a reinstall. Then it rebooted, and proceeded to install the new BIOS (I presume it copied it from the stick initially but maybe it installed it off the stick on that reboot because I had left the stick in).
Secure Boot Forced on New BIOS:
The new BIOS had a side effect. No way to disable Secure Boot. The Firmware MOK (Machine Owner Key) database was wiped out also. So, every time the machine booted up, the NVIDIA drivers would fail. Then, because we had a systemd watchdog service on the GPU, it would power down the box. We disabled the systemd watchdog service.
Re-Initialize MOK Database:
But - with no way to disable Secure Boot, we had to re-initialize the MOK database. When we did this, and set a new password, and did a hard-power down of the server, followed by a power-on, we got a new blue screen asking us to re-enroll the MOK, with the new password. From there, indeed, the server came up fine. And, the NVIDIA drivers (kernel modules) were signed properly and initialized. You see, with kernel modules, it is best to sign those with a key, so that people can't swap malicious kernel modules into your system.
Whether the new BIOS actually fixes the shutdown loss is still unproven — we need a few clean power-cycles to know. Worth doing one shutdown now that the watchdog is off.
Update: 2 days later, no issues
The real fix was the MOK enrollment restoring the firmware's trust in the signed NVIDIA modules. Once the shim/UEFI accepted the key, the modules load cleanly at every boot, so the NV_ERR_GPU_IS_LOST errors (that were hitting at those 8am scheduled reboots) are gone. Two clean boots in a row now confirms it.
The driver reinstall (dkms remove/install) and original re-signing attempt were actually the wrong direction — the modules were already signed correctly; the enrollment was the missing piece.
Up to this point, I have been doing ALL of my AI work (development, testing, et al) on a Dell T1700 server. These little servers are absolutely amazing well-built workhorses and you can push them to the limit. I had 16G RAM in mine, with 4 cores on an single socket i5-4690 chip running at 3.50GHz. No decent graphic card for AI or floating point operations.
I could do a lot with this server, but things took a lot of time and would saturate the box, especially if you didn't limit cores, or pin cores, etc. It was fine for learning - to a point. I did a lot of programming for Fintech models, applying so many topics (that can be found deeper in this blog).
When I decided to un-mothball one model in particular, and take it in a different direction and build a usable app, I realized I needed to start calling out to web-hosted models.
Then my son, who was doing extended travel, gave me a gaming server he had. This server is one of these Microcenter-like build-to-order boxes, it isn't a major brand like Dell. It has, like the Dell T1700, 16G RAM. But it is a significant upgrade on compute power.
GPU
First and foremost, this server, unlike the T1700 from Dell, has a GPU in it:
RTX 3060 Ti — 8GB GDDR6, NVIDIA GA104 chip, 200W TDP. Driver 595.84, CUDA 13.2.
Chipset
The chipset is 12th Gen Intel(R) Core(TM) i7-12700F with a single socket, 12 cores per socket. It says 2 threads. But in htop I only see 20 threads. Why?
Well...the i7-12700F is an Alder Lake hybrid chip, and that's the gotcha:
- 8 P-cores (performance) × hyperthreading = 16 threads
- 4 E-cores (efficiency) × no hyperthreading = 4 threads
- Total: 12 cores, 20 threads
E-cores don't support SMT/HT — that's why it's 20, not 24. The "12 cores × 2 = 24" math only works if all 12 cores are P-cores. It's not a reservation; the silicon just doesn't have the second thread on those 4 small cores.
Which 20 one sees in htop depends on enumeration — usually P-cores (0–15) come first, then E-cores (16–19), but variations exist. To see exactly which is which, lscpu -e lists core IDs and which threads share a core.
I haven't been able to understand why the original chip pioneers, like Intel and AMD, have not been able to pivot in order to compete with NVidia (Stock Symbol: NVDA).
I know a few things, like the fact that when gaming became popular, NVidia made the graphics chips that had graphics acceleration and such. Graphics tend to draw polygons, and drawing polygons is geometric and trigonometric - which require floating point arithmetic (non-integer based mathematics). Floating point is difficult for a CPU to do, so much so that classical CPUs either offloaded or employed other tricks to do these kinds of computations.
Now, these graphics chips are the "rave" for AI. And Nvidia stock has gone through the roof while Intel and AMD have been left behind.
So what does an AI chip have, that is different from an older CPU?
CPUs use all three of these in some form or another, but an AI chip has all three of these in a highly optimized and accelerated design. Things like prediction (such as branching prediction), parallelism, etc. They're simply better at running "algorithms".
This link, by the way, from NVidia, discusses the distinction between Training and Inference:
https://blogs.nvidia.com/blog/difference-deep-learning-training-inference-ai/
CPUs, they were so bent on running Microsoft for so long, and emulating continuous revisions of instructions to run Windows (286-->386-->486-->Pentium--> and on and on), that they just never went back and "rearchitected" or came up with new chip architectures. They sat back and collected money, along with Microsoft, to give you incremental versions of the same thing - for YEARS.
When you are doing training for an AI model, and you are running algorithmic loops millions upon millions of times, the efficiency and time start to add up - and make a huge difference in $$$ (MONEY).
So the CPU companies, in order to "catch up", I think, with NVidia, would need to come up with a whole bunch of chip design software. Then there is the software kits necessary to develop to the chips. You also have the foundry (which uses manufacturing equipment, much of it custom per the design), etc. Meanwhile, NVidia has its rocket off the ground, with decreasing G forces (so to speak), which accelerates its orbit. It is easy to see why an increasing gap would occur.
But - when you have everyone (China, Russia, Intel, AMD, ARM, et al) all racing to catch up, they will at some point, catch up. I think. When NVidia slows down. We shall see.
