This morning the fans were running on the server.
Apparently, ollama will fall back to the cpu if the gpu stops working or becomes unrecognizable (which is the case here with the "Fall off the bus" Xid79 issue).
So, we added some code to check NVRAM in the loop used to send items sent to the GPU - and if there is no gpu detected, the loop stops.
We re-ran our test and the loop ran fine - and then the Xid79 issue popped up at the end. So the issue is happening not while running the models on the GPU, but after the models have finished running!
A lot happens after the model has finished running. There is a rapid thermal contraction for starters.
What the timing actually suggests — three candidates:
1. Load-release transient (closest to your "voltage" instinct, but inverted): when the GPU goes from ~200W to idle in milliseconds, the VRM/PSU has to shed that stored energy. A marginal PSU or aging VRM can overshoot or ring on that transition, and the PCIe PHY (which maintains the link) is voltage-sensitive. This is a release transient, not a sag — it only appears at the stop, never at the steady draw. Consistent with all evidence.
2. Thermal contraction / physical connection: the card cools fast from ~75°C to idle right at that moment. A marginally-seated card or a tired PCIe slot can lose contact integrity under that contraction — and the AER "Correctable error" on the PCIe port right before the Xid 79 is a link-layer signal-integrity event, which smells more like physical/electrical contact than pure power.
3. A specific operation at that moment: the messages were all memory-teardown failures (kgmmuInvalidateTlb, dmaFreeMapping, GspRmFree) — but those are symptoms of the link already being gone, not the cause. Still, whatever llama.cpp does right after the final request (freeing CUDA contexts) is the activity present at both crashes.
First step, unseat the GPU, re-seat it, and re-test it.
