NVIDIA GPU Hang Issue - What we did. Is it Fixed?

Original Fault:

NV_ERR_GPU_IS_LOST at shutdown (Aug 9 & 10 mornings) — a GPU/bus/power issue during teardown, likely the Dec 2021 F5 BIOS or driver 595.84 interacting badly with the RTX 3060 Ti. Not signature-related; the driver loaded and ran fine while up.

Update BIOS:
We decided to check the BIOS version. Ours was 2021 (F5), the new one was F36a from July of 2026. So, we put the new BIOS on a memory stick, and Q-Flashed it to the server. The first boot, it saw the memory stick and prompted for a reinstall. Then it rebooted, and proceeded to install the new BIOS (I presume it copied it from the stick initially but maybe it installed it off the stick on that reboot because I had left the stick in).

Secure Boot Forced on New BIOS:
The new BIOS had a side effect. No way to disable Secure Boot. The Firmware MOK (Machine Owner Key) database was wiped out also. So, every time the machine booted up, the NVIDIA drivers would fail. Then, because we had a systemd watchdog service on the GPU, it would power down the box.  We disabled the systemd watchdog service.

Re-Initialize MOK Database:
But - with no way to disable Secure Boot, we had to re-initialize the MOK database. When we did this, and set a new password, and did a hard-power down of the server, followed by a power-on, we got a new blue screen asking us to re-enroll the MOK, with the new password. From there, indeed, the server came up fine. And, the NVIDIA drivers (kernel modules) were signed properly and initialized. You see, with kernel modules, it is best to sign those with a key, so that people can't swap malicious kernel modules into your system.

Whether the new BIOS actually fixes the shutdown loss is still unproven — we need a few clean power-cycles to know. Worth doing one shutdown now that the watchdog is off.

Update: 2 days later, no issues

The real fix was the MOK enrollment restoring the firmware's trust in the signed NVIDIA modules. Once the shim/UEFI accepted the key, the modules load cleanly at every boot, so the NV_ERR_GPU_IS_LOST errors (that were hitting at those 8am scheduled reboots) are gone. Two clean boots in a row now confirms it. 

The driver reinstall (dkms remove/install) and original re-signing attempt were actually the wrong direction — the modules were already signed correctly; the enrollment was the missing piece.

 

New NVIDIA GPU Server

Up to this point, I have been doing ALL of my AI work (development, testing, et al) on a Dell T1700 server. These little servers are absolutely amazing well-built workhorses and you can push them to the limit. I had 16G RAM in mine, with 4 cores on an single socket i5-4690 chip running at 3.50GHz. No decent graphic card for AI or floating point operations.

I could do a lot with this server, but things took a lot of time and would saturate the box, especially if you didn't limit cores, or pin cores, etc. It was fine for learning - to a point. I did a lot of programming for Fintech models, applying so many topics (that can be found deeper in this blog). 

When I decided to un-mothball one model in particular, and take it in a different direction and build a usable app, I realized I needed to start calling out to web-hosted models.

Then my son, who was doing extended travel, gave me a gaming server he had. This server is one of these Microcenter-like build-to-order boxes, it isn't a major brand like Dell.  It has, like the Dell T1700, 16G RAM. But it is a significant upgrade on compute power. 

GPU
First and foremost, this server, unlike the T1700 from Dell, has a GPU in it:

RTX 3060 Ti — 8GB GDDR6, NVIDIA GA104 chip, 200W TDP. Driver 595.84, CUDA 13.2.

Chipset
The chipset is 12th Gen Intel(R) Core(TM) i7-12700F with a single socket, 12 cores per socket. It says 2 threads. But in htop I only see 20 threads. Why?

Well...the i7-12700F is an Alder Lake hybrid chip, and that's the gotcha:
- 8 P-cores (performance) × hyperthreading = 16 threads
- 4 E-cores (efficiency) × no hyperthreading = 4 threads
- Total: 12 cores, 20 threads

E-cores don't support SMT/HT — that's why it's 20, not 24. The "12 cores × 2 = 24" math only works if all 12 cores are P-cores. It's not a reservation; the silicon just doesn't have the second thread on those 4 small cores.

Which 20 one sees in htop depends on enumeration — usually P-cores (0–15) come first, then E-cores (16–19), but variations exist. To see exactly which is which, lscpu -e lists core IDs and which threads share a core.

Vector Databases

 I took an interesting course in Vector Databases on Udemy.

I learned some interesting things in this course, with regards to how information can be chunked, embedded into vectors, and saved in a vector database, with hands-on real coding examples using databases like ChromaDB (with a sqlite3 plugin).

A great example of how vectors are used:

• Feline
• Cat 

AI, through training, has learned that words appearing in similar contexts get similar vectors. 

Here's how it works:
 

1. Distributional hypothesis. The core idea is: a word is characterized by the company it keeps. "Cat" and "feline" both appear in sentences about pets, animals, veterinarians, whiskers, meowing. If you collect enough text, the statistical patterns of what surrounds each word are nearly identical.
 

2. The embedding captures context, not meaning. Training creates a high-dimensional space (e.g., 768–4096 dims) where each word is a point. The training objective is essentially: predict the surrounding words. Words that co-occur with the same neighbors get pushed into the same region of the space.
 

3. Relationship = distance in that space. 

After training:
Similarity = cosine similarity between vectors. "Feline" and "cat" point in nearly the same direction → high cosine → related.
 

Analogies emerge as vector arithmetic: king − man + woman ≈ queen. The direction that encodes "gender" is a consistent axis; directions encode relationships, and positions encode meaning.

Removing Two Stale Macro Features

 

Removing Two Stale Macro Features

The model was trained on 11 features, two of which were macroeconomic sentiment indicators sourced from FRED. On inspection, both turned out to be monthly series — meaning they only update once a month and carry a publication lag on top of that. Despite this, the model had assigned them significant feature importance, essentially learning to lean on data that wasn't meaningfully changing day to day and wasn't even fully available in real time when historical training data was constructed.

Removing them dropped the feature set from 11 to 9. With those features gone, the model redistributed weight toward momentum and the remaining daily macro indicators in a more sensible way. Validation rank correlations improved on two of the three prediction horizons after the change. The two daily macro features that remained — VIX and treasury spread — are genuinely responsive to market conditions and carry the macro signal adequately on their own.

Both changes were low risk given that model predictions are used for monitoring purposes in this system rather than directly driving trading decisions.

Fixing a Train/Serve Skew in Sentiment Residuals

 

Fixing a Train/Serve Skew in Sentiment Residuals

The signal generation process uses a technique called sentiment residualization — essentially, we remove the portion of the FinBERT sentiment score that can be explained by price momentum alone, leaving behind only the genuine sentiment surprise. A stock that has been running up for 20 days will naturally attract positive news coverage, so we want to isolate the sentiment signal that exists above and beyond what the price action would predict.

The problem was subtle. During training, the residual model was fitted on tens of thousands of rows spanning months of history. But at inference time, the same calculation was being refitted fresh each day on whatever small universe of stocks passed the daily filters — typically around 30 stocks. That's a very different statistical population, which meant the sentiment residuals being fed into the composite ranking signal weren't quite the same thing the model had learned from during training. Classic train/serve skew.

The fix was straightforward — serialize the residual model coefficients to disk at the end of each training run and load those fixed coefficients at inference time rather than refitting. Now the definition of sentiment surprise is consistent from training through to live signal generation.

Adding a Layer of LLM as a Final Trading Gate

When we did our back-testing logic we really started to get a full understanding of our features and what mattered and influenced returns.

 The *ONLY* feature that made ANY difference, was the finbert residual, which attempts to ferret out "true sentiment" from momentum. 

The uncomfortable but honest conclusion: the XGB model as currently built is mostly a complicated way of rediscovering momentum, with a thin layer of sentiment on top.

Nonetheless, there is a true signal there, and the signal IS news-related. Which means it is worth investigating. For some length of time anyway. 

After all, you don't want a "news sentiment" model, that can't use news sentiment!

The Iran situation has caused huge volatility in a negative way, and this affected my trading badly as we picked stocks that were headed south. Our balance right now, is 89K from the initial 100K, so over 2 months, we have burned 11K of capital. Good thing it is paper trading.

I added two LLM aspects to the model:

1. agent_review.py which is a standalone agent analysis tool. It reads the signal file from generate_news_signal.py and searches for recent (keyword: recent) news on each candidate, and asks an LLM to recommend which stocks to buy with reasoning.

2. The portfolio_manager.py also looks at the current vix value - for the day (not the vix_0d value that was tied to the article headline date which is x days in the past), and also consults an LLM to check and see if the climate looks right to buy the selected stocks.

Both of these are designed to avoid mistakes buying. And, using an LLM is easier when you give it small tasks. Less tokens consumed, the task is more focused. Trying to give LLMs huge chunks of data to process can cause timeouts (504 errors), and strange results.

I do see a of trades being vetoed in our falling markets right now, so this is working and perhaps was added a bit late in the game. But it can maybe protect our capital so that when the regime shifts, we can start to get back up to the original 100K and then turn some profit. 

 

 

 

Backtesting - Decile Testing and Monotonocity - Part II

So now that we understand decile testing and monotonicity, we can run this on ALL features to see how they look.... 

And THIS is why the back-test was using "just" lag_ret_20d instead of the model predictions.

So the Macro features - just adding unnecessary noise, and no value!?

Is the News Sentiment adding any value at all? Maybe, they're ranked #4 and #8, but we would need to try to combine them to see if they add any value at all. Keep in mind also, that the residual (finbert_signed_resid) ferrets out "true" sentiment from momentum (as discussed in an earlier blog post).

And - if you combine them, in what ratio for them to make an optimal combination? Should we combine just two? Or more?

You can see where this can go. You almost need a permutational approach.