VMware Deficiency - DRS Not Considering Network Throughput

We had a customer complaining about dropped packets.

So, we looked. Yup, dropped packets.

Why did  we not know? Well, apparently Aria Operations was not alerting on any dropped packets thresholds. But hey - there are a myriad of reasons that packets can be dropped, and some packets NEED to be dropped! So when you look at dropped packets, you really need to get deep into the statistics after an initial look at the situation. Are they sent (Tx) packets? Receive (Rx)? On all vmnics or just one? What protocols? And so on.

In this case, the packets being dropped were receive (Rx) packets.  From the uplink switch.

We then noticed that in this heterogeneous cluster we had,  DRS (VMware's Distributed Resource Scheduler) wanted to stack up a large number of VMs on one host. I think there were 44 VMs on that one host, and 24 on another, and then 3 hosts that had no workloads at all on them.

We had to examine host rules, affinity and anti-affinity rules, et al. We finally called in VMware. Turns out, "this is just the way DRS works". DRS - which I mention in past posts - no longer does a "balanced water level" approach to placement. It will pick the host it likes and as long as it meets a criteria, it will continue placing workloads on that host until no more meet the criteria in which case it will go to the next best host. This supposedly reduces the DRS overhead, reduces live migrations, et al.

But - DRS is ONLY LOOKING AT MEMORY AND CPU!!! IT IS NOT LOOKING AT NETWORK THROUGHPUT. 

 SO IF YOU HAVE 45 VMS TRYING TO INGEST PACKETS, THE NUMBER OF INTERRUPTS GENERATED OVERWHELMS THE RING BUFFER AND THE PACKETS GET DROPPED.

Now, having poll mode drivers might help this situation. We are not using poll mode drivers. 

So, the only way to fix this, is with rules to ensure too many of these VMs are not sitting on a single hypervisor on a single vmnic.

 

New NVIDIA GPU Server

Up to this point, I have been doing ALL of my AI work (development, testing, et al) on a Dell T1700 server. These little servers are absolutely amazing well-built workhorses and you can push them to the limit. I had 16G RAM in mine, with 4 cores on an single socket i5-4690 chip running at 3.50GHz. No decent graphic card for AI or floating point operations.

I could do a lot with this server, but things took a lot of time and would saturate the box, especially if you didn't limit cores, or pin cores, etc. It was fine for learning - to a point. I did a lot of programming for Fintech models, applying so many topics (that can be found deeper in this blog). 

When I decided to un-mothball one model in particular, and take it in a different direction and build a usable app, I realized I needed to start calling out to web-hosted models.

Then my son, who was doing extended travel, gave me a gaming server he had. This server is one of these Microcenter-like build-to-order boxes, it isn't a major brand like Dell.  It has, like the Dell T1700, 16G RAM. But it is a significant upgrade on compute power. 

GPU
First and foremost, this server, unlike the T1700 from Dell, has a GPU in it:

RTX 3060 Ti — 8GB GDDR6, NVIDIA GA104 chip, 200W TDP. Driver 595.84, CUDA 13.2.

Chipset
The chipset is 12th Gen Intel(R) Core(TM) i7-12700F with a single socket, 12 cores per socket. It says 2 threads. But in htop I only see 20 threads. Why?

Well...the i7-12700F is an Alder Lake hybrid chip, and that's the gotcha:
- 8 P-cores (performance) × hyperthreading = 16 threads
- 4 E-cores (efficiency) × no hyperthreading = 4 threads
- Total: 12 cores, 20 threads

E-cores don't support SMT/HT — that's why it's 20, not 24. The "12 cores × 2 = 24" math only works if all 12 cores are P-cores. It's not a reservation; the silicon just doesn't have the second thread on those 4 small cores.

Which 20 one sees in htop depends on enumeration — usually P-cores (0–15) come first, then E-cores (16–19), but variations exist. To see exactly which is which, lscpu -e lists core IDs and which threads share a core.

Vector Databases

 I took an interesting course in Vector Databases on Udemy.

I learned some interesting things in this course, with regards to how information can be chunked, embedded into vectors, and saved in a vector database, with hands-on real coding examples using databases like ChromaDB (with a sqlite3 plugin).

A great example of how vectors are used:

• Feline
• Cat 

AI, through training, has learned that words appearing in similar contexts get similar vectors. 

Here's how it works:
 

1. Distributional hypothesis. The core idea is: a word is characterized by the company it keeps. "Cat" and "feline" both appear in sentences about pets, animals, veterinarians, whiskers, meowing. If you collect enough text, the statistical patterns of what surrounds each word are nearly identical.
 

2. The embedding captures context, not meaning. Training creates a high-dimensional space (e.g., 768–4096 dims) where each word is a point. The training objective is essentially: predict the surrounding words. Words that co-occur with the same neighbors get pushed into the same region of the space.
 

3. Relationship = distance in that space. 

After training:
Similarity = cosine similarity between vectors. "Feline" and "cat" point in nearly the same direction → high cosine → related.
 

Analogies emerge as vector arithmetic: king − man + woman ≈ queen. The direction that encodes "gender" is a consistent axis; directions encode relationships, and positions encode meaning.

Removing Two Stale Macro Features

 

Removing Two Stale Macro Features

The model was trained on 11 features, two of which were macroeconomic sentiment indicators sourced from FRED. On inspection, both turned out to be monthly series — meaning they only update once a month and carry a publication lag on top of that. Despite this, the model had assigned them significant feature importance, essentially learning to lean on data that wasn't meaningfully changing day to day and wasn't even fully available in real time when historical training data was constructed.

Removing them dropped the feature set from 11 to 9. With those features gone, the model redistributed weight toward momentum and the remaining daily macro indicators in a more sensible way. Validation rank correlations improved on two of the three prediction horizons after the change. The two daily macro features that remained — VIX and treasury spread — are genuinely responsive to market conditions and carry the macro signal adequately on their own.

Both changes were low risk given that model predictions are used for monitoring purposes in this system rather than directly driving trading decisions.

Fixing a Train/Serve Skew in Sentiment Residuals

 

Fixing a Train/Serve Skew in Sentiment Residuals

The signal generation process uses a technique called sentiment residualization — essentially, we remove the portion of the FinBERT sentiment score that can be explained by price momentum alone, leaving behind only the genuine sentiment surprise. A stock that has been running up for 20 days will naturally attract positive news coverage, so we want to isolate the sentiment signal that exists above and beyond what the price action would predict.

The problem was subtle. During training, the residual model was fitted on tens of thousands of rows spanning months of history. But at inference time, the same calculation was being refitted fresh each day on whatever small universe of stocks passed the daily filters — typically around 30 stocks. That's a very different statistical population, which meant the sentiment residuals being fed into the composite ranking signal weren't quite the same thing the model had learned from during training. Classic train/serve skew.

The fix was straightforward — serialize the residual model coefficients to disk at the end of each training run and load those fixed coefficients at inference time rather than refitting. Now the definition of sentiment surprise is consistent from training through to live signal generation.

Adding a Layer of LLM as a Final Trading Gate

When we did our back-testing logic we really started to get a full understanding of our features and what mattered and influenced returns.

 The *ONLY* feature that made ANY difference, was the finbert residual, which attempts to ferret out "true sentiment" from momentum. 

The uncomfortable but honest conclusion: the XGB model as currently built is mostly a complicated way of rediscovering momentum, with a thin layer of sentiment on top.

Nonetheless, there is a true signal there, and the signal IS news-related. Which means it is worth investigating. For some length of time anyway. 

After all, you don't want a "news sentiment" model, that can't use news sentiment!

The Iran situation has caused huge volatility in a negative way, and this affected my trading badly as we picked stocks that were headed south. Our balance right now, is 89K from the initial 100K, so over 2 months, we have burned 11K of capital. Good thing it is paper trading.

I added two LLM aspects to the model:

1. agent_review.py which is a standalone agent analysis tool. It reads the signal file from generate_news_signal.py and searches for recent (keyword: recent) news on each candidate, and asks an LLM to recommend which stocks to buy with reasoning.

2. The portfolio_manager.py also looks at the current vix value - for the day (not the vix_0d value that was tied to the article headline date which is x days in the past), and also consults an LLM to check and see if the climate looks right to buy the selected stocks.

Both of these are designed to avoid mistakes buying. And, using an LLM is easier when you give it small tasks. Less tokens consumed, the task is more focused. Trying to give LLMs huge chunks of data to process can cause timeouts (504 errors), and strange results.

I do see a of trades being vetoed in our falling markets right now, so this is working and perhaps was added a bit late in the game. But it can maybe protect our capital so that when the regime shifts, we can start to get back up to the original 100K and then turn some profit. 

 

 

 

Backtesting - Decile Testing and Monotonocity - Part II

So now that we understand decile testing and monotonicity, we can run this on ALL features to see how they look.... 

And THIS is why the back-test was using "just" lag_ret_20d instead of the model predictions.

So the Macro features - just adding unnecessary noise, and no value!?

Is the News Sentiment adding any value at all? Maybe, they're ranked #4 and #8, but we would need to try to combine them to see if they add any value at all. Keep in mind also, that the residual (finbert_signed_resid) ferrets out "true" sentiment from momentum (as discussed in an earlier blog post).

And - if you combine them, in what ratio for them to make an optimal combination? Should we combine just two? Or more?

You can see where this can go. You almost need a permutational approach.