Adding a Layer of LLM as a Final Trading Gate

When we did our back-testing logic we really started to get a full understanding of our features and what mattered and influenced returns.

 The *ONLY* feature that made ANY difference, was the finbert residual, which attempts to ferret out "true sentiment" from momentum. 

The uncomfortable but honest conclusion: the XGB model as currently built is mostly a complicated way of rediscovering momentum, with a thin layer of sentiment on top.

Nonetheless, there is a true signal there, and the signal IS news-related. Which means it is worth investigating. For some length of time anyway. 

After all, you don't want a "news sentiment" model, that can't use news sentiment!

The Iran situation has caused huge volatility in a negative way, and this affected my trading badly as we picked stocks that were headed south. Our balance right now, is 89K from the initial 100K, so over 2 months, we have burned 11K of capital. Good thing it is paper trading.

I added two LLM aspects to the model:

1. agent_review.py which is a standalone agent analysis tool. It reads the signal file from generate_news_signal.py and searches for recent (keyword: recent) news on each candidate, and asks an LLM to recommend which stocks to buy with reasoning.

2. The portfolio_manager.py also looks at the current vix value - for the day (not the vix_0d value that was tied to the article headline date which is x days in the past), and also consults an LLM to check and see if the climate looks right to buy the selected stocks.

Both of these are designed to avoid mistakes buying. And, using an LLM is easier when you give it small tasks. Less tokens consumed, the task is more focused. Trying to give LLMs huge chunks of data to process can cause timeouts (504 errors), and strange results.

I do see a of trades being vetoed in our falling markets right now, so this is working and perhaps was added a bit late in the game. But it can maybe protect our capital so that when the regime shifts, we can start to get back up to the original 100K and then turn some profit. 

 

 

 

The News Sentiment Model is Not About News Sentiment

Quick post to discuss some things...

First, what I am discovering after training this model daily, and running predictions, is that the news sentiment is NOT what is influencing this model. 

The news sentiment is based on Transformer models for financial news (Finbert). The news IS fresh, as we get these articles and and then predict over a 1-3 day period, before they're expired out to a training database where predicted returns are compared with actual returns.

News is NOT what this model is learning and training on. Instead, momentum and macro environment features are the aspects that is influencing the returns. Part of this could be the news being stale - the market already knows the news before the model is running. Also, we are only scoring headlines - not actual articles - due to processing constraints. So there's that.

From the training output, here's the feature importance ranked across all three horizons:

Macro features (dominating ~50% of importance):

• vix_0d — market fear index
• treasury_spread_0d — yield curve spread
• business_confidence_0d — FRED macro indicator
• consumer_sentiment_0d — FRED macro indicator

Momentum features (~40%):

• momentum_strength — short-term trend deceleration
• risk_adjusted_momentum — momentum relative to volatility
• trend_consistent — direction consistency

Sentiment features (~10%):

• tone_signed_resid — residualized tone score
• finbert_signed_resid — residualized FinBERT score
• tone_signed — raw tone
• finbert_signed — raw FinBERT

So the model is essentially saying: buy beaten-down stocks when the macro environment is calm. The news sentiment is contributing about 10% of the predictive signal, which is barely above noise given the R² of 0.002 on the residual models.

It's not really a news sentiment model at this point. It's a macro-regime mean-reversion model that happens to have sentiment features along for the ride.

Which raises the real question — is that actually a bad thing? Mean reversion in calm macro regimes is a legitimate strategy. The problem isn't the model's logic, it's that you only have 3 months of data so it hasn't seen enough regime diversity to be robust. And right now you're in a fear regime — elevated VIX, geopolitical uncertainty — which is exactly when this model historically underperforms.

Trying to Right the Ship on my News Sentiment Based Stock Model

Jan 12th, through Feb 3rd. A downward trend that saw my balance drop from 101K to 89K.

The reasons were very complex. I will discuss them

Bugs in the trading module

Bugs in the code that put stop limits in, which caused repetitive falling knife scenario        buying.

OTC Stock Volatility

You can make a lot of money with OTC stocks, but you can also lose a lot. I removed the OTC exchanges and left just AMEX, NASDAQ and NYSE as the exchanges.

AI making changes to the algorithms I was not reviewing properly

AI had made several mistakes in moving the model from return prediction to a rank-based approach. 

In general, the consensus on discussion was that return prediction didn't make a ton of sense, and that ranking stocks based on their cumulative scores (news sentiment, momentum and other derived features) made the most sense. The problem though, was that the filtering was being done before the predictions, not AFTER. This meant that the universe of data was restricted to the model.

Article Mapping 

Phrase mapping had several bugs in it that caused completely legit articles to be unmapped. Recall that earlier, certain mapping bugs caused certain symbols to be attributed to casual words (i.e. key). 

I was mapping articles to symbols, and this was causing article-symbol-price tuples such that symbols with more news (articles) created a lot more rows of data than those that had less articles. And this was imbalancing the model.   The fix for this, was to aggregate the symbol_day prediction, which collapses multiple articles per symbol per day into one row by averaging sentiment features.

There were other changes I made as well, such as using XGBoost instead of Random Forest. The balance has bounced back up to $97,760 at the time of this writing, so a decent recovery. Of course the market has stabilized a bit - favoring Value right now. There indeed was a pullback market regime that did occur at the beginning of my slide, so that is also a contributing factor as well.

So - will these changes work? 

We shall see.