Adding a Layer of LLM as a Final Trading Gate

When we did our back-testing logic we really started to get a full understanding of our features and what mattered and influenced returns.

 The *ONLY* feature that made ANY difference, was the finbert residual, which attempts to ferret out "true sentiment" from momentum. 

The uncomfortable but honest conclusion: the XGB model as currently built is mostly a complicated way of rediscovering momentum, with a thin layer of sentiment on top.

Nonetheless, there is a true signal there, and the signal IS news-related. Which means it is worth investigating. For some length of time anyway. 

After all, you don't want a "news sentiment" model, that can't use news sentiment!

The Iran situation has caused huge volatility in a negative way, and this affected my trading badly as we picked stocks that were headed south. Our balance right now, is 89K from the initial 100K, so over 2 months, we have burned 11K of capital. Good thing it is paper trading.

I added two LLM aspects to the model:

1. agent_review.py which is a standalone agent analysis tool. It reads the signal file from generate_news_signal.py and searches for recent (keyword: recent) news on each candidate, and asks an LLM to recommend which stocks to buy with reasoning.

2. The portfolio_manager.py also looks at the current vix value - for the day (not the vix_0d value that was tied to the article headline date which is x days in the past), and also consults an LLM to check and see if the climate looks right to buy the selected stocks.

Both of these are designed to avoid mistakes buying. And, using an LLM is easier when you give it small tasks. Less tokens consumed, the task is more focused. Trying to give LLMs huge chunks of data to process can cause timeouts (504 errors), and strange results.

I do see a of trades being vetoed in our falling markets right now, so this is working and perhaps was added a bit late in the game. But it can maybe protect our capital so that when the regime shifts, we can start to get back up to the original 100K and then turn some profit. 

 

 

 

Trying to Right the Ship on my News Sentiment Based Stock Model

Jan 12th, through Feb 3rd. A downward trend that saw my balance drop from 101K to 89K.

The reasons were very complex. I will discuss them

Bugs in the trading module

Bugs in the code that put stop limits in, which caused repetitive falling knife scenario        buying.

OTC Stock Volatility

You can make a lot of money with OTC stocks, but you can also lose a lot. I removed the OTC exchanges and left just AMEX, NASDAQ and NYSE as the exchanges.

AI making changes to the algorithms I was not reviewing properly

AI had made several mistakes in moving the model from return prediction to a rank-based approach. 

In general, the consensus on discussion was that return prediction didn't make a ton of sense, and that ranking stocks based on their cumulative scores (news sentiment, momentum and other derived features) made the most sense. The problem though, was that the filtering was being done before the predictions, not AFTER. This meant that the universe of data was restricted to the model.

Article Mapping 

Phrase mapping had several bugs in it that caused completely legit articles to be unmapped. Recall that earlier, certain mapping bugs caused certain symbols to be attributed to casual words (i.e. key). 

I was mapping articles to symbols, and this was causing article-symbol-price tuples such that symbols with more news (articles) created a lot more rows of data than those that had less articles. And this was imbalancing the model.   The fix for this, was to aggregate the symbol_day prediction, which collapses multiple articles per symbol per day into one row by averaging sentiment features.

There were other changes I made as well, such as using XGBoost instead of Random Forest. The balance has bounced back up to $97,760 at the time of this writing, so a decent recovery. Of course the market has stabilized a bit - favoring Value right now. There indeed was a pullback market regime that did occur at the beginning of my slide, so that is also a contributing factor as well.

So - will these changes work? 

We shall see.
 

 

New AI / ML Stock Picking Model

Okay. The new stock picking model is VASTLY different than the previous one.

We download news, then we feed the news into transformer models (Finbert) to gauge sentiment and calculate sentiment scores. Originally, I was using vader sentiment also, but I removed that because vader just doesn't seem to work well with financial news.

Downloading the news was a sizable effort. I decided to use a multi-threaded approach, where each news source had its own thread(s).

After all of the news has been downloaded - and scored - I send the data into module that attempts to map the news to stock symbols. Doing this well, required numerous iterative enhancements to the code. Then, the news is filtered according to various rule sets.

Originally, I saved the news in a csv file. But later, I had to convert this to a database approach. There is also some caching, to make sure we are not fetching the same article repeatedly. The code also does some heavy work to ascertain the proper date of the article. 

Once this is finished, another module sets about finding - or trying to find - prices. It will price the 0d for the symbol for the article. A cache is used so that we don't try to look up the same price for the same day for the same symbol more than once (reduced API calls). The code may pick up prices for the "zero day plus X days" depending on how long the article has been sitting. 

Once the article has been fully "aged out", it is purged and migrated into a training database. This allows us to train the model on "actual returns".

After the pricing, there are some analyzers that will examine the integrity of data. And if all looks well, the model is (re-trained) with the newly migrated data, and new predictions are made. This allows for a continual improvement.

Once predictions are made, a portfolio managed makes trades using a paper account, and some back test programs are used to compare performance against the S&P, and if desired, prior model versions. 

I am sure I have skipped over a lot of the complexity on this, but in a nutshell this is what we are doing. I will avoid discussing the "secret sauce" which is the feature engineering.

The Financial Statement Model - Retired for Now

Once I got my Stock Prediction based on Annual (10-K) and Quarterly (10-Q) statement model working, I just wasn't happy with the R-squared on it. And I didn't feel comfortable investing in the picks it made (based on predicted returns). 

The R-squared on quarterly was so low, that trying to consider stocks it predicted for a quarter-long buy hold was just not feasible.

The R-squared on annual was considerably higher. But even then, it was not high enough to justify a stock purchase for a year-long tie-up of investment money.

Frankly, the stocks it was picking looked horrendous in many respects. Falling Knives, despite efforts to contain those, dominated the list. Others had low liquidity (read my earlier post on the Liquidity Effect) - and Solvency was an issue on them. Buying stocks with low or no liquidity and practically insolvent, and trying to hold them even a quarter, no less a year, is absolutely stupid.

I did Ensemble these models. But it didn't change the picture for me. And remember, I have Macro data and Macro Interactives in this model!

The conclusion: 
Statements (fundamentals) are important - but not for picking stocks based on them necessarily. You would have to combine the fundamentals with other things. 

I kind of knew this already, based on things I had read. I guess I needed to use the effort as a proving ground to myself.

So - in the end - I shelved these models. I learned a TON and it was great doing them. It built me into an AI Powerhouse with solid fundamentals in Quant Finance, an thorough understanding of Data Science and ML/AI algorithms, statistics, beefed-up math skills, etc.

I will move on. 

AI / ML - Random Forest, Data Wrestling and Z-Scores

I was running my AI scoring algorithm, which takes as inputs a bunch of calculated metrics and ratios (features - X axis), and feeds those into a Decision Tree algorithm (Random Forest), against a price prediction (Y axis), which then prints out a report that shows how well the algorithm performed in general (R-squared), and a list of features sorted by their influence on the Y variable (price). 

There are numerous algorithms that can do this - the simplest being a Linear Regression model.  Decision Trees offer a faster and more efficient - and perhaps more accurate - alternative to linear regression, provided that the tree is pruned and managed correctly and that the tree doesn't get lopsided or imbalanced.

But I ran into problems, especially when checking the results and data carefully. And most of the issues, were related to the data itself.

Data Alignment
I noticed that the predictive z-scores for my features didn't "line up" when I printed them twice. Turns out, this was a data alignment issue. When you are using dataframes, and making copies of these dataframes and merging them, you need to be very very careful or a column can get shifted.

This alignment issue was affecting my model because the metric that WAS a profitability metric, was now being assigned to a solvency metric. Now that I have this fixed, things look much more sensible. But making sure your dataframes are aligned, is a hard-learned lesson.

Outliers
Other issues I ran into today had to do with the fact that when I printed a report out (a weighted scoring report), certain values were far and away better than others. I didn't understand this, and discussed it with the AI I am using as a consultant, who suggested I print out z-scores.

Well, if we look below, we have an evToEBITDA metric of 10.392 (insane value) on 2023 Paramount reporting data.

=== Z-scores for PARA on 2023-12-31 ===
Z-scores for PARA on 2023-12-31 in pillar 'Profitability':
  grossProfitMargin: -0.263
  operatingProfitMargin: 0.038
  netProfitMargin: 0.029
  returnOnAssets: -0.033
  returnOnEquity: -0.006
  returnOnCapitalEmployed: -0.089
  returnOnTangibleAssets: 0.004
  earningsYield: 0.008
  freeCashFlowYield: 0.000
  nopat_to_totalAssets: -0.170
  operatingReturnOnAssets: -0.215
  returnOnInvestedCapital: -0.031
  ebitda_to_totalAssets: -0.384
  operatingCashFlowToSales: 0.036
  evToSales: -0.044
  evToOperatingCashFlow: 0.054
  evToEBITDA: 10.392
  evToFreeCashFlow: 0.039
 
I audited the metrics and statements, and indeed this is correct - based on what Yahoo was returning to me on the income statement for that year (Normalized EBITDA). The unnormalized EBITDA was better, but in most cases, analysts use the Normalized value. You can't do one-offs in your code for things like this, so what do you do?

I couldn't drop the row, because I was already dropping so many 2020 rows of bad data (due to Covid I suspect). I drop rows that are missing >35% of metrics. When you get a row that has all of the values you need, you tend to want to use it. I don't have code that drops rows that don't have specific dealbreaker metrics - maybe I should, but there are so many metrics that generally I figure I can score and rank even if I am missing one here or there, even a fairly well-known or important one. 

So - what do you do?

Winsorization. In other words, capping. It might make sense to invest the effort in Winsorizing all of the metrics and ratios. But for now, I am only doing it on these EBITDA ones.