Removing Two Stale Macro Features

 

Removing Two Stale Macro Features

The model was trained on 11 features, two of which were macroeconomic sentiment indicators sourced from FRED. On inspection, both turned out to be monthly series — meaning they only update once a month and carry a publication lag on top of that. Despite this, the model had assigned them significant feature importance, essentially learning to lean on data that wasn't meaningfully changing day to day and wasn't even fully available in real time when historical training data was constructed.

Removing them dropped the feature set from 11 to 9. With those features gone, the model redistributed weight toward momentum and the remaining daily macro indicators in a more sensible way. Validation rank correlations improved on two of the three prediction horizons after the change. The two daily macro features that remained — VIX and treasury spread — are genuinely responsive to market conditions and carry the macro signal adequately on their own.

Both changes were low risk given that model predictions are used for monitoring purposes in this system rather than directly driving trading decisions.

Fixing a Train/Serve Skew in Sentiment Residuals

 

Fixing a Train/Serve Skew in Sentiment Residuals

The signal generation process uses a technique called sentiment residualization — essentially, we remove the portion of the FinBERT sentiment score that can be explained by price momentum alone, leaving behind only the genuine sentiment surprise. A stock that has been running up for 20 days will naturally attract positive news coverage, so we want to isolate the sentiment signal that exists above and beyond what the price action would predict.

The problem was subtle. During training, the residual model was fitted on tens of thousands of rows spanning months of history. But at inference time, the same calculation was being refitted fresh each day on whatever small universe of stocks passed the daily filters — typically around 30 stocks. That's a very different statistical population, which meant the sentiment residuals being fed into the composite ranking signal weren't quite the same thing the model had learned from during training. Classic train/serve skew.

The fix was straightforward — serialize the residual model coefficients to disk at the end of each training run and load those fixed coefficients at inference time rather than refitting. Now the definition of sentiment surprise is consistent from training through to live signal generation.

Adding a Layer of LLM as a Final Trading Gate

When we did our back-testing logic we really started to get a full understanding of our features and what mattered and influenced returns.

 The *ONLY* feature that made ANY difference, was the finbert residual, which attempts to ferret out "true sentiment" from momentum. 

The uncomfortable but honest conclusion: the XGB model as currently built is mostly a complicated way of rediscovering momentum, with a thin layer of sentiment on top.

Nonetheless, there is a true signal there, and the signal IS news-related. Which means it is worth investigating. For some length of time anyway. 

After all, you don't want a "news sentiment" model, that can't use news sentiment!

The Iran situation has caused huge volatility in a negative way, and this affected my trading badly as we picked stocks that were headed south. Our balance right now, is 89K from the initial 100K, so over 2 months, we have burned 11K of capital. Good thing it is paper trading.

I added two LLM aspects to the model:

1. agent_review.py which is a standalone agent analysis tool. It reads the signal file from generate_news_signal.py and searches for recent (keyword: recent) news on each candidate, and asks an LLM to recommend which stocks to buy with reasoning.

2. The portfolio_manager.py also looks at the current vix value - for the day (not the vix_0d value that was tied to the article headline date which is x days in the past), and also consults an LLM to check and see if the climate looks right to buy the selected stocks.

Both of these are designed to avoid mistakes buying. And, using an LLM is easier when you give it small tasks. Less tokens consumed, the task is more focused. Trying to give LLMs huge chunks of data to process can cause timeouts (504 errors), and strange results.

I do see a of trades being vetoed in our falling markets right now, so this is working and perhaps was added a bit late in the game. But it can maybe protect our capital so that when the regime shifts, we can start to get back up to the original 100K and then turn some profit. 

 

 

 

Fixing the Momentum Filter

 

The Problem

I noticed the model was not making ANY trades for over a week. Every day, the model generated 30+ buy signals, but the portfolio manager - which acts as a gatekeeper - blocked ALL of them due to negative momentum.

Root Cause Analysis

1. Stale momentum data:

• Using articles from the last 10 days (now 3 days)
• Momentum was calculated from prices 5-20 days old
• By the time you made trading decisions, that momentum was ancient history

2. The momentum paradox:

• momentum_strength = lag_ret_5d - lag_ret_20d
• This measures deceleration, not absolute direction
• Example: Stock up 20% over 20 days, then pulls back 4% in last 5 days = -24% momentum_strength
• The filter was blocking strong stocks taking healthy pullbacks

The Backtest Evidence

Ran analysis on 31,600 test samples and found:

OLD filter (momentum_strength >= 0.1):

• Blocked 89% of stocks
• Blocked stocks: +0.64% avg return, 59.1% win rate ✅ BETTER
• Allowed stocks: +0.23% avg return, 55.7% win rate ❌ WORSE

Best performing cohort (which were blocking):

• "Strong deceleration" (<-0.15 momentum)
• Returns: +1.05% (1d), +2.58% (3d), +2.04% (5d)
• Win rates: 61.6%, 67.4%, 63.4%

The filter was blocking the best opportunities.

The Solution

OLD: Only buy if momentum_strength > 0

NEW: Buy if EITHER:

1. lag_ret_20d > 10% (strong 20-day uptrend), OR
2. 0 < momentum_strength < 15% (mild positive momentum)

Why this works (at least according to the backtest):

• Captures pullbacks in strong uptrends (mean reversion plays)
• Captures steady risers (not overextended)
• Blocks actual falling knives (negative long-term trend)
• Blocks momentum chasers at the top (>15% recent momentum)

Results

• 28/30 signals now pass the filter (vs 0/30 before)
• Deployed 9 trades today with the new logic
• All are strong stocks pulling back (exactly what backtest said to buy)

Now we wait to see if these actually perform as the backtest predicted.

Trying to Right the Ship on my News Sentiment Based Stock Model

Jan 12th, through Feb 3rd. A downward trend that saw my balance drop from 101K to 89K.

The reasons were very complex. I will discuss them

Bugs in the trading module

Bugs in the code that put stop limits in, which caused repetitive falling knife scenario        buying.

OTC Stock Volatility

You can make a lot of money with OTC stocks, but you can also lose a lot. I removed the OTC exchanges and left just AMEX, NASDAQ and NYSE as the exchanges.

AI making changes to the algorithms I was not reviewing properly

AI had made several mistakes in moving the model from return prediction to a rank-based approach. 

In general, the consensus on discussion was that return prediction didn't make a ton of sense, and that ranking stocks based on their cumulative scores (news sentiment, momentum and other derived features) made the most sense. The problem though, was that the filtering was being done before the predictions, not AFTER. This meant that the universe of data was restricted to the model.

Article Mapping 

Phrase mapping had several bugs in it that caused completely legit articles to be unmapped. Recall that earlier, certain mapping bugs caused certain symbols to be attributed to casual words (i.e. key). 

I was mapping articles to symbols, and this was causing article-symbol-price tuples such that symbols with more news (articles) created a lot more rows of data than those that had less articles. And this was imbalancing the model.   The fix for this, was to aggregate the symbol_day prediction, which collapses multiple articles per symbol per day into one row by averaging sentiment features.

There were other changes I made as well, such as using XGBoost instead of Random Forest. The balance has bounced back up to $97,760 at the time of this writing, so a decent recovery. Of course the market has stabilized a bit - favoring Value right now. There indeed was a pullback market regime that did occur at the beginning of my slide, so that is also a contributing factor as well.

So - will these changes work? 

We shall see.
 

 

Jan 12th - The Death Spiral - Trading Lessons Learned

I learned quickly all of the issues with Algo Trading.

Lesson 1 - 4-to-1 Margins Were Enabled

I didn't take the time to look at the Configuration. So I was trading on Margin, and had to Liquidate a bunch of positions, some of which lost money.

Lesson 2 - Stop Limits Can Lead to a HOST of Problems

I had stop limit orders being set up to protect my downside. But, this led to THIS problem:

Buy HMY @ $25 → gap to $23 → stop → rebuy $23.50 
→ gap to $21 → stop → rebuy $21.80 → PDT flag → can't exit → $18

Basically, the stop limits were causing repetitive buying in falling-knife scenarios, because the model would pick the same symbol the next trade.

Lesson 3 - Some Symbols Are Halted or Delisted

I had one position, CLCO, which was halted. I have tried to force liquidate this position, but it is absolutely stuck. I will probably have to contact support. But this has thrown a major wrench into my Portfolio Manager, which needs to know that this money is "locked up" and account for that.

I also had issues with the model itself - in that the way the algorithm was working, it would pick the best ranked stocks, even if the signal strength or momentum was negative. So this had to be fixed as well, with some enhanced signal strength and momentum logic.

So - very expensive lessons. The back-testing of the model is turning out fine. But the Porfolio Manager and its transaction management was killing us. We burned about $10K of our $100K on this. Hopefully it starts to work better now with fixes to these issues.

The Algo Trading Model is Live!

I am now using a Paper Trading account using Alpaca!

The paper trading account seeds you with $100K of money. This is a bit similar to the Investopedia Simulator, except that you can work with this account programmatically through APIs. It looks like a lot of day traders (options, derivatives, et al) use this. But I am using it just to buy and sell stocks.

Their user interface is a bit confusing, but with the help of some Python, I am able to get the status information I need. For example, the Positions tab doesn't carry much information. You have to actually consult the Orders tab, and marry things together to get a fuller view.

I needed a Portfolio Manager to make decisions on trading. I was going to use an Agentic AI approach using Langchain, or maybe Crew or MCP. Something along that vein.  But the AIs themselves, pointed out that these are too quirky and "black box" to be able to rely on them in a real trading situation. They suggested I go with a Deterministic approach.

So - with the help of an LLM that is trained on more technical topics, I had it generate me a few drafts until I saw something that looked simple enough to get started with.  It simply takes the predictions csv, and uses that as input for its trading. It will buy positions, hold them for 5 days, and sell them. Simple enough.

 

Updates on the Short-Term Stock Picking Model

I have made MANY MANY changes to the model, iteratively.

Model Strategy:

We started by picking stocks based on return predictions, using news sentiment scores. We quickly ditched vader, because it simply wasn't working well for the context of finance news. This left us with some transformer models (2) that seem to work better. 

But we decided to add some new features into the model: 

• Macro Indicators (i.e. Inflation Expectations, Ten Year Yield, et al). 
• Momentum Indicators (and also trend consistency)

To add the momentum, we had to collect more data, which was a big change. We had to get lag data (5 days and 20 days).  Because of this, I had to re-think the feature engineering logic, and compartmentalize it as much as possible so that we can add or remove features without "turning everything upside down and inside out".

After this, we learned that Momentum was - by far - the dominant predictive influencer. Indeed, some AIs I consulted told me that the news sentiment was just noise, and that I should ditch it and go just with Momentum alone.

The R-squared on these models is terrible - and is negative. But - you don't want to just invert that necessarily. This could have been because of the fact that we simply didn't have enough training data - and training data that crossed regimes (up market down market).

But guess what? When I put some back-testing modules together and looked at actual returns,  the model did better - much better - with news sentiment used in conjunction with momentum than it would have with just momentum used alone. But - the suggestion came, to use a rank approach as opposed to just using the returns themselves.

I consulted with some Quant Algo Traders on this, and they (I presume smarter than I), agreed.

So now, the model is using the rank approach.

New AI / ML Stock Picking Model

Okay. The new stock picking model is VASTLY different than the previous one.

We download news, then we feed the news into transformer models (Finbert) to gauge sentiment and calculate sentiment scores. Originally, I was using vader sentiment also, but I removed that because vader just doesn't seem to work well with financial news.

Downloading the news was a sizable effort. I decided to use a multi-threaded approach, where each news source had its own thread(s).

After all of the news has been downloaded - and scored - I send the data into module that attempts to map the news to stock symbols. Doing this well, required numerous iterative enhancements to the code. Then, the news is filtered according to various rule sets.

Originally, I saved the news in a csv file. But later, I had to convert this to a database approach. There is also some caching, to make sure we are not fetching the same article repeatedly. The code also does some heavy work to ascertain the proper date of the article. 

Once this is finished, another module sets about finding - or trying to find - prices. It will price the 0d for the symbol for the article. A cache is used so that we don't try to look up the same price for the same day for the same symbol more than once (reduced API calls). The code may pick up prices for the "zero day plus X days" depending on how long the article has been sitting. 

Once the article has been fully "aged out", it is purged and migrated into a training database. This allows us to train the model on "actual returns".

After the pricing, there are some analyzers that will examine the integrity of data. And if all looks well, the model is (re-trained) with the newly migrated data, and new predictions are made. This allows for a continual improvement.

Once predictions are made, a portfolio managed makes trades using a paper account, and some back test programs are used to compare performance against the S&P, and if desired, prior model versions. 

I am sure I have skipped over a lot of the complexity on this, but in a nutshell this is what we are doing. I will avoid discussing the "secret sauce" which is the feature engineering.

The Financial Statement Model - Retired for Now

Once I got my Stock Prediction based on Annual (10-K) and Quarterly (10-Q) statement model working, I just wasn't happy with the R-squared on it. And I didn't feel comfortable investing in the picks it made (based on predicted returns). 

The R-squared on quarterly was so low, that trying to consider stocks it predicted for a quarter-long buy hold was just not feasible.

The R-squared on annual was considerably higher. But even then, it was not high enough to justify a stock purchase for a year-long tie-up of investment money.

Frankly, the stocks it was picking looked horrendous in many respects. Falling Knives, despite efforts to contain those, dominated the list. Others had low liquidity (read my earlier post on the Liquidity Effect) - and Solvency was an issue on them. Buying stocks with low or no liquidity and practically insolvent, and trying to hold them even a quarter, no less a year, is absolutely stupid.

I did Ensemble these models. But it didn't change the picture for me. And remember, I have Macro data and Macro Interactives in this model!

The conclusion: 
Statements (fundamentals) are important - but not for picking stocks based on them necessarily. You would have to combine the fundamentals with other things. 

I kind of knew this already, based on things I had read. I guess I needed to use the effort as a proving ground to myself.

So - in the end - I shelved these models. I learned a TON and it was great doing them. It built me into an AI Powerhouse with solid fundamentals in Quant Finance, an thorough understanding of Data Science and ML/AI algorithms, statistics, beefed-up math skills, etc.

I will move on.