Removing Two Stale Macro Features

 

Removing Two Stale Macro Features

The model was trained on 11 features, two of which were macroeconomic sentiment indicators sourced from FRED. On inspection, both turned out to be monthly series — meaning they only update once a month and carry a publication lag on top of that. Despite this, the model had assigned them significant feature importance, essentially learning to lean on data that wasn't meaningfully changing day to day and wasn't even fully available in real time when historical training data was constructed.

Removing them dropped the feature set from 11 to 9. With those features gone, the model redistributed weight toward momentum and the remaining daily macro indicators in a more sensible way. Validation rank correlations improved on two of the three prediction horizons after the change. The two daily macro features that remained — VIX and treasury spread — are genuinely responsive to market conditions and carry the macro signal adequately on their own.

Both changes were low risk given that model predictions are used for monitoring purposes in this system rather than directly driving trading decisions.

Adding a Layer of LLM as a Final Trading Gate

When we did our back-testing logic we really started to get a full understanding of our features and what mattered and influenced returns.

 The *ONLY* feature that made ANY difference, was the finbert residual, which attempts to ferret out "true sentiment" from momentum. 

The uncomfortable but honest conclusion: the XGB model as currently built is mostly a complicated way of rediscovering momentum, with a thin layer of sentiment on top.

Nonetheless, there is a true signal there, and the signal IS news-related. Which means it is worth investigating. For some length of time anyway. 

After all, you don't want a "news sentiment" model, that can't use news sentiment!

The Iran situation has caused huge volatility in a negative way, and this affected my trading badly as we picked stocks that were headed south. Our balance right now, is 89K from the initial 100K, so over 2 months, we have burned 11K of capital. Good thing it is paper trading.

I added two LLM aspects to the model:

1. agent_review.py which is a standalone agent analysis tool. It reads the signal file from generate_news_signal.py and searches for recent (keyword: recent) news on each candidate, and asks an LLM to recommend which stocks to buy with reasoning.

2. The portfolio_manager.py also looks at the current vix value - for the day (not the vix_0d value that was tied to the article headline date which is x days in the past), and also consults an LLM to check and see if the climate looks right to buy the selected stocks.

Both of these are designed to avoid mistakes buying. And, using an LLM is easier when you give it small tasks. Less tokens consumed, the task is more focused. Trying to give LLMs huge chunks of data to process can cause timeouts (504 errors), and strange results.

I do see a of trades being vetoed in our falling markets right now, so this is working and perhaps was added a bit late in the game. But it can maybe protect our capital so that when the regime shifts, we can start to get back up to the original 100K and then turn some profit. 

 

 

 

Backtesting - Decile Testing and Monotonocity - Part II

So now that we understand decile testing and monotonicity, we can run this on ALL features to see how they look.... 

And THIS is why the back-test was using "just" lag_ret_20d instead of the model predictions.

So the Macro features - just adding unnecessary noise, and no value!?

Is the News Sentiment adding any value at all? Maybe, they're ranked #4 and #8, but we would need to try to combine them to see if they add any value at all. Keep in mind also, that the residual (finbert_signed_resid) ferrets out "true" sentiment from momentum (as discussed in an earlier blog post).

And - if you combine them, in what ratio for them to make an optimal combination? Should we combine just two? Or more?

You can see where this can go. You almost need a permutational approach.  

Backtesting - Decile Testing and Monotonicity

 

The Main Backtest (what runs by default)

Uses ONLY lag_ret_20d and momentum_strength.

That's it. The cohort analysis, decile analysis, 2D grid, and portfolio simulations all just look at raw price features directly from the database. No sentiment, no macro, no model. It's purely:

• Filter: is momentum_strength < 0 and lag_ret_20d > 5%?
• Rank: sort by lag_ret_20d descending
• Pick top N

The sentiment and macro features aren't even loaded in the default run. Look at the SQL query in load_from_db — it only pulls lag_ret_5d, lag_ret_20d, volatility_5d, and the return columns.

How lag_ret_20d Became the Ranking Signal

It wasn't chosen upfront. It emerged from the decile analysis in the backtest output. When all stocks are sorted by lag_ret_20d from lowest to highest and look at actual forward returns across deciles, you get this clean pattern:

D1  (avg lag20d: -26%) → ret_5d: -0.544%
D2  (avg lag20d: -13%) → ret_5d: -1.201%
...
D9  (avg lag20d: +15%) → ret_5d: +0.915%
D10 (avg lag20d: +29%) → ret_5d: +1.358%

 

 

7/9 monotonic — returns increase almost perfectly as lag_ret_20d increases. That's the definition of a good ranking signal. The data told you: stocks that have been strong over 20 days continue to be strong over the next 5 days.

When you ran the same decile test on momentum_strength you only got 4/9 monotonic — weak. And when you ran it on avg_pred (the model) you got 5/9 — moderate.

So the answer to "why lag_ret_20d" is simply: it produced the most monotonic decile separation on the holdout test set, which means it's the most reliable ranker of the three things tested.

---

What Hasn't Been Tested Yet

Here's the important caveat: you've only tested three ranking signals so far — momentum_strength, lag_ret_20d, and avg_pred. There could be better signals you haven't looked at yet, like volume, or lag_ret_20d combined with something else. But of the options tested, lag_ret_20d won clearly.

ok so the decile test is being run on all features, and lag_ret_20d is the one that had the best score? i don't know what monotonic means

5:16 PM

Not quite — the decile test was only run on a few features, not all of them. And let me explain monotonic in plain terms first because it's the key concept.

---

Monotonic in Plain English

Imagine you rank all students in a class by how much they studied, from least to most. Then you look at their test scores. If the more-they-studied, the-better-they-scored — perfectly and consistently — that's monotonic. Every step up in studying corresponds to a step up in grade.

In the backtest, you're asking: if I sort stocks by signal X from lowest to highest, do returns also go from lowest to highest? If yes, the signal is monotonic and you can trust it to rank stocks reliably.

Here's what lag_ret_20d looked like:

D1  lowest lag_ret_20d  → ret_5d: -1.201%   ← worst returns
D2                      → ret_5d: -0.544%
D3                      → ret_5d: -0.293%
...
D9                      → ret_5d: +0.915%
D10 highest lag_ret_20d → ret_5d: +1.358%   ← best returns

Almost perfectly stair-stepping up. That's what 7/9 monotonic means — 7 out of 9 consecutive steps went in the right direction.

Compare to momentum_strength which only got 4/9 — it was all over the place, stepping up then down then up. You can't rely on it to rank stocks.

---

Which Features Were Actually Decile-Tested

Only three were tested in the backtest:

• momentum_strength → 4/9 ❌ weak
• lag_ret_20d → 7/9 ✅ strong
• avg_pred (model output) → 5/9 ⚠️ moderate

That's it. The other features — sentiment, macro, volatility, trend_consistent — were never individually decile-tested. They only appear bundled together inside the model's avg_pred.

---

Should You Be Testing the Others?

Yes, if you want to know whether you're leaving alpha on the table.

The right way to do this is to run the decile test on every feature individually and see which ones have the cleanest monotonic separation. You'd be looking for features that score 6/9 or better. Then you test combinations of the top performers.

The candidates sitting untested right now:

Probably worth testing:

• lag_ret_5d — you use it to construct momentum_strength but never tested it alone
• volatility_5d — low volatility stocks might have cleaner signals
• finbert_signed — raw sentiment score before residualization
• finbert_signed_resid — sentiment surprise (sentiment unexplained by momentum)

Probably less useful alone but worth checking:

• tone_signed and tone_signed_resid
• vix_0d — same value for all stocks on a given day so won't help rank stocks against each other, but might tell you which days to trade
• risk_adjusted_momentum — momentum normalized by volatility, might be cleaner than raw momentum_strength

---

What Was Added

A new --feature-tournament flag that runs a systematic decile test on every feature and prints a ranked leaderboard. When you run it:

bash

python backtest_signal_quality.py --feature-tournament

It tests all of these in one shot:

Price/momentum: lag_ret_20d, lag_ret_5d, momentum_strength, risk_adjusted_momentum, trend_consistent, volatility_5d

Sentiment (raw): finbert_signed, tone_signed

Sentiment (residual): finbert_signed_resid, tone_signed_resid

Macro: vix_0d, treasury_spread_0d, business_confidence_0d, consumer_sentiment_0d

For each feature it reports monotonicity score, Spearman correlation, and D10-D1 spread (best decile return minus worst decile return). Then it prints a clean leaderboard ranked by monotonicity, and runs head-to-head portfolio simulations for the top 3 features.

You can also combine it with --model-lift:

bash

python backtest_signal_quality.py --feature-tournament --model-lift

The output will tell you definitively whether lag_ret_20d is actually the best signal available, or whether something else deserves to be the ranking signal in production.

---

The Leaderboard Summary

Rank	Feature	Mono	Spearman	Portfolio Sharpe
#1	volatility_5d	6/9	-0.014	2.70
#2	lag_ret_20d	5/9	+0.120	7.05
#3	lag_ret_5d	5/9	+0.055	5.38
#4	finbert_signed_resid	5/9	+0.006 (not significant)	—
Model (avg_pred)	—	5/9	+0.124	2.88

The Volatility Problem

The tournament crowned volatility_5d as the best feature by monotonicity (6/9). But look at what actually happens when you trade it — Sharpe 2.70, worst day -6.77%, a +13.96% outlier day that saved the whole period. That's a lottery ticket strategy, not a signal. It's picking the most volatile stocks and occasionally getting lucky with a huge mover.

This exposes a flaw in using monotonicity as the sole ranking criterion. Monotonicity measures consistency of direction, not quality of risk-adjusted returns. Volatility ranked stocks happen to step up consistently across deciles, but the actual portfolio is chaotic and dangerous.

lag_ret_20d has lower monotonicity (5/9) but Sharpe 7.05 vs 2.70 — more than 2.5x better risk-adjusted performance. It's the right choice for a strategy you'd actually trade.

---

What the Tournament Confirms

Sentiment is weak. Raw finbert_signed and tone_signed score 2/9 — essentially noise as standalone signals. The residual versions do better (5/9 for finbert_signed_resid) but the Spearman correlation is tiny (+0.006) and statistically insignificant. Sentiment is not currently a useful standalone ranking signal.

Macro features are useless for cross-sectional ranking. VIX and treasury spread score 4/9 and business confidence/consumer sentiment score 0/9. This makes sense — they're the same number for every stock on a given day, so they can't tell you which stocks to pick. They might be useful as day-level filters ("don't trade on high VIX days") but that's a different analysis.

lag_ret_5d is a viable alternative to lag_ret_20d. Same monotonicity (5/9), similar cumulative return (+17.01% vs +17.01% — identical in this test period), but lower Sharpe (5.38 vs 7.05) and a much scarier worst day (-5.87% vs -2.40%). The 20-day window is smoother and more reliable.

The model (avg_pred) at 5/9 monotonicity is now in proper context. It ties with lag_ret_20d and lag_ret_5d on monotonicity, but produces Sharpe 2.88 vs 7.05. The model is combining features in a way that degrades the clean signal from lag_ret_20d rather than enhancing it.

---

Bottom Line

lag_ret_20d remains the right ranking signal — not because it won a clean tournament, but because it has the best combination of monotonicity, Spearman correlation (+0.12 and statistically significant), and actual portfolio performance. The tournament confirms there is no obvious better single feature hiding in the data that you were previously ignoring.

The one thing worth investigating further: could lag_ret_20d + finbert_signed_resid combined beat lag_ret_20d alone? Both score 5/9 and their Spearman correlations suggest they might be capturing different things. That would be a combined ranking signal test — a logical next step.

AI / ML - Feature Engineering - Earnings Surprises

I really felt that news would be a great thing to add to the model. But the problem with news, is that news is recent, and this data I am using with XGBoost is historical time-series data. 

If you added news, what would you do - cram the values only into the most recent year?

I think if you go with stuff that is changing and close to real-time, you need to re-think the whole model including the type of model. Maybe news works better with a Transformer model or a LSTM Neural Network model than a predictive regression model.

So - I am running out of new things to add to my model to try and boost the predictability of it (increase the R-squared).

Then I came up with the idea of adding earnings hits, misses and meets. A quick consult with an LLM suggested using an earnings_surprise score, so that not only do we get the misses/meets/beats counts but also we capture the magnitude.  A great idea.

I implemented this, and lo and behold, the earnings_surprise score moves the needle. Substantially and consistently.

The best thing about this, is that the earnings_surprise score is symbol-specific, and so it is not some macro feature I have to figure out how to interact with the symbol data. 

AI / ML - Altman-Z Score

I saw a website that was showing an Altman-Z score for companies in their Solvency section.

Not fully aware of this calculation, I decided to jump in and add it to my model.

I quickly backed it out.

Why?

Altman-Z uses different calculations based on the industry a company is in. 

Manufacturers use one calculation, other companies use another, and banks and finance companies don't calculate it at all. 

So imagine calculating this and feeding it into XGBoost / SHAP to predict price or return on a security. 

First of all, because you have so many NaN values (nonexistents), you have a Missingness issue. Then, the values differ due to different calculation methods. If you don't cap the score, you can get outliers that wreak havoc.

So in the end, it's fine to calculate it, but if you calculate it, don't model it as a predictive feature. 

Just calculate it and "tack it on" (staple it) to any sector-specific scores you are generating for purposes of stuff like rank within sector. 

AI / ML Feature Explosion and Normalization

I got into some trouble with my model where it blew up to 150-180 features. When I finally took the time to really scrutinize things, I noticed that I had duplicates of raw metrics alongside their sector-z-scored components.  I don't know how that happened, or how those sector-z-scored components got into the feature set submitted to XGBoost and SHAP.  Probably logic inserted into the wrong place.

I wound up removing all of the sector-z scored metrics for now. 

But this highlighted a problem. Semantics.

I had some metrics that needed to be normalized out for scoring and comparison purposes. Mostly raw metrics, and to do this, we divided the value by TotalAssets.  For metrics that we did NOT want to do this to, we had some exclusion logic based on Regular Expressions (regex). We looked for metrics that had "Per" and "To" (among others).

This seems to have fixed our set of features, and it is so much better to see 30 features of 80 selected instead of 100 features of 180. It reduced a ton of noise on the model, improving its integrity.

Now I do need to go back and examine why we did the sector z-scores initially, to see if that is something we do need to engineer back in. I think we need to do that in the cases where we are producing a Top-X-By-Sector report. 

AI/ML - Feature Engineering - Normalization

On my quarterly financials model, the R² is awful. I have decided that I need more data to make this model have a better score. 3-4 quarters is probably not enough. That means I need to build something to go to the SEC and parse it. 

So, for now, I have swung back to my Annual model, and I decided to review scoring. 

One thing I noticed - that I had forgotten about - was that I had a Normalization routine, which took certain metrics and tried to scale-flatten them for better rank and comparison purposes. This routine, it takes certain metrics, and divides them by Total Assets. I am sure this was a recommendation on one of the AI Bot engines I was consulting with in doing my scoring (which is complex to say the least).  

Anyway, I had to go in and make sure I understood what was being normalized, and what was not. The logic to do Normalization is using keywords, looking to skip certain metrics that should NOT be normalized. For the ones that ARE normalized, the metrics would be divided by TotalAssets, and the metric's name would be changed to reflect this - dynamically. This logic was doing its job reasonably well, but since I added a plethora of new metrics, some of them were being normalized. 

So this is the new logic. 
    # --- Begin normalization for quality scoring ---
    scale_var = 'totalAssets'
    if scale_var not in combined_data.columns:
        raise ValueError(f"{scale_var} not found in data columns!")

    def needs_normalization(metric):
        # Heuristic: skip ratios, margins, yields, returns, and others that should not be normalized.
        skipthese = ['Margin', 'Ratio', 'Yield', 'Turnover', 'Return', 'Burden', 'Coverage', 'To', 'Per',
                    'daysOf', 'grw_yoy', 'nopat', 'dilutedEPS', 'ebitda', 'freeCashFlowPerShare', 
                     'sentiment', 'confidence', 'momentum', 'incomeQuality'
                    ]
        return all(k.lower() not in metric.lower() for k in skipthese)

And this took me some time to get working properly. Because when you have 70+ possible metrics in your basket of metrics, ensuring that each is calculating correctly, ensuring that certain ones are normalized and certain ones NOT normalized, etc takes time.

 

AI/ML Feature Engineering - Adding Feature-Based Features

I added some new features (metrics) to my model. The Quarterly model.

To recap, I have downloaded quarterly statements for stock symbols, and I use these to calculate an absolute slew of metrics and ratios. Then I feed them into the XGBoost regression model, to figure out whether they can predict a forward return of stock price.

I added some macro economic indicators, because I felt that those might impact the quarterly price of a stock (short term) more than the pure fundamentals of the stock. 

The fundamentals are used in an annual model - a separate model - and in that model, the model is not distracted or interrupted with "events" or macroeconomics that get in the way of understanding the true health of a company based on fundamentals over a years-long period of time.

So - what did I add to the quarterly model?

• Consumer Sentiment
• Business Confidence
• Inflation Expectations
• Treasury Data (1,3,10 year)
• Unemployment 

And wow - did these variables kick in. At one point, I had the model up to .16. 

Unemployment did nothing, actually. And I wound up removing it as a noise factor. I also realized I had the fiscal quarter included, and removed that too since it, like sector and other descriptive variables, should not be in the model.

But - as I was about to put a wrap on it, I decided to do one more "push" to improve the R-squared value, and started fiddling around. I got cute, adding derived features. One of the things I did, was to add lag features for business confidence, consumer sentiment, inflation expectations. Interestingly, two of these shot to the top of influential metrics.

Feature Importance List Sorted by Importance (return price influence).
feature                                                 weight
business_confidence_lag1                0.059845
inflation_lag1                                       0.054764

But, others were a bust, with .00000 values.

I tried removing the original metrics and JUST keeping the lags - didn't really help.

Another thing worth noting, is that I added SHAP values - a topic I will get into more depth about shortly, perhaps in a subsequent post. SHAP (SHapley Additive exPlanations) is a method used to explain the output of machine learning models by assigning each feature an importance value for a specific prediction, so that models - like so many - are not completely "black box".

But one thing I noticed when I added the SHAP feature list, is that it does NOT match / line up with the feature importances that the XGBoost model espouses. 

So I definitely need to look into this.