Removing Two Stale Macro Features

 

Removing Two Stale Macro Features

The model was trained on 11 features, two of which were macroeconomic sentiment indicators sourced from FRED. On inspection, both turned out to be monthly series — meaning they only update once a month and carry a publication lag on top of that. Despite this, the model had assigned them significant feature importance, essentially learning to lean on data that wasn't meaningfully changing day to day and wasn't even fully available in real time when historical training data was constructed.

Removing them dropped the feature set from 11 to 9. With those features gone, the model redistributed weight toward momentum and the remaining daily macro indicators in a more sensible way. Validation rank correlations improved on two of the three prediction horizons after the change. The two daily macro features that remained — VIX and treasury spread — are genuinely responsive to market conditions and carry the macro signal adequately on their own.

Both changes were low risk given that model predictions are used for monitoring purposes in this system rather than directly driving trading decisions.

Adding a Layer of LLM as a Final Trading Gate

When we did our back-testing logic we really started to get a full understanding of our features and what mattered and influenced returns.

 The *ONLY* feature that made ANY difference, was the finbert residual, which attempts to ferret out "true sentiment" from momentum. 

The uncomfortable but honest conclusion: the XGB model as currently built is mostly a complicated way of rediscovering momentum, with a thin layer of sentiment on top.

Nonetheless, there is a true signal there, and the signal IS news-related. Which means it is worth investigating. For some length of time anyway. 

After all, you don't want a "news sentiment" model, that can't use news sentiment!

The Iran situation has caused huge volatility in a negative way, and this affected my trading badly as we picked stocks that were headed south. Our balance right now, is 89K from the initial 100K, so over 2 months, we have burned 11K of capital. Good thing it is paper trading.

I added two LLM aspects to the model:

1. agent_review.py which is a standalone agent analysis tool. It reads the signal file from generate_news_signal.py and searches for recent (keyword: recent) news on each candidate, and asks an LLM to recommend which stocks to buy with reasoning.

2. The portfolio_manager.py also looks at the current vix value - for the day (not the vix_0d value that was tied to the article headline date which is x days in the past), and also consults an LLM to check and see if the climate looks right to buy the selected stocks.

Both of these are designed to avoid mistakes buying. And, using an LLM is easier when you give it small tasks. Less tokens consumed, the task is more focused. Trying to give LLMs huge chunks of data to process can cause timeouts (504 errors), and strange results.

I do see a of trades being vetoed in our falling markets right now, so this is working and perhaps was added a bit late in the game. But it can maybe protect our capital so that when the regime shifts, we can start to get back up to the original 100K and then turn some profit. 

 

 

 

My New Stock Prediction Model - Short Term Stock Prediction

Someone I work with has been working extensively on a Swing Trading model.

He has great financial experience from what I understand, but as he is in more of a management role and not in a day-to-day technical role, his programming skills might be just a tad or a step behind mine.

I have been watching him publish his short-term predictions, and his model is based on all kinds of things. I won't publish his secret sauce here, but he is using things that day traders tend to use, like RSI and Stochastics and such.

But, like my Financial Statement model - which tried to predict longer term buy-hold stocks,  his wasn't holding water either through back testing and results. One problem is that everything looks great in a bull market (rising tides lift all boats). And, to quote Buffet, "only when the tide goes out can you see who is swimming naked". So these models need to hold up in BOTH upturns and downturns.

I have decided to work on a new stock picking model. I am not sure yet how much I will blog about the specifics of it. But it is also a short-term model.

Stay Tuned 

The Financial Statement Model - Retired for Now

Once I got my Stock Prediction based on Annual (10-K) and Quarterly (10-Q) statement model working, I just wasn't happy with the R-squared on it. And I didn't feel comfortable investing in the picks it made (based on predicted returns). 

The R-squared on quarterly was so low, that trying to consider stocks it predicted for a quarter-long buy hold was just not feasible.

The R-squared on annual was considerably higher. But even then, it was not high enough to justify a stock purchase for a year-long tie-up of investment money.

Frankly, the stocks it was picking looked horrendous in many respects. Falling Knives, despite efforts to contain those, dominated the list. Others had low liquidity (read my earlier post on the Liquidity Effect) - and Solvency was an issue on them. Buying stocks with low or no liquidity and practically insolvent, and trying to hold them even a quarter, no less a year, is absolutely stupid.

I did Ensemble these models. But it didn't change the picture for me. And remember, I have Macro data and Macro Interactives in this model!

The conclusion: 
Statements (fundamentals) are important - but not for picking stocks based on them necessarily. You would have to combine the fundamentals with other things. 

I kind of knew this already, based on things I had read. I guess I needed to use the effort as a proving ground to myself.

So - in the end - I shelved these models. I learned a TON and it was great doing them. It built me into an AI Powerhouse with solid fundamentals in Quant Finance, an thorough understanding of Data Science and ML/AI algorithms, statistics, beefed-up math skills, etc.

I will move on. 

Target Encoding & One-Hot Encoding

I had completely overlooked these concepts earlier on, and somehow not until just now - this late in the game - did they come up as I was reading some of the advanced sections of the Jansen Algorithmic Trading book.

When you run a model, there are always going to be things you do NOT want to include. For many reasons (circular correlation, redundancy, etc). Generally, you would not want to include symbol, sector and industry in your model because the symbol after all, is nothing more than an index in the data - and the symbol in and of itself should have NOTHING to do with predicting returns.

So we keep an Exclusion array of metrics we don't want included:
EXCLUDED_FEATURES = [
    # 'symbol', 
    'date', 'year', 'quarter', 'close', 'fwdreturn',  # identity + target
    # 'sector', 'industry',  optional: include via one-hot or embeddings if desired
...
]

Symbol, Sector and Industry were initially excluded because they are text strings, and models don't like text data (some won't run without only numerics, some give warnings, others unpredictable results if they tolerate it).

Then I read about target encodings for symbols - which allow you to KEEP symbols in the data, and something called "one-hot" encodings for things like sectors and industries.

What these encodings do, is they take categorical data - like sectors and industries - and they convert them to a numeric so that they will be considered by the model. This captures the association between category and target, sometimes boosting model power.

So - adding the code to do this wasn't terribly painful. The symbol was encoded as a single encoded target feature (symbol_enc) - and removed from the above-mentioned EXCLUDED_FEATURES list.  The code to one-hot encode the Sector and Industry was snapped in, and off we went...

The R-squared dropped from .11 to .06. Yikes...
🪓  Pruning all splits - train, val, test - to the pruned set of features (final_features).
Training pruned model...
FULL R² -- Train: 0.4989, Validation: 0.0776, Test: 0.0673
PRUNED R² -- Train: 0.2216, Validation: 0.0348, Test: 0.0496
Selected FULL model based on test R².
Previous best test R²: 0.1126
Current model test R²: 0.0673
Current model did NOT exceed best. Loading previous best model and features.
Final Model Test Metrics -- R²: 0.1126, RMSE: 0.2133, MAE: 0.1471


I quickly realized, that by one-hot encoding the Industry, we got so many feature category permutations of the industries, that it undoubtedly screwed up the model.

I decided to run again, and only one-hot encode the Sector. So now with this run, we have a target encoding for symbol, and a one-hot encoding for sector. There are not that many sectors, so this doesn't become too unwieldy.

But - here is what we got on our SHAP analysis after this change:
🧮 Feature Selection Overview:
✅  consumer_sentiment_×_earnings_yield                SHAP=0.04511
✅  dilutedEPS                                         SHAP=0.02566
✅  rev_grw_4q                                         SHAP=0.01191
✅  evToSales                                          SHAP=0.01181
✅  roe                                                SHAP=0.01071
✅  symbol_enc                                         SHAP=0.01055
✅  business_confidence_×_capexToRevenue               SHAP=0.00925
✅  treasury_spread_x_debt_to_equity                   SHAP=0.00867
✅  cpippi_marginsqueeze                               SHAP=0.00719
✅  earningsYield                                      SHAP=0.00703
✅  inflation_x_debt_to_equity                         SHAP=0.00701
⛔  ppi_x_capextorevenue                               SHAP=0.00628
⛔  operatingCashFlow_to_totalAssets                   SHAP=0.00610
⛔  momentum_1y                                        SHAP=0.00586
⛔  gross_margin                                       SHAP=0.00575
⛔  realized_vol_3m                                    SHAP=0.00543
⛔  long_term_debt_to_equity                           SHAP=0.00526
⛔  rev_grw_qoq                                        SHAP=0.00500
⛔  vix_×_debt_to_equity                               SHAP=0.00480
⛔  goodwill_to_totalAssets                            SHAP=0.00432
⛔  cpi_x_netprofitmargin                              SHAP=0.00371
⛔  incomeQuality                                      SHAP=0.00360
⛔  rev_grw_qoq_to_totalAssets                         SHAP=0.00341
⛔  netDebtToEBITDA                                    SHAP=0.00340
⛔  evToFreeCashFlow                                   SHAP=0.00304
⛔  debt_to_equity                                     SHAP=0.00303
⛔  salesGeneralAndAdministrativeToRevenue             SHAP=0.00288
⛔  vix_x_evtoebitda                                   SHAP=0.00273
⛔  rev_grw_pop_sector_z                               SHAP=0.00268
⛔  eps_grw_qoq                                        SHAP=0.00258
⛔  evToEBITDA                                         SHAP=0.00258
⛔  interestBurden                                     SHAP=0.00255
⛔  momentum_1y_sector_z                               SHAP=0.00252
⛔  evToEBITDA_sector_z                                SHAP=0.00237
⛔  asset_turnover                                     SHAP=0.00235
⛔  totalEquity                                        SHAP=0.00231
⛔  net_margin                                         SHAP=0.00228
⛔  workingCapital                                     SHAP=0.00218
⛔  cc_delinquency_rate_×_debt_ratio                   SHAP=0.00209
⛔  inventory_turnover                                 SHAP=0.00209
⛔  capexToRevenue                                     SHAP=0.00197
⛔  freeCashFlowPerShare                               SHAP=0.00191
⛔  daysOfInventoryOutstanding_sector_z                SHAP=0.00186
⛔  fcf_ps_grw_qoq                                     SHAP=0.00186
⛔  eps_grw_qoq_to_totalAssets                         SHAP=0.00182
⛔  daysOfInventoryOutstanding                         SHAP=0.00180
⛔  daysOfPayablesOutstanding                          SHAP=0.00179
⛔  capexToRevenue_sector_z                            SHAP=0.00179
⛔  operatingCashFlow                                  SHAP=0.00173
⛔  totalEquity_to_totalAssets                         SHAP=0.00163
⛔  stockBasedCompensationToRevenue                    SHAP=0.00159
⛔  cash_ratio                                         SHAP=0.00158
⛔  evToOperatingCashFlow                              SHAP=0.00156
⛔  debt_ratio                                         SHAP=0.00156
⛔  eps_grw_4q                                         SHAP=0.00155
⛔  receivables_turnover                               SHAP=0.00153
⛔  ocf_to_current_liabilities                         SHAP=0.00151
⛔  totalDebt                                          SHAP=0.00146
⛔  operating_margin                                   SHAP=0.00143
⛔  debt_ratio_sector_z                                SHAP=0.00143
⛔  workingCapital_to_totalAssets                      SHAP=0.00135
⛔  operatingReturnOnAssets                            SHAP=0.00130
⛔  daysOfSalesOutstanding                             SHAP=0.00129
⛔  ordinary_shares                                    SHAP=0.00128
⛔  roa                                                SHAP=0.00127
⛔  earnings_surprise                                  SHAP=0.00122
⛔  bookValue                                          SHAP=0.00118
⛔  totalLiabilities                                   SHAP=0.00114
⛔  fcf_ps_grw_qoq_to_totalAssets                      SHAP=0.00113
⛔  goodwill                                           SHAP=0.00112
⛔  fcf_ps_grw_4q                                      SHAP=0.00106
⛔  eps_grw_pop_sector_z                               SHAP=0.00100
⛔  freeCashFlow                                       SHAP=0.00090
⛔  current_ratio                                      SHAP=0.00090
⛔  ebitda_margin                                      SHAP=0.00087
⛔  totalRevenue                                       SHAP=0.00082
⛔  quick_ratio_sector_z                               SHAP=0.00078
⛔  free_cash_flow                                     SHAP=0.00073
⛔  ocf_to_total_liabilities                           SHAP=0.00062
⛔  sector_Energy                                      SHAP=0.00062
⛔  returnOnTangibleAssets                             SHAP=0.00059
⛔  avgDilutedShares                                   SHAP=0.00050
⛔  sector_Basic Materials                             SHAP=0.00034
⛔  sector_Technology                                  SHAP=0.00025
⛔  sector_Real Estate                                 SHAP=0.00025
⛔  sector_Financial Services                          SHAP=0.00019
⛔  sector_Industrials                                 SHAP=0.00011
⛔  sector_Consumer Cyclical                           SHAP=0.00010
⛔  sector_Communication Services                      SHAP=0.00009
⛔  sector_Healthcare                                  SHAP=0.00007
⛔  sector_Utilities                                   SHAP=0.00004
⛔  sector_Consumer Defensive                          SHAP=0.00002
⛔  fcf_ps_grw_pop                                     SHAP=0.00000
⛔  rev_grw_pop                                        SHAP=0.00000
⛔  eps_grw_pop                                        SHAP=0.00000
⛔  unemp_x_rev_grw                                    SHAP=0.00000
⛔  ocf_to_net_income                                  SHAP=0.00000
⛔  quick_ratio                                        SHAP=0.00000
⛔  current_ratio_sector_z                             SHAP=0.00000

The symbol target encoding? SHAP likes it. It moves the proverbial needle.

But when it comes to the sector and industry, the one-hot encodings, neither of those seem to be adding any real benefit.

Our R-squared was better than the one we had with one-hot encoded industries - but less than our previous and best runs with NO encodings as shown below.

Symbol Target-Encoded and Sector One-Hot Encoded:
🪓  Pruning all splits - train, val, test - to the pruned set of features (final_features).
Training pruned model...
FULL R² -- Train: 0.4708, Validation: 0.0757, Test: 0.0868
PRUNED R² -- Train: 0.3847, Validation: 0.0779, Test: 0.0486
Selected FULL model based on test R².
Previous best test R²: 0.1126
Current model test R²: 0.0868
Current model did NOT exceed best. Loading previous best model and features.
Final Model Test Metrics -- R²: 0.1126, RMSE: 0.2154, MAE: 0.1485
Feature importance summary:
  → Total features evaluated: 78

So - one last run, with only the Symbol Target Encoding (not Sector and Industry)
🪓  Pruning all splits - train, val, test - to the pruned set of features (final_features).
Training pruned model...
FULL R² -- Train: 0.5169, Validation: 0.0788, Test: 0.0463
PRUNED R² -- Train: 0.2876, Validation: 0.0456, Test: 0.0083
Selected FULL model based on test R².
Previous best test R²: 0.1126
Current model test R²: 0.0463
Current model did NOT exceed best. Loading previous best model and features.
Final Model Test Metrics -- R²: 0.1126, RMSE: 0.2154, MAE: 0.1479
Feature importance summary:
  → Total features evaluated: 78

And...back to no symbol, sector, industry
🪓  Pruning all splits - train, val, test - to the pruned set of features (final_features).
Training pruned model...
FULL R² -- Train: 0.4876, Validation: 0.1095, Test: 0.0892
PRUNED R² -- Train: 0.2870, Validation: 0.0713, Test: 0.0547
Selected FULL model based on test R².
Previous best test R²: 0.1126
Current model test R²: 0.0892
Current model did NOT exceed best. Loading previous best model and features.
Final Model Test Metrics -- R²: 0.1126, RMSE: 0.2188, MAE: 0.1486

Once again - no encoding at all on symbol, sector and industry...
🪓  Pruning all splits - train, val, test - to the pruned set of features (final_features).
Training pruned model...
FULL R² -- Train: 0.4985, Validation: 0.1051, Test: 0.0969
PRUNED R² -- Train: 0.2742, Validation: 0.0981, Test: 0.0869
Selected FULL model based on test R².
Previous best test R²: 0.1126
Current model test R²: 0.0969
Current model did NOT exceed best. Loading previous best model and features.
Final Model Test Metrics -- R²: 0.1126, RMSE: 0.2160, MAE: 0.1487
Feature importance summary:
  → Total features evaluated: 78

Conclusion: These encodings did not move the R-squared at all, and in fact moved it backwards. So I need to make sure that there is no benefit to doing this encoding and if not, I will leave those out as the model was running originally.

I Have More Data Now - Enough for an AI RNN LSTM Model?

I have a LOT more data now than I did before. And an advanced architecture to process it.

Should I consider an RNN?

I knew I couldn't really pull it off with the Annual data I had - because by the time you split the data for training, validation, and testing there isn't enough to feed the algorithm.  

But - Quarterly! I have a LOT of quarterly data now, many statements per symbol across quarter-dates. ~70K rows of data!!!

So let's try doing an LSTM....I wrote a standalone LSTM, using Keras. Just a few lines of code. 

One important note about this! 

Do NOT mix your data processing, and or your XGBoost code, with neural network code!!! ALWAYS create a brand new virtual environment for your neural RNN code, because if you choose Keras or the other competing frameworks, they will require specific versions of Python libraries that may conflict with your data processing and/or XGBoost libraries!

Now. With that important disclaimer, the small sample of code. We will highlight in blue since my blog tool apparently has no code block format.

# -------------------------
# Train/test split
# -------------------------
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=TEST_SIZE, random_state=RANDOM_STATE
)

# -------------------------
# Build LSTM model
# -------------------------
model = Sequential()
model.add(LSTM(32, input_shape=(SEQ_LEN, len(feature_cols)), return_sequences=False))
model.add(Dropout(0.2))
model.add(Dense(16, activation='relu'))
model.add(Dense(1))  # regression output

model.compile(optimizer='adam', loss='mse')

# Early stopping
es = EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)

# -------------------------
# Train
# -------------------------
history = model.fit(
    X_train, y_train,
    validation_split=0.1,
    epochs=50,
    batch_size=16,
    callbacks=[es],
    verbose=1
)

# -------------------------
# Evaluate
# -------------------------
y_pred = model.predict(X_test).flatten()
r2 = r2_score(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)
print(f"R²: {r2:.3f}, MAE: {mae:.3f}")

Well, how did it go?

The previous XGBoost r-squared value, was .11-.12 consistently. Now, we are getting .17-.19. This is a noticeable significant improvement!