Residuals - How to Separate two Inter-related Variables in a Model

I had someone on an Algotrading site, suggest that I should try to extract how much of momentum was sentiment - and vice-versa.

I had to look into this. It took me down a road. Which - as you shall see - introduced me to the concept of Residuals. And that is what this blog subject is about.

Chicken or egg? Is sentiment causing returns, or just reacting to them? Raw scores conflate these.

Put another way, if news sentiment causes/impacts/influences Momentum, how much of the Momentum is due to the Sentiment? 

Well, there is a way to ferret this out. It is called Residuals.  

Step 1: Regress sentiment on momentum features:

finbert_raw ~ lag_ret_20d + lag_ret_5d + volatility_5d

Step 2: Residual = what's left after stripping out price explanation:

finbert_signed_resid = finbert_raw - predicted(finbert | momentum)

 

Now, you can use the residual as a feature in the model, instead of just tossing in the raw values. Residuals are the cleanest sentiment alpha you can extract. Raw scores are 90% noise.

What does a residual capture?

✅ News sentiment BEFORE price moves (true alpha)
✅ Analyst upgrades/downgrades not reflected in price yet  
✅ Management guidance changes
✅ Product launch sentiment
❌ Earnings reactions (already in lag_ret_20d)

Jan 12th - The Death Spiral - Trading Lessons Learned

I learned quickly all of the issues with Algo Trading.

Lesson 1 - 4-to-1 Margins Were Enabled

I didn't take the time to look at the Configuration. So I was trading on Margin, and had to Liquidate a bunch of positions, some of which lost money.

Lesson 2 - Stop Limits Can Lead to a HOST of Problems

I had stop limit orders being set up to protect my downside. But, this led to THIS problem:

Buy HMY @ $25 → gap to $23 → stop → rebuy $23.50 
→ gap to $21 → stop → rebuy $21.80 → PDT flag → can't exit → $18

Basically, the stop limits were causing repetitive buying in falling-knife scenarios, because the model would pick the same symbol the next trade.

Lesson 3 - Some Symbols Are Halted or Delisted

I had one position, CLCO, which was halted. I have tried to force liquidate this position, but it is absolutely stuck. I will probably have to contact support. But this has thrown a major wrench into my Portfolio Manager, which needs to know that this money is "locked up" and account for that.

I also had issues with the model itself - in that the way the algorithm was working, it would pick the best ranked stocks, even if the signal strength or momentum was negative. So this had to be fixed as well, with some enhanced signal strength and momentum logic.

So - very expensive lessons. The back-testing of the model is turning out fine. But the Porfolio Manager and its transaction management was killing us. We burned about $10K of our $100K on this. Hopefully it starts to work better now with fixes to these issues.

My New Stock Prediction Model - Short Term Stock Prediction

Someone I work with has been working extensively on a Swing Trading model.

He has great financial experience from what I understand, but as he is in more of a management role and not in a day-to-day technical role, his programming skills might be just a tad or a step behind mine.

I have been watching him publish his short-term predictions, and his model is based on all kinds of things. I won't publish his secret sauce here, but he is using things that day traders tend to use, like RSI and Stochastics and such.

But, like my Financial Statement model - which tried to predict longer term buy-hold stocks,  his wasn't holding water either through back testing and results. One problem is that everything looks great in a bull market (rising tides lift all boats). And, to quote Buffet, "only when the tide goes out can you see who is swimming naked". So these models need to hold up in BOTH upturns and downturns.

I have decided to work on a new stock picking model. I am not sure yet how much I will blog about the specifics of it. But it is also a short-term model.

Stay Tuned