New AI / ML Stock Picking Model

Okay. The new stock picking model is VASTLY different than the previous one.

We download news, then we feed the news into transformer models (Finbert) to gauge sentiment and calculate sentiment scores. Originally, I was using vader sentiment also, but I removed that because vader just doesn't seem to work well with financial news.

Downloading the news was a sizable effort. I decided to use a multi-threaded approach, where each news source had its own thread(s).

After all of the news has been downloaded - and scored - I send the data into module that attempts to map the news to stock symbols. Doing this well, required numerous iterative enhancements to the code. Then, the news is filtered according to various rule sets.

Originally, I saved the news in a csv file. But later, I had to convert this to a database approach. There is also some caching, to make sure we are not fetching the same article repeatedly. The code also does some heavy work to ascertain the proper date of the article. 

Once this is finished, another module sets about finding - or trying to find - prices. It will price the 0d for the symbol for the article. A cache is used so that we don't try to look up the same price for the same day for the same symbol more than once (reduced API calls). The code may pick up prices for the "zero day plus X days" depending on how long the article has been sitting. 

Once the article has been fully "aged out", it is purged and migrated into a training database. This allows us to train the model on "actual returns".

After the pricing, there are some analyzers that will examine the integrity of data. And if all looks well, the model is (re-trained) with the newly migrated data, and new predictions are made. This allows for a continual improvement.

Once predictions are made, a portfolio managed makes trades using a paper account, and some back test programs are used to compare performance against the S&P, and if desired, prior model versions. 

I am sure I have skipped over a lot of the complexity on this, but in a nutshell this is what we are doing. I will avoid discussing the "secret sauce" which is the feature engineering.