AI / ML - Data Source Comparison with More Data

"To validate generalizability, I ran the same model architecture against two datasets: a limited Yahoo-based dataset and a deeper Stockdex/Macrotrends dataset. Not only did the model trained on Stockdex data achieve a higher R² score (benefiting from more years of data), but the feature importances and pillar scores remained largely stable between the two. This consistency reinforces the robustness of the feature selection process and confirms that the model is learning persistent financial patterns, not artifacts of a specific data source or time window."

Details:

I wanted to move to an LSTM model to predict stock returns, but I was fortunate and patience enough to really read-up and plan this, before just diving into the deep end of the pool.

I learned that the "true AI" models - Transformers, RNNs, et al (of which LSTM is a subclass), require more data. I didn't have anywhere enough data using Yahoo, which gives 4-5 years (at best) of data. And because I was calculating momentum, yoy growth and such, I would always lose one of the years (rows) right off the bat - a significant percentage of already-scarce data.

So, in digging around, I found a Python library called stockdex. It is architected to be able to use multiple data sets, but the default is macrotrends. 

But using this library and source this left several challenges:

1. No quarterly data in the Python API - although the website does have a "Format" drop down for Quarterly and Annual.
2. The data was pivoted from Yahoo data. Yahoo put columns as items (x), and the time periods as rows (y). This stockdex API downloaded it opposite. 
3. The stockdex had no "names" for the items. 

Ultimately, I decided to use this because it returned a LOT more years of data.

1. First, I put some code together to download the raw data (statements), and then "pivot" the data to match Yahoo's format. 
2. Then, I used a mapping approach to change the columns from Macrotrends to Yahoo - so that I didn't have to change my logic that parsed statements.
3. I did have to do a run-tweak on the Metrics and Ratios, and fix certain columns that were not coming in correctly.
4. Lastly, I ran the model - same one as Yahoo and was able to keep the model logic essentially unchanged. 

The training took a LOT longer on Stockdex. The combined train+val had 14,436 rows on it.
Here is what we got:
FULL R² -- Train: 0.8277, Validation: 0.3441, Test: 0.3537
PRUNED R² -- Train: 0.7714, Validation: 0.3146, Test: 0.3429
Selected FULL model based on test R².
Final Model Test Metrics -- R²: 0.3537, RMSE: 0.3315, MAE: 0.2282
Feature importance summary:
  → Total features evaluated: 79
  → Non-zero importance features: 75

The model running and scoring, it took a very very long time. Finally it came out with this Top 25 list.
Top 25 Stocks by Final adj_pillar_score:
     symbol  adj_pillar_score  improvement_bonus  pillar_score
1209    RUN          1.156474               0.05      1.106474
884     LZM          1.020226               0.05      0.970226
97     ARBK          1.018518               0.00      1.018518
277     CGC          1.009068               0.02      0.989068
262     CCM          0.982228               0.02      0.962228
821    KUKE          0.964131               0.00      0.964131
1415   TRIB          0.963591               0.02      0.943591
1473   UXIN          0.961206               0.05      0.911206
571    FWRD          0.957156               0.05      0.907156
859    LOCL          0.935929               0.00      0.935929
1069   OTLY          0.896289               0.00      0.896289
894     MBI          0.895565               0.05      0.845565
1159   QDEL          0.890248               0.05      0.840248
1039    ODV          0.861127               0.00      0.861127
1522    WBX          0.860391               0.00      0.860391
1578   ZEPP          0.856097               0.02      0.836097
860    LOGC          0.846546               0.05      0.796546
990     NIO          0.811563               0.02      0.791563
1428    TSE          0.775067               0.05      0.725067
930    MODV          0.773322               0.05      0.723322
817    KRNY          0.770282               0.05      0.720282
1545    WNC          0.767113               0.02      0.747113
65     ALUR          0.756362               0.00      0.756362
813    KPTI          0.749644               0.05      0.699644
1316   SRFM          0.743651               0.00      0.743651

Then I ran the smaller Yahoo model:
Training pruned model...
FULL R² -- Train: 0.8181, Validation: 0.3613, Test: 0.2503
PRUNED R² -- Train: 0.8310, Validation: 0.3765, Test: 0.2693
Selected PRUNED model based on test R².
Final Model Test Metrics -- R²: 0.2693, RMSE: 0.4091, MAE: 0.2606
Feature importance summary:
  → Total features evaluated: 30
  → Non-zero importance features: 30

And, the Top 25 report for that one looks like this:
Top 25 Stocks by Final adj_pillar_score:
     symbol  adj_pillar_score  improvement_bonus  pillar_score
907    MOGU          1.532545               0.05      1.482545
1233    SKE          1.345178               0.05      1.295178
1170   RPTX          1.334966               0.02      1.314966
419      DQ          1.305644               0.05      1.255644
1211    SES          1.280886               0.05      1.230886
702    IFRX          1.259426               0.00      1.259426
908    MOLN          1.244191               0.02      1.224191
1161   RLYB          1.237648               0.00      1.237648
176    BHVN          1.232199               0.05      1.182199
512    FEDU          1.218868               0.05      1.168868
977    NPWR          1.205679               0.00      1.205679
1533     YQ          1.204367               0.02      1.184367
11     ABUS          1.201539               0.00      1.201539
58     ALLK          1.192839               0.02      1.172839
1249    SMR          1.154462               0.00      1.154462
63     ALXO          1.148672               0.02      1.128672
1482    WBX          1.140147               0.00      1.140147
987    NUVB          1.139138               0.00      1.139138
1128     QS          1.130001               0.02      1.110001
864     LZM          1.098632               0.00      1.098632
16     ACHR          1.094872               0.02      1.074872
1176    RUN          1.059293               0.02      1.039293
758    JMIA          1.053711               0.00      1.053711
94     ARBK          1.049382               0.00      1.049382
1086   PHVS          1.039269               0.05      0.989269

Symbols that appear in both top 25 lists:

• RUN (Stockdex rank 1, Yahoo rank 23)

• LZM (Stockdex rank 2, Yahoo rank 21)

• ARBK (Stockdex rank 3, Yahoo rank 24)

• WBX (Stockdex rank 16, Yahoo rank 17)

interesting...

Comparing the top sector reports:

Side-by-side Overlap Analysis Approach

Sector	Symbol(s) (Overlap)	Stockdex Rank & Score	Yahoo Rank & Score	Notes
Basic Materials	LZM, ODV, MAGN	LZM #1 (1.0202), ODV #2 (0.8611), MAGN #4 (0.7287)	LZM #2 (1.0986), ODV #3 (0.9233), MAGN #5 (0.8681)	Close agreement; Yahoo scores higher overall
Communication Services	KUKE, FUBO	KUKE #1 (0.9641), FUBO #4 (0.5936)	KUKE #5 (0.5559), FUBO #4 (0.6544)	Generally consistent rank order
Consumer Cyclical	NIO, UXIN, LOGC	UXIN #1 (0.9612), LOGC #2 (0.8465), NIO #3 (0.8116)	NIO #5 (0.9276), SES #2 (1.2809) not in Stockdex	Partial overlap; Yahoo picks also include SES, MOGU
Consumer Defensive	YQ, OTLY, LOCL	LOCL #1 (0.9359), OTLY #2 (0.8963), YQ #4 (0.6482)	YQ #2 (1.2044), FEDU #1 (1.2189), LOCL missing	Some overlap, differences in top picks
Energy	DWSN, PBF	DWSN #2 (0.6201), PBF #3 (0.4305)	DWSN #1 (0.7613), PBF #3 (0.3556)	Rankings closely aligned
Financial Services	ARBK, MBI, KRNY	ARBK #1 (1.0185), MBI #2 (0.8956), KRNY #3 (0.7703)	ARBK #1 (1.0494), GREE #2 (0.9502) missing MBI, KRNY	Partial overlap
Healthcare	CGC, CCM, TRIB	CGC #1 (1.0091), CCM #2 (0.9822), TRIB #3 (0.9636)	RPTX #1 (1.3350), IFRX #2 (1.2594) missing CGC,etc	Different picks mostly
Industrials	FWRD, EVTL	FWRD #1 (0.9572), EVTL #4 (0.7314)	NPWR #1 (1.2057), EVTL #5 (0.9657)	Some overlap
Real Estate	OPAD, AIV	AIV #1 (0.7303), OPAD #2 (0.7286)	DOUG #1 (0.8116), OPAD #5 (0.5578)	Partial overlap
Technology	RUN, WBX, ZEPP	RUN #1 (1.1565), WBX #2 (0.8604), ZEPP #3 (0.8561)	WBX #2 (1.1401), RUN #3 (1.0593), ZEPP missing	Strong agreement
Utilities	AQN	AQN #1 (0.7382)	OKLO #1 (0.8269) no AQN	Different picks

So - this is excellent model validation, I think. We see some differences due to the amount of time-period data we have, but the results are not widly different. 

I think I can now use this data in LSTM perhaps. Or whatever my next steps turn out to be, because I may - before LSTM - try to do some earnings transcript parsing for these if it's possible.

AI / ML - Modeling Fundamentals - Mistakes Found and Corrected

After adding Earnings Surprise Score data into my 1 year fwd return predicting model, I kind of felt as though I had hit the road on the model. The Earnings Surprise Score did move the needle. But with all of the effort in Feature Engineering I had put into this model, the only thing I really felt I could add to it, was sentiment (news). Given that news is more of a real-time concept, grows stale, and would be relevant for only the latest row of data, I decided to do some final reviews, and move on, or "graduate" to some new things - like maybe trying out a neural network or doing more current or real-time analysis. In fact, I had already tried a Quarterly model, but the R-squared on it was terrible and I decided not to use it - not to even ensemble it with the annual report data model.

So - I asked a few different LLMs to code review my model. And I was horrified to learn that because of using LLMs to continually tweak my model, I had wound up with issues related to "Snippet Integration". 

Specifically, I had some major issues:

1. Train/Test Split Happened Too Early or Multiple Times

•  Splitting data before full preprocessing (e.g., before feature scaling, imputation, or log-transforming the target).
•  Redundant train/test splits defined in multiple places — some commented, some active — leading to potential inconsistencies depending on which was used.

2. No Validation Set

•  Originally, data was split into training and test sets.
•  This meant that model tuning (e.g. SHAP threshold, hyperparameter selection) was inadvertently leaking test set information. 
  
  
•  Now corrected with a clean train/val/test split.

3. Inconsistent Preprocessing Between Train and Test

•  Preprocessing steps like imputation, outlier clipping, or scaling were not always applied after the split.
•  This risked information bleeding from test → train, violating standard ML practice.

4. Improper Handling of Invalid Target Values (fwdreturn)

•  NaN, inf, and unrealistic values (like ≤ -100%) were not being consistently filtered.

•  This led to silent corruption of both training and evaluation scores.

•  Now fixed with a strict invalid_mask and logging/reporting of dropped rows.

5. Redundant or Conflicting Feature Definitions

•  There were multiple, overlapping blocks like:

                features = [col for col in df.columns if ...]
                X = df[features]

•  This made it unclear which feature list was actually being used.

•  Sector z-scored and raw versions were sometimes duplicated or mixed without clarity.

6. Scaling and Z-Scoring Logic Was Not Modular or Controlled

•  Originally, some features were being z-scored after asset-scaling (which didn’t make sense).

•  Some metrics were scaled both to assets and z-scored sector-wise, which polluted the modeling signal.

•  Now addressed with clear separation and feature naming conventions.

7. SHAP Was Applied to a Noisy or Unclean Feature Space

•  Without proper pruning first (e.g., dropping all-NaN columns), SHAP feature importance included irrelevant or broken columns.
•  This could inflate feature count or misguide model interpretation.

•  Now resolved by cleaning feature set before SHAP and applying SHAP-based selection only on valid, imputed data.

One issue, was that the code for modeling was in the main() function which had gotten way too long and hard to read. This function had all of the training, testing, splitting/pruning, it had the model fitting in it, AND all of the scoring. I broke out the process of train/validate/test - and put that process into play for both the full model, as well as the SHAP-pruned model. Then I took the best r-squared from both runs and used the winning model.

AI / ML - Using a DAG Workflow for your Data Pipeline

I have been trying to find a lightweight mechanism to run my increasing number of scripts for my data pipeline.

I have looked at a lot of them. Most are heavy - requiring databases, message queues, and all that goes with a typical 3-4 tier application. Some run in Docker.

I tried using Windmill at first. I liked the GUI - very nice. The deal killer for me, was that Windmill wants to soak in and make its own copies of anything it runs. It can't just reach out and run scripts that, for example, are sitting in a directory path. It can't apparently (could be wrong on this but I think I am right), do a git clone to a directory and run the content from where it sits. It wants to pull it into its own internal database - as a copy. It wants to be a development environment. Not for me. Not what I was looking for. And I only want my "stuff" to be in a single spot.

I then tried using Prefect. What a mess. You have to write Python to use Prefect. Right away, the SQLite Eval database was locking when I did anything asynchronous. Then, stack traces, issues with the CLI, etc. I think they're changing this code too much. Out of frustration I killed the server and moved on.

My latest is DAGU - out of GitHub - Open Source. Wow - simple, says what it does, does what it says. It does not have some of the more advanced features, but it has a nice crisp well-designed and responsive UI, and it runs my stuff in a better way than cron can do.

Here is a sample screenshot. I like it.

 

AI/ML - Feature Engineering - Normalization

On my quarterly financials model, the R² is awful. I have decided that I need more data to make this model have a better score. 3-4 quarters is probably not enough. That means I need to build something to go to the SEC and parse it. 

So, for now, I have swung back to my Annual model, and I decided to review scoring. 

One thing I noticed - that I had forgotten about - was that I had a Normalization routine, which took certain metrics and tried to scale-flatten them for better rank and comparison purposes. This routine, it takes certain metrics, and divides them by Total Assets. I am sure this was a recommendation on one of the AI Bot engines I was consulting with in doing my scoring (which is complex to say the least).  

Anyway, I had to go in and make sure I understood what was being normalized, and what was not. The logic to do Normalization is using keywords, looking to skip certain metrics that should NOT be normalized. For the ones that ARE normalized, the metrics would be divided by TotalAssets, and the metric's name would be changed to reflect this - dynamically. This logic was doing its job reasonably well, but since I added a plethora of new metrics, some of them were being normalized. 

So this is the new logic. 
    # --- Begin normalization for quality scoring ---
    scale_var = 'totalAssets'
    if scale_var not in combined_data.columns:
        raise ValueError(f"{scale_var} not found in data columns!")

    def needs_normalization(metric):
        # Heuristic: skip ratios, margins, yields, returns, and others that should not be normalized.
        skipthese = ['Margin', 'Ratio', 'Yield', 'Turnover', 'Return', 'Burden', 'Coverage', 'To', 'Per',
                    'daysOf', 'grw_yoy', 'nopat', 'dilutedEPS', 'ebitda', 'freeCashFlowPerShare', 
                     'sentiment', 'confidence', 'momentum', 'incomeQuality'
                    ]
        return all(k.lower() not in metric.lower() for k in skipthese)

And this took me some time to get working properly. Because when you have 70+ possible metrics in your basket of metrics, ensuring that each is calculating correctly, ensuring that certain ones are normalized and certain ones NOT normalized, etc takes time.

 

AI/ML Feature Engineering - Adding Feature-Based Features

I added some new features (metrics) to my model. The Quarterly model.

To recap, I have downloaded quarterly statements for stock symbols, and I use these to calculate an absolute slew of metrics and ratios. Then I feed them into the XGBoost regression model, to figure out whether they can predict a forward return of stock price.

I added some macro economic indicators, because I felt that those might impact the quarterly price of a stock (short term) more than the pure fundamentals of the stock. 

The fundamentals are used in an annual model - a separate model - and in that model, the model is not distracted or interrupted with "events" or macroeconomics that get in the way of understanding the true health of a company based on fundamentals over a years-long period of time.

So - what did I add to the quarterly model?

• Consumer Sentiment
• Business Confidence
• Inflation Expectations
• Treasury Data (1,3,10 year)
• Unemployment 

And wow - did these variables kick in. At one point, I had the model up to .16. 

Unemployment did nothing, actually. And I wound up removing it as a noise factor. I also realized I had the fiscal quarter included, and removed that too since it, like sector and other descriptive variables, should not be in the model.

But - as I was about to put a wrap on it, I decided to do one more "push" to improve the R-squared value, and started fiddling around. I got cute, adding derived features. One of the things I did, was to add lag features for business confidence, consumer sentiment, inflation expectations. Interestingly, two of these shot to the top of influential metrics.

Feature Importance List Sorted by Importance (return price influence).
feature                                                 weight
business_confidence_lag1                0.059845
inflation_lag1                                       0.054764

But, others were a bust, with .00000 values.

I tried removing the original metrics and JUST keeping the lags - didn't really help.

Another thing worth noting, is that I added SHAP values - a topic I will get into more depth about shortly, perhaps in a subsequent post. SHAP (SHapley Additive exPlanations) is a method used to explain the output of machine learning models by assigning each feature an importance value for a specific prediction, so that models - like so many - are not completely "black box".

But one thing I noticed when I added the SHAP feature list, is that it does NOT match / line up with the feature importances that the XGBoost model espouses. 

So I definitely need to look into this.

AI/ML - Feature Engineering

Originally, when I first started this project to learn AI, I set it up thus:

Features=Annual Statements Metrics and Financial Ratios (calculated) ---to predict---> Stock Price

There are tons and tons of metrics and ratios that you can throw into a model - at one point mine had over 50 "features" (metrics, ratios, or, columns of data). 

Quickly, you get into Feature Engineering. 

You see, certain metrics are "circular" and co-dependent. You can not use price-derived metrics to try and predict price. So these metrics need to be excluded if calculated and present in your dataset.

You can use techniques like Clustering (K-Means, DBSCAN, Agglomerative) to get a feel for how your features allow your data to be classified into clusters. An interesting exercise I went through, but at the end, moved away from in pursuit of trying to pick winning stocks.

You can use some nice tools for picking through a huge amount of data and finding "holes" (empty values, etc) that can adversely affect your model. 

From a column (feature) perspective, you can: 

• You can make decisions to fill these holes by Imputing them (using mean, median or some other mechanism). 
• Or, you can drop these holes.

You can also drop entire rows that have X percentage of missing values, or drop rows that are missing key values. Figuring all of this out takes time. It is part of the Data Engineering.

Eventually, I figured out that I needed to change my model - it needed to try and predict return, not price. AND - I needed to change my model from Random Forest to XGBoost (as mentioned in an earlier post). 

So now, we will be doing this...

Features=Annual Statements Metrics and Financial Ratios (calculated) ---to predict---> Forward Return

Well, guess what? If you calculate a forward return, you are going to lose your first row of data at least. Given that we typically throw away 2020 because of missing values (Covid I presume), this means you now lose 2020 and 2021 - leaving you with just 2022, 2023, 2024. Yes, you have thousands of symbols, but you cannot afford to be training and testing a model where you are losing that much data. But - that is the way it has to be...most financial models are seeking return, not a price. Enlightening. Makes sense.

 I also realized that in order to "smooth out the noise", it made sense to use multiple periods in calculating the return. This causes MORE data to be lost. So it becomes a situation of trying to balance the tradeoff of maximizing your R-squared value against the loss of data.

I added some additional metrics (features): 

• qoq growth (eps, revenue, free cash flow) 
• momentum  

 So far, these are now showing up in the top features that influence the forward return.

But the R-squared for the Quarterly model is .09 - which is extremely low and not very valuable. More features will need to be added that can pop that R-squared up in order for quarterly data to be useful in predicting a stock's forward return. 

AI / ML - Fetching Data, Quality Control, Optimization and Review

Most of my time lately has been "refining" the model. 

For example, one of the things you need to really think about doing AI is where your data is coming from, and the quality of that data - and the price of that data. 

Originally, I was using FMP for data. But the unpaid version only gives you access to 100 symbols. You cannot get far with 100 symbols, even if you collect scores of metrics and ratios on them. So when you build your initial model on FMP, say using the TTM API on 100 symbols, you will need to consider an ante-up for more symbols, or look for symbols and data elsewhere.

I have considered writing an intelligent bot to "scour" the universe of financial sites to pull in data on symbols. There are a lot more of these where you can get current data (i.e. stock price), but when it comes to statements, you are going to need to hit the SEC itself, or an intermediary or broker. If you hit these sites without reading their disclosures, you can get banned. 

At a minimum, there is the rate limit. It is critical to understand how to rate limit your fetches. So using a scheduler, and running in batches (if they're supported) can really help you.  Another thing is intelligent caching. It makes no sense to get into hot water fetching the same statement for the same symbol you just fetched an hour ago. Once you have a complete statement, you probably want to keep it in your pocket, and only update on a lower frequency if you decide to update old data at all.

So most of my time lately has been spot checking the data, building some caching and trying to do some general improvement on the processing and flow.

I found a couple of nice python tools one can use to load csv files: tabview, and vizidata. The latter is a bit more robust. Having a csv viewer is a game changer if you want to stay in a terminal and not "point and click".

With a tool like this, you can really start to backtrack into missing holes of data. I had one metric for example that was a typo (single letter) and had NO data for that metric. I had other issues with division by zero errors, Panda Dataframe vs Series issues, etc. 

You also have to pay attention to these python algorithms, and what they spit out. The stuff may look like intimidating jibberish, but it's there for a reason and taking the time to really examine it can pay off quite a bit. For example, I decided to exclude certain metrics because they had circular influence. And when you make a change like that, the feature influences can change drastically. 

 

AI / ML - Random Forest, Data Wrestling and Z-Scores

I was running my AI scoring algorithm, which takes as inputs a bunch of calculated metrics and ratios (features - X axis), and feeds those into a Decision Tree algorithm (Random Forest), against a price prediction (Y axis), which then prints out a report that shows how well the algorithm performed in general (R-squared), and a list of features sorted by their influence on the Y variable (price). 

There are numerous algorithms that can do this - the simplest being a Linear Regression model.  Decision Trees offer a faster and more efficient - and perhaps more accurate - alternative to linear regression, provided that the tree is pruned and managed correctly and that the tree doesn't get lopsided or imbalanced.

But I ran into problems, especially when checking the results and data carefully. And most of the issues, were related to the data itself.

Data Alignment
I noticed that the predictive z-scores for my features didn't "line up" when I printed them twice. Turns out, this was a data alignment issue. When you are using dataframes, and making copies of these dataframes and merging them, you need to be very very careful or a column can get shifted.

This alignment issue was affecting my model because the metric that WAS a profitability metric, was now being assigned to a solvency metric. Now that I have this fixed, things look much more sensible. But making sure your dataframes are aligned, is a hard-learned lesson.

Outliers
Other issues I ran into today had to do with the fact that when I printed a report out (a weighted scoring report), certain values were far and away better than others. I didn't understand this, and discussed it with the AI I am using as a consultant, who suggested I print out z-scores.

Well, if we look below, we have an evToEBITDA metric of 10.392 (insane value) on 2023 Paramount reporting data.

=== Z-scores for PARA on 2023-12-31 ===
Z-scores for PARA on 2023-12-31 in pillar 'Profitability':
  grossProfitMargin: -0.263
  operatingProfitMargin: 0.038
  netProfitMargin: 0.029
  returnOnAssets: -0.033
  returnOnEquity: -0.006
  returnOnCapitalEmployed: -0.089
  returnOnTangibleAssets: 0.004
  earningsYield: 0.008
  freeCashFlowYield: 0.000
  nopat_to_totalAssets: -0.170
  operatingReturnOnAssets: -0.215
  returnOnInvestedCapital: -0.031
  ebitda_to_totalAssets: -0.384
  operatingCashFlowToSales: 0.036
  evToSales: -0.044
  evToOperatingCashFlow: 0.054
  evToEBITDA: 10.392
  evToFreeCashFlow: 0.039
 
I audited the metrics and statements, and indeed this is correct - based on what Yahoo was returning to me on the income statement for that year (Normalized EBITDA). The unnormalized EBITDA was better, but in most cases, analysts use the Normalized value. You can't do one-offs in your code for things like this, so what do you do?

I couldn't drop the row, because I was already dropping so many 2020 rows of bad data (due to Covid I suspect). I drop rows that are missing >35% of metrics. When you get a row that has all of the values you need, you tend to want to use it. I don't have code that drops rows that don't have specific dealbreaker metrics - maybe I should, but there are so many metrics that generally I figure I can score and rank even if I am missing one here or there, even a fairly well-known or important one. 

So - what do you do?

Winsorization. In other words, capping. It might make sense to invest the effort in Winsorizing all of the metrics and ratios. But for now, I am only doing it on these EBITDA ones.

AI / ML - It's All About the Data. Imputation and Clustering Algorithms

In spare time, I have been working on a Fintech project, which is done in conjunction with a thick book I have been reading called Machine Learning for Algorithmic Trading by Stefan Jansen.

I am mostly finished with this book, and have coded - from scratch - my own implementations of the concepts introduced in this book. 

What have I learned thus far?

It is ALL ABOUT THE DATA. Most of my time has been scrutinizing the data. Disqualifying data, throwing away of imputing data that has no values, and Winsorizing/capping data values so that they don't skew into outliers.

Dates. Dates have always been a problem. Dropping timestamps off of dates properly so that date comparisons and date math work properly.

So far, a lot of what I have done is data clustering, using algorithms like DBSCAN, K-Means, Agglomerative, etc to find useful cluster patterns. Regression techniques to find correlations. The models and scoring so far are my own "secret sauce" Deterministic models. But I do plan to snap in some AI to do automatic weight adjustment soon. 

Right now, I am using my own Deterministic scoring model - so it can be used as a comparative baseline. But eventually I will enhance this to be more dynamic through self-learning.