If someone walked up to you on the street and asked you to predict whether a shopper entering a store (who you couldn’t see) would make a purchase, you’d have a 50/50 shot of getting it right.
But let’s say you were then able to see data on the shopper. You could analyze their purchase history, whether they’ve bought anything from the brand before, if they made any relevant Google searches about the types of products sold by the brand, etc. Would your chance of predicting it right increase?
Retail companies have to make predictions all the time. Without proper forecasting, they can’t know what to make, how much to manufacture, where to send it or stock it, and so on. But how can retailers be sure they’re getting the best output possible from their predictive models?
You can’t know which model is right for your organization—and a specific situation—unless you test it.
What is champion/challenger testing?
When a company needs to do any kind of predictions or forecasting, they’ll typically run it against a Machine Learning (ML) model. But the industry (our company included) is heading in a different direction.
With champion/challenger testing (also called multi-model, shadow, or A/B testing), you can test the same datasets against multiple models—e.g., XGBoost, Prophet, LightGBM, Graph Neural Networks (GNN), Random Forest, Linear Regression/Logistic, Regression, etc.—and compare the results, then tweak each model to find the best fit.
Which one is better for your organization or a specific situation? There’s no right answer until you test, and simultaneous A/B testing is the ideal way to do it. It’s just like when a marketing team runs A/B testing on their website, only this version is with datasets.
With our own Ekyam clients and prospects, we’ve seen that it’s not just “Model A is better.” It’s more nuanced than that. One model might be better during a high-traffic month, while a different one is better for low-traffic periods. Every model runs differently because it’s built on its own algorithm.
And a piece of good news: Champion/challenger testing has gotten easier. All the more reason to do it.
In the past, performing these tests on different models required a ton of effort to set up the infrastructure. But with Ekyam, retailers have more flexibility to perform and compare results against multiple models.
(Want to see how Ekyam runs multi-model testing to get you better results with a higher confidence level? Reach out to us.)
Why does champion/challenger testing matter for retail?
Champion/challenger testing has endless use cases in every industry, including retail. Here’s just one example where a retailer could use multi-model testing to calculate when orders will be delivered to customers:
- Mariah lives in NYC and placed an order today.
- If the retailer sends the order to Vendor X, our Model A model predicts it will ship in 2 days.
- But knowing the history of this vendor and their reliability, our Model B predicts 3 days to ship.
- We can then see when the vendor ultimately ships and use that to learn about the efficacy of each model.
The end customer (Mariah) won’t necessarily notice anything different on their end. But if the retailer uses champion/challenger testing over time to better predict shipping and decide which printers to use, then they can create surprise and delight for the customer if, for example, they get their order in 2 days instead of 4 days.
Are there any use cases that the end customer would notice? Not really. If the retailer is using multi-modeling on the website, the user might be shown different results (e.g. one model predicts these 3 products go best with a pair of jeans, and another model picks the 2 of the same but 1 is different). Whether it’s obvious to them is another question, but it’s unlikely.
There are endless champion/challenger testing use cases to understand how your business can improve: Predicting when you’ll run out of inventory, how many orders you’ll get, how a store will perform, how a specific product in a specific store will perform, etc.
It all comes down to increasing conversion. These tests can create very small differences, but they matter a lot. Better predictions → better planning → increasing conversion and LTV.
What the (near) future looks like
As a consumer, you are probably already seeing multi-model testing in your daily life. Streaming services like Netflix use it all the time to test how different models perform versus each other. What you see on your homepage is different from what your neighbor sees.
I predict that champion/challenger testing will become a common practice for retailers in the not-so-distant future: The next few years, if not months.
Uncovering high-success rate models for each situation—in one season
Champion/challenger testing has gotten easier and we’re using it here at Ekyam. Here’s how it works:
- You choose which of our models you want to execute the testing against.
- We ingest the data from your systems in a bulk upload. That ingestion is run against our multiple ML models on a real-time basis.
- We give the output back to your systems to consume and make a decision.
Example in practice: One client had over 100,000 orders and needed to know the expected output if they sent the order to a certain manufacturer. Ekaym ran multi-model testing to predict the results for them in a few minutes: i.e. 50% chance it will be delivered on X date, 80% chance it’s Y date, 90% it’s Z date.
A great part about running champion/challenger testing over time is that we learn which model works best for a certain retailer in a specific time period: X model works best for ABC Retailer during peak times, Y model works best during slow times.
That means the next time, you don’t have to debate which model will give you a high success rate. It only takes one season to have a high confidence level on which model works best.
Start your path to better predictions and unified systems: Book a demo
