Ask HN: How do you test your Machine Learning models?
Hey all!
I'm looking to get some inputs from professional engineers in Machine Learning about how they test their models that might enter production.
I ran a survey in my own network before through a Twitter poll, and found out that over 70% of respondents were not fully confident in the abilities of their models, with 33% of all respondents saying that they are not confident in the abilities of their models at all.
Now I'd like to find out why and how to solve this. I prepared this survey and it would mean a lot if you could take 2 minutes to fill this one out.
I'm looking to write an article about ML testing methods based on those, and will be sure to share that here (and to anyone who requests it).
https://dominicmonn.typeform.com/to/n2AbR7
For production we usually just use some form of a/b testing, as we found out that the offline metrics are often misleading. When the offline metrics of your model Like f1 or whatever are good enough we just promote the model to production competing we the other models in a multiarmed bandit way.
I'm interested to share our experience. Do you have a link to your Twitter poll? You can find my contact information in the profile if you don't want to share infos here, happy to chat!
The survey is here: https://dominicmonn.typeform.com/to/n2AbR7
This is the poll that sprouted this all: https://twitter.com/dqmonn/status/1139092798521712641