Ask HN: How do you test your Machine Learning models?

13 points by dmonn a month ago

Hey all!

I'm looking to get some inputs from professional engineers in Machine Learning about how they test their models that might enter production.

I ran a survey in my own network before through a Twitter poll, and found out that over 70% of respondents were not fully confident in the abilities of their models, with 33% of all respondents saying that they are not confident in the abilities of their models at all.

Now I'd like to find out why and how to solve this. I prepared this survey and it would mean a lot if you could take 2 minutes to fill this one out.

I'm looking to write an article about ML testing methods based on those, and will be sure to share that here (and to anyone who requests it).

chudi a month ago

For production we usually just use some form of a/b testing, as we found out that the offline metrics are often misleading. When the offline metrics of your model Like f1 or whatever are good enough we just promote the model to production competing we the other models in a multiarmed bandit way.