It certainly vexes me that, when training and evaluating models, I sometimes get better results and sometimes worse. There's a certain amount of randomness associated with the evaluation data (e.g. you might get the right or wrong answer for a particular question with a binomial distribution, or in a real world problem where you train + eval everyday you'll find some days you have easier problems than other days.) Another problem though is that you might sometimes get a better model or sometimes get a worse model and that's what this paper is about.

It drives me up the wall that people write papers where they compare a large number of models and mark the highest accuracy/F1/AUC in bold but never do an analysis to ask "are the differences significant?"