In plain words: Picking the best-looking trained network or settings only after seeing how each scored is cherry-picking that hides the failures. The usual fix, cross-validation on data splits, does not make it honest: such picks based on grading errors remain statistically untrustworthy.
Abstract · Misconduct in Post-Selections and Deep Learning
This is a theoretical paper on "Deep Learning" misconduct in particular and Post-Selection in general. As far as the author knows, the first peer-reviewed papers on Deep Learning misconduct are [32], [37], [36]. Regardless of learning modes, e.g., supervised, reinforcement, adversarial, and evolutional, almost all machine learning methods (except for a few methods that train a sole system) are rooted in the same misconduct -- cheating and hiding -- (1) cheating in the absence of a test and (2) hiding bad-looking data. It was reasoned in [32], [37], [36] that authors must report at least the average error of all trained networks, good and bad, on the validation set (called general cross-validation in this paper). Better, report also five percentage positions of ranked errors. From the new analysis here, we can see that the hidden culprit is Post-Selection. This is also true for Post-Selection on hand-tuned or searched hyperparameters, because they are random, depending on random observation data. Does cross-validation on data splits rescue Post-Selections from the Misconducts (1) and (2)? The new result here says: No. Specifically, this paper reveals that using cross-validation for data splits is insufficient to exonerate Post-Selections in machine learning. In general, Post-Selections of statistical learners based on their errors on the validation set are statistically invalid.
Juyang Weng
arXiv:2403.00773 · cs.LG · submitted Feb 13, 2024
abstract · pdf · html · 9 pages, 2 figures, published in peer-viewed conference proceedings, Proc. 2023 the 8th International Conf. on Control, Robotics and Cybernetics (CRC 2023), pp. 1-9, IEEE Press, ISBN: 979-8-3503-3057-1, Changsha, China, Dec. 22-24, 2023