Winner's Curse? On Pace, Progress, and Empirical Rigor
Abstract: The field of ML is distinguished both by rapid innovation and rapid dissemination of results. While the pace of progress has been extraordinary by any measure, in this paper we explore potential issues that we believe to be arising as a result. In particular, we observe that the rate of empirical advancement may not have been matched by consistent increase in the level of empirical rigor across the field as a whole. This short position paper highlights examples where progress has actually been slowed as a result, offers thoughts on incentive structures currently at play, and gives suggestions as seeds for discussions on productive change.
Keywords: empirical rigor
TL;DR: A position paper on empirical rigor in machine learning, meant to foster a discussion on the subject.
9 Replies
Loading