Double descent and sufficiency

It has recently been shown that the bias-variance tradeoff disappears when using machine learning techniques, particulary those associated with deep learning, in an overparametrised regime (i.e., p much bigger than n). Both in simulation experiments and in theoretical work, it is observed that the test error (or risk) of an estimator exhibits a double descent behaviour as a function of the number of parameters, meaning that the traditional U-shaped bias variance tradeoff persists in the underparametrised regime (p smaller than n), but that the risk tapers off when entering the overparamerised regime. This phenomenon runs counter to conventional statistical wisdom. A classical statistical result says that an estimator can always be improved upon by conditioning on a sufficienct statistic (the Rao--Blackwell theorem), thus connecting the notion of sufficiency to the bias-variance tradeoff. When venturing outside of model conditions, however, sufficiency is an elusive concept that may be operationalised in different ways. The theme of this research is to develop concepts of (approximate) sufficiency amenable to theoretical work with minimal conditions on the data generating process; and to connect classical results on sufficiency and estimation with modern findings on the bias-variances tradeoff, with the goal of providing novel insights into the latter (and perhaps also the former).