مقاله سال ۲۰۱۶

Consider the case where an analyst has two competing one-step-ahead forecasts for a time series variable yt , namely yˆ۱,t and yˆ۲,t , for a sample t = 1, 2, . . . , T . The forecasts have the associated forecast errors εˆ۱,t and εˆ۲,t , respectively. To examine which of the two sets of forecasts provides the best accuracy, the analyst can use criteria based on some average or median of loss functions of the forecast errors. Well-known examples include the root mean squared error (RMSE) and the median absolute error (MAE); see Hyndman and Koehler (2006) for an exhaustive list of criteria, and also Table 1.

t of criteria, and also Table 1. As there is always one set of forecasts that scores lower on some criterion, it seems wise to test whether any observed differences in forecast performances are statistically significant. To test statistically whether the obtained values of these criteria are equal, the analyst can rely on the methodology proposed by Diebold and Mariano (1995) (DM); see also Diebold (2015) for a recent review. This methodology is based on the loss functions li,t = f(yt, yˆi,t) for i = 1, 2. Denoting the sample mean loss differential by d¯ ۱۲, that is, d¯ ۱۲ = ۱ T T 1 (l1,t − l2,t), and a consistentestimate of the standard deviation of d¯ ۱۲ by σˆd¯ ۱۲ , the DM test for one-step-ahead forecasts is DM = d¯ ۱۲ σˆd¯ ۱۲ ∼ N(0, 1),

under the null hypothesis of equal forecast accuracy. Even though Diebold and Mariano (1995, p. 254) claim that this result holds for any arbitrary function f , it is quite clear that the function should allow for proper moment conditions in order to yield the asymptotic normality of the test. In fact, as will be argued in Section 2 below, many of the functions that are commonly applied in the forecast literature fail to qualify as useful functions for the DM methodology. This note continues with a brief summary of typical functions in Section 2, along with a concise discussion of which of these functions are useful in the DM framework. It is found that the absolute scaled error (ASE) recommended by Hyndman and Koehler (2006) does have the favorable properties, while various other criteria do not. Section 3 reports on limited simulation experiments which support these insights. The main conclusion of this note is to confirm that the use of the Mean ASE (MASE) criterion is recommended.