Abstract
Dimitriadis et al. (2021) have introduced the CORP (Consistent, Optimally binned, Reproducible, and Pool-Adjacent-Violators (PAV) algorithm-based) approach based on nonparametric isotonic regression for calibration of probabilistic forecasts. The CORP approach generates reliability curves, being the graph of the PAV-(re)calibrated forecast probability. Dimitriadis et al. (2024) proposed a triplet of diagnostic graphics representing different features of forecast performance based on Brier’s score decomposition: reliability curves produced by CORP, receiver operating characteristic (ROC) curves which evaluate discrimination ability and Murphy curves for overall assessment of predictive performance and economic utility. The area under the ROC curve (AUC) is a popular metric for the class-distribution robust learning framework. However, the traditional machine learning models trained with AUC are not well studied for cost sensitive decision problems. The exeption is the work of Hern´andes-Orallo et al. (2013) demonstrating that the ROC curves can be transformed into the cost space. This update is equivalent to computing the area under the convex hull of ROC curves. Thus, the AUC can be seen as the performance of the model with uniform cost distribution, being an unreasonable for practical needs. Using the idea of Hern´andes-Orallo et al. (2013), Shao et al. (2023) introduced the notion of weighted ROC curve in cost space joining the robustness of the model to the class distribution and cost distribution, i.e., transforming AUC to the cost-sensitive learning. A new environment is created where the costs are treated like a dataset to share out an arbitrary unknown cost distribution. The aim in this talk is to apply the methodology suggested by Shao et al. (2023) to get the corresponding weighted version in cost space of Murphy curves, enriching the existing graphical tools and their interpretations. Related minimization of the mean cost functional will be discussed as well. The proposed tools will be illustrated in empirical economics example.
Organisation
Sabrina Mulinacci