A Model-Agnostic Heuristics for Selective Classification

You are here

TitleA Model-Agnostic Heuristics for Selective Classification
Publication TypeConference Paper
Year of Publication2023
AuthorsPugnana, A, Ruggieri, S
Conference NameThirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, 2023, Washington, DC, USA, February 7-14, 2023
PublisherAAAI Press
AbstractSelective classification (also known as classification with reject option) conservatively extends a classifier with a selection function to determine whether or not a prediction should be accepted (i.e., trusted, used, deployed). This is a highly relevant issue in socially sensitive tasks, such as credit scoring. State-of-the-art approaches rely on Deep Neural Networks (DNNs) that train at the same time both the classifier and the selection function. These approaches are model-specific and computationally expensive. We propose a model-agnostic approach, as it can work with any base probabilistic binary classification algorithm, and it can be scalable to large tabular datasets if the base classifier is so. The proposed algorithm, called SCROSS, exploits a cross-fitting strategy and theoretical results for quantile estimation to build the selection function. Experiments on real-world data show that SCROSS improves over existing methods.