14:20  14:40
Continual RareClass Recognition with Emerging Novel Subclasses (152)
Hung Nguyen (Carnegie Mellon University), Xuejian Wang (Carnegie Mellon University), Leman Akoglu (Carnegie Mellon University)
Given a labeled dataset that contains a rare (or minority) class of ofinterest instances, as well as a large class of instances that are not of interest,how can we learn to recognize future ofinterest instances over a continuous stream?We introduce RaRecognize, which (i) estimates a general decision boundary between the rare and the majority class, (ii) learns to recognize individual rare subclasses that exist within the training data, as well as (iii) flags instances from previously unseen rare subclasses as newly emerging.The learner in (i) is general in the sense that by construction it is dissimilar to the specialized learners in (ii), thus distinguishes minority from the majority without overly tuning to what is seen in the training data. Thanks to this generality, RaRecognize ignores all future instances that it labels as majorityand recognizes the recurrent as well as emerging rare subclasses only. This saves effort at test time as well as ensures that the model size grows moderately over time as it only maintains specialized minority learners.Through extensive experiments, we show that RaRecognize outperforms stateofthe art baselines on three realworld datasets that contain corporaterisk and disaster documents as rare classes.
Reproducible Research

15:00  15:20
Unjustified Classification Regions and Counterfactual Explanations In Machine Learning (226)
Thibault Laugel (Sorbonne Université), MarieJeanne Lesot (Sorbonne Université), Christophe Marsala (Sorbonne Université), Xavier Renard (AXA, Paris), Marcin Detyniecki (Sorbonne Université; AXA, Paris; Polish Academy of Science)
Posthoc interpretability approaches, although powerful tools to generate explanations for predictions made by a trained blackbox model, have been shown to be vulnerable to issues caused by lack of robustness of the classifier. In particular, this paper focuses on the notion of explanation justification, defined as connectedness to groundtruth data, in the context of counterfactuals. In this work, we explore the extent of the risk of generating unjustified explanations. We propose an empirical study to assess the vulnerability of classifiers and show that the chosen learning algorithm heavily impacts the vulnerability of the model. Additionally, we show that stateoftheart posthoc counterfactual approaches can minimize the impact of this risk by generating less local explanations.
Reproducible Research

14:40  15:00
Shift Happens: Adjusting Classifiers (479)
Theodore James Thibault Heiser (University of Tartu), MariLiis Allikivi (University of Tartu), Meelis Kull (University of Tartu)
Minimizing expected loss measured by a proper scoring rule, such as Brier score or logloss (crossentropy), is a common objective while training a probabilistic classifier. If the data have experienced dataset shift where the class distributions change posttraining, then often the model's performance will decrease, overestimating the probabilities of some classes while underestimating the others on average. We propose unbounded and bounded general adjustment (UGA and BGA) methods that transform all predictions to (re)equalize the average prediction and the class distribution. These methods act differently depending on which proper scoring rule is to be minimized, and we have a theoretical guarantee of reducing loss on test data, if the exact class distribution is known. We also demonstrate experimentally that, when in practice the class distribution is known only approximately, there is often still a reduction in loss depending on the amount of shift and the precision to which the class distribution is known.
Reproducible Research

15:20  15:40
Nonparametric Bayesian Isotonic Calibration: Fighting Overconfidence in Binary Classification (587)
MariLiis Allikivi (University of Tartu), Meelis Kull (University of Tartu)
Classifiers can often output a score or a probability indicating how sure they are about the predicted class. Classifier calibration methods can map these into calibrated class probabilities, supporting costoptimal decision making. Isotonic calibration is the standard nonparametric calibration method for binary classifiers, and it can be shown to yield the most likely monotonic calibration map on the given data, where monotonicity means that instances with higher predicted scores are more likely to be positive. Another nonparametric method is ENIR (ensemble of nearisotonic regression models) which allows for some nonmonotonicity, but adds a penalty for it. We first demonstrate that these two methods tend to be overconfident and show that applying label smoothing improves calibration of both methods in more than 90% of studied cases. Unfortunately, label smoothing reduces confidence on the underconfident predictions also, and it does not reduce the raggedness of isotonic calibration. As the main contribution we propose a nonparametric Bayesian isotonic calibration method which has the flexibility of isotonic calibration to fit maps of all monotonic shapes but it adds smoothness and reduces overconfidence without requiring label smoothing. The method introduces a prior over piecewise linear monotonic calibration maps and uses a simple Monte Carlo sampling based approach to approximate the posterior mean calibration map.Our experiments demonstrate that on average the proposed method results in better calibrated probabilities than the stateoftheart calibration methods, including isotonic calibration and ENIR.
Reproducible Research

14:00  14:20
Aggregating Algorithm for Prediction of Packs (J02)
Dmitry Adamskiy, Anthony Bellotti, Raisa Dzhamtyrova, Yuri Kalnishkan

15:40  16:00
Deeply Supervised Model for ClickThrough Rate Prediction in Sponsored Search (J03)
Jelena Gligorijevic, Djordje Gligorijevic, Ivan Stojkovic, Xiao Bai, Amit Goyal, Zoran Obradovic
