QuaPy/docs/source/manuals/label-shift-adaptation.md

Label Shift Adaptation

Quantification methods estimate the class prevalence values of a test sample, but a prevalence estimate is not, by itself, a classifier that has been corrected for the shift it describes. In some scenarios you may want the latter: a classifier whose decisions account for the (estimated) change in class priors between the training and the test distributions, rather than a single prevalence vector.

QuaPy provides two building blocks for this:

• qp.method.aggregative.ImportanceWeightQuantifier, an interface implemented by the quantifiers that natively compute a vector of importance weights as part of estimating the target prevalence.
• qp.classification.labelshift.LabelShiftedClassifier, a classifier wrapper that uses any quantifier (whether or not it implements the interface above) to adapt a classifier to a specific, (potentially) shifted batch of test instances.

The ImportanceWeightQuantifier interface

Some quantifiers (currently, RLLS, BBSEhard, and BBSEsoft; see the {ref}Regularized Learning under Label Shift (RLLS) <manuals/methods:Regularized Learning under Label Shift (RLLS)> and {ref}Black Box Shift Estimation (BBSE) <manuals/methods:Black Box Shift Estimation (BBSE)> sections of the methods manual) work by first estimating a vector of importance weights

w_y =

with :math:P and :math:Q the training and target class distributions, and only then rescaling the training prevalence by these weights to obtain the target prevalence estimate: :math:\hat{Q}(y) \propto w_y \cdot P(y). For these methods, the weight vector is not an afterthought computed from the prevalence estimate; it is the primitive quantity from which the estimate itself is derived.

ImportanceWeightQuantifier exposes this quantity directly, in addition to the usual predict method inherited from any quantifier:

• get_importance_weights(instances) returns the weight vector w estimated for a given batch of (unlabelled) target instances.
• quantify_and_weigh(instances) returns a tuple (prevalence, weights), computed from a single pass of classifier predictions over the instances (avoiding the double classification that calling predict and get_importance_weights separately would incur).

Neither method mutates any internal state of the quantifier, so both are safe to call concurrently, on the same fitted instance, for different batches of target instances.

import quapy as qp
from quapy.method.aggregative import BBSEhard
from sklearn.linear_model import LogisticRegression

train, test = qp.datasets.fetch_UCIBinaryDataset('haberman').train_test

model = BBSEhard(LogisticRegression(max_iter=2000), val_split=5)
model.fit(*train.Xy)

prevalence, weights = model.quantify_and_weigh(test.X)

Adapting a classifier: LabelShiftedClassifier

Rescaling the classifier’s posterior probabilities after the fact (as, e.g., EMQ does) is one way of accounting for an estimated prevalence shift. Another common practice is instead to retrain the classifier with a reweighted loss, via scikit-learn’s class_weight constructor parameter; this changes the decision function of the classifier itself, rather than only rescaling its outputs, which can matter for classifiers whose decision boundary is not a simple function of the posterior probabilities alone.

LabelShiftedClassifier automates this second strategy. Given a base classifier h and any quapy quantifier q, it fits q on the labelled training data (keeping a copy of it), and, for a given batch of (unlabelled) target instances, estimates the importance weights: directly, via get_importance_weights, if q is an ImportanceWeightQuantifier; or otherwise by dividing q’s estimated target prevalence by the training prevalence. It then retrains a fresh copy of h, with class_weight set according to these weights, on the original training data, and hands off classification of the target instances to this adapted classifier. Neither the classifier instance h passed at construction time, nor q’s own internal classifier (if any), are ever mutated.

import quapy as qp
from quapy.classification.labelshift import LabelShiftedClassifier
from quapy.method.aggregative import ACC
from sklearn.linear_model import LogisticRegression

train, test = qp.datasets.fetch_UCIBinaryDataset('haberman').train_test

h = LogisticRegression(max_iter=2000)
q = ACC(LogisticRegression(max_iter=2000), val_split=5)

adapter = LabelShiftedClassifier(h, q)
adapter.fit(*train.Xy)

# label predictions and posterior probabilities from a classifier adapted to test's estimated distribution
y_pred = adapter.predict(test.X)
y_proba = adapter.predict_proba(test.X)

# or, to obtain the adapted classifier itself
adapted_h = adapter.get_classifier(test.X)

h is required to accept a class_weight parameter in its constructor, as LogisticRegression and many other scikit-learn classifiers do; q can be any (fitted or unfitted) quapy quantifier, and need not be an ImportanceWeightQuantifier.

Since the adaptation is specific to the batch of target instances given, predict and predict_proba retrain h on every call. If you plan to issue several predictions against the same (suspected) shifted distribution, call get_classifier once and reuse the classifier instance it returns, rather than calling predict/predict_proba repeatedly on the same or related samples.

A small positive floor, controlled by the weight_epsilon constructor parameter (default 1e-4), is applied to the estimated weights before they are used as class_weight; this keeps a class with training support from being assigned a weight of exactly 0, which would silently drop it from the retraining loss.