109 lines
5.7 KiB
Markdown
109 lines
5.7 KiB
Markdown
# Label Shift Adaptation
|
|
|
|
Quantification methods estimate the class prevalence values of a test sample, but a prevalence estimate is
|
|
not, by itself, a classifier that has been corrected for the shift it describes. In some scenarios you may
|
|
want the latter: a classifier whose decisions account for the (estimated) change in class priors between
|
|
the training and the test distributions, rather than a single prevalence vector.
|
|
|
|
QuaPy provides two building blocks for this:
|
|
|
|
* `qp.method.aggregative.ImportanceWeightQuantifier`, an interface implemented by the quantifiers that
|
|
natively compute a vector of importance weights as part of estimating the target prevalence.
|
|
* `qp.classification.labelshift.LabelShiftedClassifier`, a classifier wrapper that uses any quantifier
|
|
(whether or not it implements the interface above) to adapt a classifier to a specific, (potentially)
|
|
shifted batch of test instances.
|
|
|
|
## The `ImportanceWeightQuantifier` interface
|
|
|
|
Some quantifiers (currently, `RLLS`, `BBSEhard`, and `BBSEsoft`; see the
|
|
{ref}`Regularized Learning under Label Shift (RLLS) <manuals/methods:Regularized Learning under Label Shift (RLLS)>`
|
|
and {ref}`Black Box Shift Estimation (BBSE) <manuals/methods:Black Box Shift Estimation (BBSE)>` sections of
|
|
the methods manual) work by first estimating a vector of importance weights
|
|
|
|
:::{math}
|
|
w_y = \frac{Q(y)}{P(y)}
|
|
:::
|
|
|
|
with :math:`P` and :math:`Q` the training and target class distributions, and only then rescaling the
|
|
training prevalence by these weights to obtain the target prevalence estimate:
|
|
:math:`\hat{Q}(y) \propto w_y \cdot P(y)`. For these methods, the weight vector is not an afterthought
|
|
computed from the prevalence estimate; it is the primitive quantity from which the estimate itself is
|
|
derived.
|
|
|
|
`ImportanceWeightQuantifier` exposes this quantity directly, in addition to the usual `predict` method
|
|
inherited from any quantifier:
|
|
|
|
* `get_importance_weights(instances)` returns the weight vector $w$ estimated for a given batch of
|
|
(unlabelled) target instances.
|
|
* `quantify_and_weigh(instances)` returns a tuple `(prevalence, weights)`, computed from a single pass of
|
|
classifier predictions over the instances (avoiding the double classification that calling `predict` and
|
|
`get_importance_weights` separately would incur).
|
|
|
|
Neither method mutates any internal state of the quantifier, so both are safe to call concurrently, on the
|
|
same fitted instance, for different batches of target instances.
|
|
|
|
```python
|
|
import quapy as qp
|
|
from quapy.method.aggregative import BBSEhard
|
|
from sklearn.linear_model import LogisticRegression
|
|
|
|
train, test = qp.datasets.fetch_UCIBinaryDataset('haberman').train_test
|
|
|
|
model = BBSEhard(LogisticRegression(max_iter=2000), val_split=5)
|
|
model.fit(*train.Xy)
|
|
|
|
prevalence, weights = model.quantify_and_weigh(test.X)
|
|
```
|
|
|
|
## Adapting a classifier: `LabelShiftedClassifier`
|
|
|
|
Rescaling the classifier's posterior probabilities after the fact (as, e.g., `EMQ` does) is one way of
|
|
accounting for an estimated prevalence shift. Another common practice is instead to retrain the classifier
|
|
with a reweighted loss, via scikit-learn's `class_weight` constructor parameter; this changes the decision
|
|
function of the classifier itself, rather than only rescaling its outputs, which can matter for classifiers
|
|
whose decision boundary is not a simple function of the posterior probabilities alone.
|
|
|
|
`LabelShiftedClassifier` automates this second strategy. Given a base classifier `h` and any `quapy`
|
|
quantifier `q`, it fits `q` on the labelled training data (keeping a copy of it), and, for a given batch of
|
|
(unlabelled) target instances, estimates the importance weights: directly, via `get_importance_weights`, if
|
|
`q` is an `ImportanceWeightQuantifier`; or otherwise by dividing `q`'s estimated target prevalence by the
|
|
training prevalence. It then retrains a *fresh copy* of `h`, with `class_weight` set according to these
|
|
weights, on the original training data, and hands off classification of the target instances to this
|
|
adapted classifier. Neither the classifier instance `h` passed at construction time, nor `q`'s own internal
|
|
classifier (if any), are ever mutated.
|
|
|
|
```python
|
|
import quapy as qp
|
|
from quapy.classification.labelshift import LabelShiftedClassifier
|
|
from quapy.method.aggregative import ACC
|
|
from sklearn.linear_model import LogisticRegression
|
|
|
|
train, test = qp.datasets.fetch_UCIBinaryDataset('haberman').train_test
|
|
|
|
h = LogisticRegression(max_iter=2000)
|
|
q = ACC(LogisticRegression(max_iter=2000), val_split=5)
|
|
|
|
adapter = LabelShiftedClassifier(h, q)
|
|
adapter.fit(*train.Xy)
|
|
|
|
# label predictions and posterior probabilities from a classifier adapted to test's estimated distribution
|
|
y_pred = adapter.predict(test.X)
|
|
y_proba = adapter.predict_proba(test.X)
|
|
|
|
# or, to obtain the adapted classifier itself
|
|
adapted_h = adapter.get_classifier(test.X)
|
|
```
|
|
|
|
`h` is required to accept a `class_weight` parameter in its constructor, as `LogisticRegression` and many
|
|
other scikit-learn classifiers do; `q` can be any (fitted or unfitted) `quapy` quantifier, and need not be an
|
|
`ImportanceWeightQuantifier`.
|
|
|
|
Since the adaptation is specific to the batch of target instances given, `predict` and `predict_proba`
|
|
retrain `h` on every call. If you plan to issue several predictions against the same (suspected) shifted
|
|
distribution, call `get_classifier` once and reuse the classifier instance it returns, rather than calling
|
|
`predict`/`predict_proba` repeatedly on the same or related samples.
|
|
|
|
A small positive floor, controlled by the `weight_epsilon` constructor parameter (default `1e-4`), is
|
|
applied to the estimated weights before they are used as `class_weight`; this keeps a class with training
|
|
support from being assigned a weight of exactly `0`, which would silently drop it from the retraining loss.
|