Conformal prediction#

Conformal prediction methods are a type of calibration methods that, starting from uncertainty estimates, provide conformal sets, i.e. rigorous sets of predictions with a user-chosen level of probability. We support conformal methods for both classification and regression.

class fortuna.conformal.classification.adaptive_prediction.AdaptivePredictionConformalClassifier[source]#

conformal_set(val_probs, val_targets, test_probs, error)#

Return type:: List[List[int]]

get_scores(val_probs, val_targets, test_probs)#

Return type:: Tuple[Union[Array, ndarray], Union[Array, ndarray]]

is_in(values, conformal_sets)#

Check whether the values lie within their respective conformal sets.

Parameters:

values (Array) – Values to check if they lie in the respective conformal sets.
conformal_sets (Array) – A conformal set for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal sets.

Return type:

Array

score_fn(probs, targets)[source]#

class fortuna.conformal.classification.adaptive_prediction.CVPlusAdaptivePredictionConformalClassifier[source]#

conformal_set(cross_val_probs, cross_val_targets, cross_test_probs, error)#

Return type:: List[List[int]]

get_scores(cross_val_probs, cross_val_targets, cross_test_probs)#

Return type:: Tuple[Union[Array, ndarray], Union[Array, ndarray]]

is_in(values, conformal_sets)#

Check whether the values lie within their respective conformal sets.

Parameters:

values (Array) – Values to check if they lie in the respective conformal sets.
conformal_sets (Array) – A conformal set for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal sets.

Return type:

Array

score_fn(probs, targets)[source]#

fortuna.conformal.classification.adaptive_prediction.score_fn(probs, targets)[source]#

class fortuna.conformal.classification.simple_prediction.CVPlusSimplePredictionConformalClassifier[source]#

conformal_set(cross_val_probs, cross_val_targets, cross_test_probs, error)#

Return type:: List[List[int]]

get_scores(cross_val_probs, cross_val_targets, cross_test_probs)#

Return type:: Tuple[Union[Array, ndarray], Union[Array, ndarray]]

is_in(values, conformal_sets)#

Check whether the values lie within their respective conformal sets.

Parameters:

values (Array) – Values to check if they lie in the respective conformal sets.
conformal_sets (Array) – A conformal set for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal sets.

Return type:

Array

score_fn(probs, targets)[source]#

class fortuna.conformal.classification.simple_prediction.SimplePredictionConformalClassifier[source]#

conformal_set(val_probs, val_targets, test_probs, error)#

Return type:: List[List[int]]

get_scores(val_probs, val_targets, test_probs)#

Return type:: Tuple[Union[Array, ndarray], Union[Array, ndarray]]

is_in(values, conformal_sets)#

Check whether the values lie within their respective conformal sets.

Parameters:

values (Array) – Values to check if they lie in the respective conformal sets.
conformal_sets (Array) – A conformal set for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal sets.

Return type:

Array

score_fn(probs, targets)[source]#

fortuna.conformal.classification.simple_prediction.score_fn(probs, targets)[source]#

class fortuna.conformal.classification.adaptive_conformal_classifier.AdaptiveConformalClassifier(conformal_classifier)[source]#

An adaptive conformal classifier class (see Gibbs & Candes, 2021). It takes any conformal classifier and adds the functionality to update the coverage error to take into account distributional shifts in the data.

Parameters:: conformal_classifier (ConformalClassifier) – A conformal method for classification.

abstract get_scores(*args, **kwargs)#

Return type:: Tuple[Union[Array, ndarray], Union[Array, ndarray]]

is_in(values, conformal_sets)#

Check whether the values lie within their respective conformal sets.

Parameters:

values (Array) – Values to check if they lie in the respective conformal sets.
conformal_sets (Array) – A conformal set for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal sets.

Return type:

Array

abstract score_fn(probs, targets)#

update_error(conformal_set, error, target, target_error, gamma=0.005, weights=None, were_in=None, return_were_in=False)[source]#

Update the coverage error based on the test target variable belonging or not to the conformal set.

Parameters:

conformal_set (List[int]) – A conformal set for the current test target variable.
error (float) – The current coverage error to update.
target (Array) – The observed test target variable.
target_error (float) – The target coverage error.
gamma (float) – The step size for the coverage error update.
weights (Optional[Array]) – Weights over the considered past time steps and the current one. This must be a one-dimensional array of increasing components between 0 and 1, summing up to 1.
were_in (Optional[Array]) – It indicates whether the target variables of the considered past time steps fell within the respective conformal sets. This must be a one-dimensional array of 1’s and 0’s. Its length must be the length of weights minus one, as it refers to all the past time steps but not the current one.
return_were_in (bool) – It returns an updated were_in, which includes whether the current test target variable falls within its conformal set.

Returns:

The updated coverage error.

Return type:

Array

class fortuna.conformal.regression.quantile.QuantileConformalRegressor[source]#

conformal_interval(val_lower_bounds, val_upper_bounds, test_lower_bounds, test_upper_bounds, val_targets, error, quantile=None)[source]#

Coverage interval of each of the test inputs, at the desired coverage error. This is supported only for scalar target variables.

Parameters:

val_lower_bounds (Array) – Interval lower bounds computed on a validation set.
val_upper_bounds (Array) – Interval upper bounds computed on a validation set.
test_lower_bounds (Array) – Interval lower bounds computed on a test set.
test_upper_bounds (Array) – Interval upper bounds computed on a test set.
val_targets (Array) – A two-dimensional array of validation target variables.
error (float) – Coverage error. This must be a scalar between 0 and 1, extremes included. This should correspond to the coverage error for which val_lower_bounds, val_upper_bounds, test_lower_bounds and test_upper_bounds were computed.
quantile (Optional[float]) – Conformal quantiles. This should be the output of quantile().

Returns:

The conformal intervals. The two components of the second axis correspond to the left and right interval bounds.

Return type:

jnp.ndarray

is_in(values, conformal_intervals)#

Check whether the values lie within their respective conformal intervals.

Parameters:

values (Array) – Values to check if they lie in the respective conformal intervals.
conformal_intervals (Array) – A conformal interval for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal intervals.

Return type:

Array

quantile(val_lower_bounds, val_upper_bounds, val_targets, error, scores=None)[source]#

Compute a quantile of the scores.

Parameters:

val_lower_bounds (Array) – Interval lower bounds computed on a validation set.
val_upper_bounds (Array) – Interval upper bounds computed on a validation set.
val_targets (Array) – A two-dimensional array of validation target variables.
error (float) – Coverage error. This must be a scalar between 0 and 1, extremes included. This should correspond to the coverage error for which val_lower_bounds, val_upper_bounds, test_lower_bounds and test_upper_bounds were computed.
scores (Optional[float]) – Conformal scores. This should be the output of score().

Returns:

The conformal quantile.

Return type:

float

score(val_lower_bounds, val_upper_bounds, val_targets)[source]#

Compute score function.

Parameters:

val_lower_bounds (Array) – Interval lower bounds computed on a validation set.
val_upper_bounds (Array) – Interval upper bounds computed on a validation set.
val_targets (Array) – A two-dimensional array of validation target variables.

Returns:

The conformal scores.

Return type:

jnp.ndarray

class fortuna.conformal.regression.onedim_uncertainty.OneDimensionalUncertaintyConformalRegressor[source]#

conformal_interval(val_preds, val_uncertainties, test_preds, test_uncertainties, val_targets, error, quantile=None)[source]#

Coverage interval of each of the test inputs, at the desired coverage error. This is supported only for one-dimensional target variables.

Parameters:

val_preds (Array) – A two-dimensional array of predictions over the validation data points.
test_preds (Array) – A two-dimensional array of predictions over the test data points.
val_uncertainties (Array) – A two-dimensional array of uncertainty estimates (e.g. the standard deviation). The first dimension is over the validation inputs. The second must have only one component.
test_uncertainties (Array) – A two-dimensional array of uncertainty estimates (e.g. the standard deviation). The first dimension is over the test inputs. The second must have only one component.
val_targets (Array) – A two-dimensional array of validation target variables.
error (float) – Coverage error. This must be a scalar between 0 and 1, extremes included.
quantile (Optional[float]) – Conformal quantile. This should be the output of quantile().

Returns:

The conformal intervals. The two components of the second axis correspond to the left and right interval bounds.

Return type:

jnp.ndarray

is_in(values, conformal_intervals)#

Check whether the values lie within their respective conformal intervals.

Parameters:

values (Array) – Values to check if they lie in the respective conformal intervals.
conformal_intervals (Array) – A conformal interval for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal intervals.

Return type:

Array

quantile(val_preds, val_uncertainties, val_targets, error, scores=None)[source]#

Compute a quantile of the scores.

Parameters:

val_preds (Array) – A two-dimensional array of predictions over the validation data points.
val_uncertainties (Array) – A two-dimensional array of uncertainty estimates (e.g. the standard deviation). The first dimension is over the validation inputs. The second must have only one component.
val_targets (Array) – A two-dimensional array of validation target variables.
error (float) – Coverage error. This must be a scalar between 0 and 1, extremes included.
scores (Optional[float]) – Conformal scores. This should be the output of score().

Returns:

The conformal quantile.

Return type:

float

score(val_preds, val_uncertainties, val_targets)[source]#

Compute the conformal scores.

Parameters:

val_preds (Array) – A two-dimensional array of predictions over the validation data points.
val_uncertainties (Array) – A two-dimensional array of uncertainty estimates (e.g. the standard deviation). The first dimension is over the validation inputs. The second must have only one component.
val_targets (Array) – A two-dimensional array of validation target variables.

Returns:

Scores.

Return type:

jnp.ndarray

class fortuna.conformal.regression.cvplus.CVPlusConformalRegressor[source]#

This class implements the CV+ method introduced in Barber et al., 2021. It is an extension of the jackknife+ method, introduced in the same work, that consider a K-Fold instead of a leave-one-out strategy. If K=n, where n is the total number of training data, then CV+ reduces to jackknife+.

conformal_interval(cross_val_outputs, cross_val_targets, cross_test_outputs, error)[source]#

Coverage interval of each of the test inputs, at the desired coverage error. This is supported only for one-dimensional target variables.

Parameters:

cross_val_outputs (List[Array]) – Outputs of the models used during cross validation evaluated at their respective validation inputs. More precisely, we assume the training data has been jointly partitioned in K subsets. The i-th element of the list of :code: cross_val_outputs is a model trained on all data but the i-th partition, and has been evaluated at the inputs of the partition i-th itself, for i=1, 2, ..., K.
cross_val_targets (List[Array]) – Target variables organized in the same partitions used for cross_val_outputs. More precisely, the i-th element of cross_val_targets includes the array of target variables of the i-th partition of the training data, for i=1, 2, ..., K.
cross_test_outputs (List[Array]) – Outputs of the models used during cross validation evaluated at the test inputs. More precisely, consider the same partition of data as the one used for cross_val_outputs. Then the i-th element of cross_test_outputs represents the outputs of the model that has been trained upon all the training data but the i-th partition, and evaluated at the test inputs, for i=1, 2, ..., K.
error (float) – The desired coverage error. This must be a scalar between 0 and 1, extremes included.

Returns:

The conformal intervals. The two components of the second axis correspond to the left and right interval bounds.

Return type:

jnp.ndarray

is_in(values, conformal_intervals)#

Check whether the values lie within their respective conformal intervals.

Parameters:

values (Array) – Values to check if they lie in the respective conformal intervals.
conformal_intervals (Array) – A conformal interval for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal intervals.

Return type:

Array

class fortuna.conformal.regression.jackknifeplus.JackknifePlusConformalRegressor[source]#

This class implements the jackknife+ method introduced in Barber et al., 2021. Given a desired coverage of \(1-α\), jackknife+ guarantees a minimal coverage of \(1 - 2α\).

conformal_interval(loo_val_outputs, loo_val_targets, loo_test_outputs, error)[source]#

Coverage interval of each of the test inputs, at the desired coverage error. This is supported only for one-dimensional target variables.

Parameters:

loo_val_outputs (Array) – Outputs of the models used during leave-out-out evaluated at their respective left-out validation inputs. More precisely, the i-th element of :code: loo_val_outputs is a model that has been trained upon all the training data but the i-th data point, and evaluated at the i-th input, for all training data points.
loo_val_targets (Array) – The array of target variables of the left-out data points in the leave-one-out procedure described for loo_val_outputs.
loo_test_outputs (Array) – Outputs of the models used during leave-one-out evaluated at the test inputs. More precisely, consider the same leave-one-out procedure as the one used for loo_val_outputs. Then the i-th element of the first dimension of loo_test_outputs represents the outputs of the model that has been trained upon all the training data but the i-th data point, and evaluated at the test inputs. The second dimension of loo_test_outputs is over the different test inputs.
error (float) – The desired coverage error. This must be a scalar between 0 and 1, extremes included.

Returns:

The conformal intervals. The two components of the second axis correspond to the left and right interval bounds.

Return type:

jnp.ndarray

is_in(values, conformal_intervals)#

Check whether the values lie within their respective conformal intervals.

Parameters:

values (Array) – Values to check if they lie in the respective conformal intervals.
conformal_intervals (Array) – A conformal interval for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal intervals.

Return type:

Array

class fortuna.conformal.regression.jackknife_minmax.JackknifeMinmaxConformalRegressor[source]#

This class implements the jackknife-minmax method introduced in Barber et al., 2021. Jackknife-minmax guarantees the desired minimal coverage \(1-α\), while jackknife+ only guarantees a minimal coverage of \(1 - 2α\).

conformal_interval(loo_val_outputs, loo_val_targets, loo_test_outputs, error)[source]#

Coverage interval of each of the test inputs, at the desired coverage error. This is supported only for one-dimensional target variables.

Parameters:

loo_val_outputs (Array) – Outputs of the models used during leave-out-out evaluated at their respective left-out validation inputs. More precisely, the i-th element of :code: loo_val_outputs is a model that has been trained upon all the training data but the i-th data point, and evaluated at the i-th input, for all training data points.
loo_val_targets (Array) – The array of target variables of the left-out data points in the leave-one-out procedure described for loo_val_outputs.
loo_test_outputs (Array) – Outputs of the models used during leave-one-out evaluated at the test inputs. More precisely, consider the same leave-one-out procedure as the one used for loo_val_outputs. Then the i-th element of the first dimension of loo_test_outputs represents the outputs of the model that has been trained upon all the training data but the i-th data point, and evaluated at the test inputs. The second dimension of loo_test_outputs is over the different test inputs.
error (float) – The desired coverage error. This must be a scalar between 0 and 1, extremes included.

Returns:

The conformal intervals. The two components of the second axis correspond to the left and right interval bounds.

Return type:

jnp.ndarray

is_in(values, conformal_intervals)#

Check whether the values lie within their respective conformal intervals.

Parameters:

values (Array) – Values to check if they lie in the respective conformal intervals.
conformal_intervals (Array) – A conformal interval for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal intervals.

Return type:

Array

class fortuna.conformal.regression.enbpi.EnbPI(aggregation_fun=lambda x: ...)[source]#

Ensemble Batch Prediction Intervals (EnbPI) is a conformal prediction algorithm for time series regression. By bootstrapping the data and training the model on each of the bootstrap samples, EnbPI is able to compute conformal intervals that satisfy an approximate marginal guarantee on each test data point. Furthermore, EnbPI can incorporate online feedback from incoming batches of data, and improve the conformal intervals without having to retrain the model.

Parameters:: aggregation_fun (Callable[[Array], Array]) – Aggregation function that takes a group of model predictions from different bootstrap samples and reduces them to a single one.

conformal_interval(bootstrap_indices, bootstrap_train_preds, bootstrap_test_preds, train_targets, error, return_residuals=False)[source]#

Parameters:

bootstrap_indices (Array) – The indices, randomly sampled with replacement, of the training data in the time series used to train the model. The first dimension is over the different samples of indices. The second dimension contain the data points for each sample, which must be as many as thnumber of data points in the time series used for training. A simple way of obtaining indices randomly sampled with replacement is numpy.random.choice(T, size=(B, T)), where T is the number of training points in the time series, and B is the number of bootstrap samples. It is the user job to make sure that the models are trained upon the data corresponding to the random indices.
bootstrap_train_preds (Array) – Model predictions for each of the bootstrap samples of data of the time series used for training the model, evaluated at each of the training data inputs of the time series. The first dimension is over the different bootstrap samples. The second dimensions is over the training inputs. There may be a third dimension, corresponding to the dimensionality of the predictions, but if so this must be one.
bootstrap_test_preds (Array) – Model predictions for each of the bootstrap samples of data of the time series used for training the model, evaluated at each of the test data inputs of the time series. The first dimension is over the different bootstrap samples. The second dimensions is over the test inputs. There may be a third dimension, corresponding to the dimensionality of the predictions, but if so this must be one.
train_targets (Array) – The target variables of the training data points in the time series.
error (float) – The desired coverage error. This must be a scalar between 0 and 1, extremes included.
return_residuals (bool) – If True, return the residual errors computed over the training data. These are used in conformal_interval_from_residuals().

Returns:

The conformal intervals. The two components of the second dimension correspond to the left and right interval bounds. If return_residuals is set to True, then it returns also the residuals computed on the training set.

Return type:

Union[Array, Tuple[Array, Array]]

conformal_interval_from_residuals(train_residuals, bootstrap_new_train_preds, bootstrap_new_test_preds, new_train_targets, error)[source]#

Compute a coverage interval for each of the test inputs of the time series, at the desired coverage error. This is supported only for one-dimensional target variables, and for one times series at the time. This method assumes the residuals over the training sets have already been computed via conformal_interval() using the flag return_residuals=True. If so, this method takes in the predictions for each bootstrap samples over the new incoming training and test data points, and computes conformal intervals without having to retrain the model. Compared to predicting conformal intervals over all test data points at once with conformal_interval(), this method exploits incoming data points as a form of online feedback, and improves the conformal intervals accordingly.

Parameters:

train_residuals (Array) – Residuals over the training data points computed via conformal_interval() using the flag return_residuals=True.
bootstrap_new_train_preds (Array) – Model predictions for each of the bootstrap samples of data of the time series used for training the model, evaluated at the new incoming training data inputs of the time series. The first dimension is over the different bootstrap samples. The second dimensions is over the new training inputs. There may be a third dimension, corresponding to the dimensionality of the predictions, but if so this must be one. As an example, suppose that the bootstrap has been done over data points from 1 to T, and the model was trained over each bootstrap sample. A prediction was then made on a batch of data points from T+1 to T+T1. After the data from T+1 to T+T1 is observed, this will be used as the new batch of training data.
bootstrap_new_test_preds (Array) – Model predictions for each of the bootstrap samples of data of the time series used for training the model, evaluated at the new incoming test data inputs of the time series. The first dimension is over the different bootstrap samples. The second dimensions is over the new test inputs. There may be a third dimension, corresponding to the dimensionality of the predictions, but if so this must be one. As an example, suppose that the bootstrap has been done over data points from 1 to T, and the model was trained over each bootstrap sample. A prediction was then made on a batch of data points from T+1 to T+T1. After the data from T+1 to T+T1 is observed, data from T+T1+1 to T+2*T1 may be taken as the new batch of test data.
new_train_targets (Array) – The target variables of the new batch of training data points in the time series.
error (float) – The desired coverage error. This must be a scalar between 0 and 1, extremes included.

Returns:

The conformal intervals. The two components of the second dimension correspond to the left and right

interval bounds. - A new set of residuals, which includes the residuals computed on the new batch of training data points. The number of stored training residuals is kept constant by eliminating the oldest ones.

Return type:

Union[Array, Tuple[Array, Array]]

is_in(values, conformal_intervals)#

Check whether the values lie within their respective conformal intervals.

Parameters:

values (Array) – Values to check if they lie in the respective conformal intervals.
conformal_intervals (Array) – A conformal interval for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal intervals.

Return type:

Array

class fortuna.conformal.regression.adaptive_conformal_regressor.AdaptiveConformalRegressor(conformal_regressor)[source]#

An adaptive conformal regressor class (see Gibbs & Candes, 2021). It takes any conformal regressor and adds the functionality to update the coverage error to take into account distributional shifts in the data.

Parameters:: conformal_regressor (ConformalRegressor) – A conformal method for regression.

update_error(conformal_interval, error, target, target_error, gamma=0.005, weights=None, were_in=None, return_were_in=False)[source]#

Update the coverage error based on the test target variable belonging or not to the conformal interval.

Parameters:

conformal_interval (List[int]) – A conformal interval for the current test target variable.
error (float) – The current coverage error to update.
target (Array) – The observed test target variable.
target_error (float) – The target coverage error.
gamma (float) – The step size for the coverage error update.
weights (Optional[Array]) – Weights over the considered past time steps and the current one. This must be a one-dimensional array of increasing components between 0 and 1, summing up to 1.
were_in (Optional[Array]) – It indicates whether the target variables of the considered past time steps fell within the respective conformal intervals. This must be a one-dimensional array of 1’s and 0’s. Its length must be the length of weights minus one, as it refers to all the past time steps but not the current one.
return_were_in (bool) – It returns an updated were_in, which includes whether the current test target variable falls within its conformal interval.

Returns:

The updated coverage error.

Return type:

Array

class fortuna.conformal.regression.batch_mvp.BatchMVPConformalRegressor(seed=0)[source]#

This class implements a classification version of BatchMVP [Jung et al., 2022], a multivalid conformal prediction method that satisfies coverage guarantees conditioned on group membership and non-conformity threshold.

Parameters:: seed (int) – Random seed.

apply_patches(groups=None, thresholds=None)#

Apply the patches to the model evaluations.

Parameters:

groups (Array) – A list of groups \(g(x)\) evaluated over some inputs. This should be a two-dimensional array of bool elements. The first dimension is over the data points, the second dimension is over the number of groups.
values (Optional[Array]) – The initial model evaluations \(f(x)\) evaluated over some inputs. If not provided, these are set to 0.

Returns:

The calibrated values.

Return type:

Array

property bucket_types: Tuple[str]#

calibrate(scores, groups=None, thresholds=None, test_groups=None, test_thresholds=None, atol=1e-4, rtol=1e-6, min_prob_b='auto', n_buckets=100, n_rounds=1000, eta=0.1, split=0.8, bucket_types=('>=', '<='), patch_type='additive', coverage=0.95)#

Calibrate the model by finding a list of patches to the model that bring the calibration error below a certain threshold.

Parameters:

scores (Array) – A list of scores \(s(x, y)\) computed on the calibration data. This should be a one-dimensional array of elements between 0 and 1.
groups (Array) – A list of groups \(g(x)\) computed on the calibration data. This should be a two-dimensional array of bool elements. The first dimension is over the data points, the second dimension is over the number of groups.
thresholds (Optional[Array]) – The initial model evalutions \(f(x)\) on the calibration data. If not provided, these are set to 0.
test_groups (Optional[Array]) – A list of groups \(g(x)\) computed on the test data. This should be a two-dimensional array of bool elements. The first dimension is over the data points, the second dimension is over the number of groups.
test_thresholds (Optional[Array]) – The initial model evaluations \(f(x)\) on the test data. If not provided, these are set to 0.
atol (float) – Absolute tolerance on the mean squared error.
rtol (float) – Relative tolerance on the mean squared error.
min_prob_b (float) – Minimum probability of the conditioning set \(B_t\) for the patch to be applied.
n_buckets (int) – The number of buckets used in the algorithm. The smaller the number of buckets, the simpler the model, the better its generalization abilities. If not provided, We start from 2 buckets, and progressively double the number of buckets until we find a value for which the calibration error falls below the given tolerance. Such number of buckets is guaranteed to exist.
n_rounds (int) – The maximum number of rounds to run the method for.
eta (float) – Step size. By default, this is set to 1.
split (float) – Split the calibration data into calibration and validation, according to the given proportion. The validation data will be used for early stopping.
bucket_types (Tuple[str, ...]) –
Types of buckets. The following types are currently supported:
- ”=”, corresponding of buckets like \(\{f(x) = v\}\);
- ”>=”, corresponding of buckets like \(\{f(x) \ge v\}\);
- ”<=”, corresponding of buckets like \(\{f(x) \le v\}\).
coverage (float) – The desired level of coverage. This must be a scalar between 0 and 1.

Returns:

A status including the number of rounds taken to reach convergence and the calibration errors computed during the training procedure. if test_thresholds and test_groups are provided, the list of patches will be applied to test_thresholds, and the calibrated test thresholds will be returned together with the status.

Return type:

Union[Dict, Tuple[Array, Dict]]

calibration_error(scores, groups=None, thresholds=None, n_buckets=10, **kwargs)#

The reweighted average squared calibration error \(\mu(g) K_2(f, g, \mathcal{D})\).

Parameters:

scores (Array) – A score for each data point.
groups (Array) – A list of groups \(g(x)\) evaluated over some inputs. This should be a two-dimensional array of bool elements. The first dimension is over the data points, the second dimension is over the number of groups.
values (Array) – The model evaluations, before or after calibration.
n_buckets (int) – The number of buckets used in the algorithm.

Returns:

The computed calibration error for each group

Return type:

Array

property eta#

abstract get_scores(*args, **kwargs)#

Return type:: Tuple[Union[Array, ndarray], Union[Array, ndarray]]

is_in(values, conformal_sets)#

Check whether the values lie within their respective conformal sets.

Parameters:

values (Array) – Values to check if they lie in the respective conformal sets.
conformal_sets (Array) – A conformal set for each input data point.

Returns:

An array of ones or zero, indicating whether the values lie within their respective conformal sets.

Return type:

Array

property n_buckets#

property patch_type#

property patches#

pinball_loss(values, scores, coverage)#

The pinball loss between the model evaluations and the scores.

Parameters:

values (Array) – The model evaluations.
scores (Array) – The scores.
coverage (float) – The target coverage.

Returns:

The pinball loss evaluation.

Return type:

Array

abstract score_fn(probs, targets)#