Metric

We support some metrics for both classification and regression. Metrics are NumPy-compatible, therefore feel free to bring your own and apply them on Fortuna’s predictions.

fortuna.metric.classification.accuracy(preds, targets)[source]

Compute the accuracy given predictions and target variables.

Parameters:

  • preds (Array) – A one-dimensional array of predictions over the data points.

  • targets (Array) – A one-dimensional array of target variables.

Returns:

The computed accuracy.

Return type:

jnp.ndarray

fortuna.metric.classification.brier_score(probs, targets)[source]

Brier score (see Brier, 1950). This can be used for both binary and multi-class classification.

Parameters:

  • probs (Array) – A one- or two-dimensional array of class probabilities for each data point.

  • targets (Array) – A one-dimensional array of target variables.

Returns:

The Brier score.

Return type:

jnp.ndarray

fortuna.metric.classification.ece(preds, probs, targets, plot=False, plot_options=None)[source]

See expected_calibration_error().

Return type:

Array

fortuna.metric.classification.expected_calibration_error(preds, probs, targets, plot=False, plot_options=None)[source]

Compute the Expected Calibration Error (ECE) (see Naeini et al., 2015 and Guo et al., 2017). Optionally, plot and save a reliability diagram.

Parameters:

  • preds (Array) – A one-dimensional array of predictions over the data points.

  • probs (Array) – A two-dimensional array of class probabilities for each data point.

  • targets (Array) – A one-dimensional array of target variables.

  • plot (bool) – Whether to plot a reliability diagram.

  • plot_options (dict) – Options for the reliability diagram plot; see plot_reliability_diagram().

Returns:

The value of the ECE.

Return type:

jnp.ndarray

fortuna.metric.classification.maximum_calibration_error(preds, probs, targets, plot=False, plot_options=None)[source]

Compute the Maximum Calibration Error (MCE) (see Naeini et al., 2015). Optionally, plot and save a reliability diagram.

Parameters:

  • preds (Array) – A one-dimensional array of predictions over the data points.

  • probs (Array) – A two-dimensional array of class probabilities for each data point.

  • targets (Array) – A one-dimensional array of target variables.

  • plot (bool) – Whether to plot a reliability diagram.

  • plot_options (dict) – Options for the reliability diagram plot; see plot_reliability_diagram().

Returns:

The value of the MCE.

Return type:

jnp.ndarray

fortuna.metric.classification.mce(preds, probs, targets, plot=False, plot_options=None)[source]

See maximum_calibration_error().

Return type:

Array

fortuna.metric.regression.mae(preds, targets)[source]

See mean_absolute_error().

Return type:

float

fortuna.metric.regression.mean_absolute_error(preds, targets)[source]

Compute the mean-absolute error (MAE).

Parameters:

  • preds (Array) – A two-dimensional array of predictions over the data points.

  • targets (Array) – A two-dimensional array of target variables.

Returns:

The computed MAE.

Return type:

Array

fortuna.metric.regression.mean_squared_error(preds, targets)[source]

Compute the mean-squared error (MSE).

Parameters:

  • preds (Array) – A two-dimensional array of predictions over the data points.

  • targets (Array) – A two-dimensional array of target variables.

Returns:

The computed MSE.

Return type:

Array

fortuna.metric.regression.mse(preds, targets)[source]

See mean_squared_error().

Return type:

float

fortuna.metric.regression.picp(lower_bounds, upper_bounds, targets)[source]

See prediction_interval_coverage_probability().

Return type:

float

fortuna.metric.regression.prediction_interval_coverage_probability(lower_bounds, upper_bounds, targets)[source]

Compute the prediction interval coverage probability (PICP). This is the fraction of data points for which the true targets lie within the estimated interval. This is supported only for scalar target data.

Parameters:

  • lower_bounds (Array) – Predictive lower bounds. These are the lower bounds of the estimated predictive intervals. This can either be a one-dimensional array with entry corresponding to different data points, or a two-dimensional array with first axis corresponding to different data points, and second axis with only one dimension.

  • upper_bounds (Array) – Predictive upper bounds. These are the upper bounds of the estimated predictive intervals. This can either be a one-dimensional array with entry corresponding to different data points, or a two-dimensional array with first axis corresponding to different data points, and second axis with only one dimension.

  • targets (Array) – A two-dimensional array of target variables, or a one-dimensional array with second dimension of size 1.

Returns:

The computed PICP.

Return type:

Array

fortuna.metric.regression.rmae(preds, targets)[source]

See root_mean_absolute_error().

Return type:

float

fortuna.metric.regression.rmse(preds, targets)[source]

See root_mean_squared_error().

Return type:

float

fortuna.metric.regression.root_mean_absolute_error(preds, targets)[source]

Compute the root-mean-absolute error (RMAE).

Parameters:

  • preds (Array) – A two-dimensional array of predictions over the data points.

  • targets (Array) – A two-dimensional array of target variables.

Returns:

The computed RMAE.

Return type:

float

fortuna.metric.regression.root_mean_squared_error(preds, targets)[source]

Compute the root-mean-squared error (RMSE).

Parameters:

  • preds (Array) – A two-dimensional array of predictions over the data points.

  • targets (Array) – A two-dimensional array of target variables.

Returns:

The computed RMSE.

Return type:

float