metrics#

This module provides different metrics to evaluate the performance of a model.

class Metric(*args, **kwargs)#

Bases: Protocol

Protocol for metrics.

mean_absolute_error(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean absolute error.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean absolute error.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

Example

>>> import numpy as np
>>> from simba_ml.prediction.time_series.metrics import metrics
>>> y_true = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> y_pred = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> metrics.mean_absolute_error(y_true, y_pred)
0.0

>>> y_true = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> y_pred = np.array([[[2, 4, 6], [8, 10, 12]]])
>>> metrics.mean_absolute_error(y_true, y_pred)
3.5
mean_absolute_error_matrix(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → ndarray[Any, dtype[float64]]#

Calculates the mean absolute error matrix.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean absolute error score for each timestamp and attribute as a 2D matrix.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

mean_absolute_percentage_error(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean absolute error.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean absolute error.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

Example

>>> import numpy as np
>>> from simba_ml.prediction.time_series.metrics import metrics
>>> y_true = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> y_pred = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> metrics.mean_absolute_percentage_error(y_true, y_pred)
0.0

>>> y_true = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> y_pred = np.array([[[2, 4, 6], [8, 10, 12]]])
>>> metrics.mean_absolute_percentage_error(y_true, y_pred)
1.0
mean_absolute_percentage_error_matrix(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → ndarray[Any, dtype[float64]]#

Calculates the mean absolute percentage error matrix.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean absolute percentage error score for each timestamp

and attribute as a 2D matrix.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

mean_absolute_scaled_error(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean absolute scaled error.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The the mean absolute scaled error.

mean_average_precision(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]], treshold: float) → float64#

Calculates the mean average precision with a given treshold.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

  • treshold – The maximal absolute error, which is considered correct.

Returns:

The the mean average precision.

mean_average_precision_0_1(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean average precision with treshold=0.1.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The the mean average precision.

mean_average_precision_1(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean average precision with treshold=1.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The the mean average precision.

mean_average_precision_10(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean average precision with treshold=10.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The the mean average precision.

mean_directional_accuracy(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean directional accuracy.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean directional accuracy.

mean_squared_error(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean absolute error.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean absolute error.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

Example

>>> import numpy as np
>>> from simba_ml.prediction.time_series.metrics import metrics
>>> y_true = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> y_pred = np.array([[[1, 2, 3], [4, 5, 6]]])
>>> metrics.mean_squared_error(y_true, y_pred)
0.0

>>> y_true = np.array([[[1, 2], [3, 5]]])
>>> y_pred = np.array([[[2, 4], [6, 10]]])
>>> metrics.mean_squared_error(y_true, y_pred)
9.75
mean_squared_error_matrix(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → ndarray[Any, dtype[float64]]#

Calculates the mean square error matrix.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean square error score for each timestamp and attribute as a 2D matrix.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

msle(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the mean squared log error.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The mean squared log error.

normalized_root_mean_squared_error(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the normalized root mean squared error using the mean of y_true.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The normalized root mean squared error.

r_square(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the r2 score.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The average r2 score for each series.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

r_square_matrix(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → ndarray[Any, dtype[float64]]#

Calculates the r2 score as 2D matrix.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The r2 score for each timestamp and attribute as a 2D matrix.

Raises:

  • ValueError – If the shape of y_true and y_pred is not the same.

  • ValueError – If the shape of y_true and y_pred is not lower than 2D.

rmsle(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the root mean squared log error.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The root mean squared log error.

root_mean_squared_error(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → float64#

Calculates the root mean squared error.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Returns:

The root mean squared error.

test_input(y_true: ndarray[Any, dtype[float64]], y_pred: ndarray[Any, dtype[float64]]) → None#

Test that y_true and y_pred have both the same 3D shape.

Parameters:

  • y_true – The ground truth labels.

  • y_pred – The predicted labels.

Raises:

  • ValueError – If the dimension of y_true and y_pred are not 3.

  • ValueError – If the shape of y_true and y_pred are not the same.