turicreate.evaluation.roc_curve¶

turicreate.evaluation.
roc_curve
(targets, predictions, average=None, index_map=None)¶ Compute an ROC curve for the given targets and predictions. Currently, only binary classification is supported.
Parameters:  targets : SArray
An SArray containing the observed values. For binary classification, the alphanumerically first category is considered the reference category.
 predictions : SArray
The prediction that corresponds to each target value. This vector must have the same length as
targets
. Target scores, can either be probability estimates of the positive class, confidence values, or binary decisions. average : string, [None (default)]
Metric averaging strategies for multiclass classification. Averaging strategies can be one of the following:
 None: No averaging is performed and a single metric is returned for each class.
 index_map : dict[int], [None (default)]
For binary classification, a dictionary mapping the two target labels to either 0 (negative) or 1 (positive). For multiclass classification, a dictionary mapping potential target labels to the associated index into the vectors in
predictions
.
Returns:  out : SFrame
Each row represents the predictive performance when using a given cutoff threshold, where all predictions above that cutoff are considered “positive”. Four columns are used to describe the performance:
 tpr : True positive rate, the number of true positives divided by the number of positives.
 fpr : False positive rate, the number of false positives divided by the number of negatives.
 p : Total number of positive values.
 n : Total number of negative values.
 class : Reference class for this ROC curve.
See also
Notes
 For binary classification, when the target label is of type “string”,
then the labels are sorted alphanumerically and the largest label is
chosen as the “positive” label. For example, if the classifier labels
are {“cat”, “dog”}, then “dog” is chosen as the positive label for the
binary classification case. This behavior can be overridden by providing
an explicit
index_map
.  For multiclass classification, when the target label is of type
“string”, then the probability vector is assumed to be a vector of
probabilities of classes as sorted alphanumerically. Hence, for the
probability vector [0.1, 0.2, 0.7] for a dataset with classes “cat”,
“dog”, and “rat”; the 0.1 corresponds to “cat”, the 0.2 to “dog” and the
0.7 to “rat”. This behavior can be overridden by providing an explicit
index_map
.  The ROC curve is computed using a binning approximation with 1M bins and is hence accurate only to the 5th decimal.
References
An introduction to ROC analysis. Tom Fawcett.
Examples
>>> targets = turicreate.SArray([0, 1, 1, 0]) >>> predictions = turicreate.SArray([0.1, 0.35, 0.7, 0.99]) # Calculate the roccurve. >>> roc_curve = turicreate.evaluation.roc_curve(targets, predictions) ++++++  threshold  fpr  tpr  p  n  ++++++  0.0  1.0  1.0  2  2   9.99999974738e06  1.0  1.0  2  2   1.99999994948e05  1.0  1.0  2  2   2.99999992421e05  1.0  1.0  2  2   3.99999989895e05  1.0  1.0  2  2   4.99999987369e05  1.0  1.0  2  2   5.99999984843e05  1.0  1.0  2  2   7.00000018696e05  1.0  1.0  2  2   7.9999997979e05  1.0  1.0  2  2   9.00000013644e05  1.0  1.0  2  2  ++++++ [100001 rows x 5 columns]
For the multiclass setting, an ROC curve is returned for each class.
# Targets and Predictions >>> targets = turicreate.SArray([0, 1, 2, 3, 0, 1, 2, 3]) >>> predictions = turicreate.SArray([1, 0, 2, 1, 3, 1, 2, 1]) # Micro average of the recall scores for each class. >>> turicreate.evaluation.recall(targets, predictions, ... average = 'micro') 0.375 # Macro average of the recall scores for each class. >>> turicreate.evaluation.recall(targets, predictions, ... average = 'macro') 0.375 # Recall score for each class. >>> turicreate.evaluation.recall(targets, predictions, ... average = None) {0: 0.0, 1: 0.5, 2: 1.0, 3: 0.0}
This metric also works in the multiclass setting.
# Targets and Predictions >>> targets = turicreate.SArray([ 1, 0, 2, 1]) >>> predictions = turicreate.SArray([[.1, .8, 0.1], ... [.9, .1, 0.0], ... [.8, .1, 0.1], ... [.3, .6, 0.1]]) # Compute the ROC curve. >>> roc_curve = turicreate.evaluation.roc_curve(targets, predictions) +++++++  threshold  fpr  tpr  p  n  class  +++++++  0.0  1.0  1.0  1  3  0   1e05  1.0  1.0  1  3  0   2e05  1.0  1.0  1  3  0   3e05  1.0  1.0  1  3  0   4e05  1.0  1.0  1  3  0   5e05  1.0  1.0  1  3  0   6e05  1.0  1.0  1  3  0   7e05  1.0  1.0  1  3  0   8e05  1.0  1.0  1  3  0   9e05  1.0  1.0  1  3  0  +++++++ [300003 rows x 6 columns]
This metric also works for string classes.
# Targets and Predictions >>> targets = turicreate.SArray(["cat", "dog", "foosa", "dog"]) >>> predictions = turicreate.SArray([[.1, .8, 0.1], ... [.9, .1, 0.0], ... [.8, .1, 0.1], ... [.3, .6, 0.1]]) # Compute the ROC curve. >>> roc_curve = turicreate.evaluation.roc_curve(targets, predictions) +++++++  threshold  fpr  tpr  p  n  class  +++++++  0.0  1.0  1.0  1  3  cat   1e05  1.0  1.0  1  3  cat   2e05  1.0  1.0  1  3  cat   3e05  1.0  1.0  1  3  cat   4e05  1.0  1.0  1  3  cat   5e05  1.0  1.0  1  3  cat   6e05  1.0  1.0  1  3  cat   7e05  1.0  1.0  1  3  cat   8e05  1.0  1.0  1  3  cat   9e05  1.0  1.0  1  3  cat  +++++++ [300003 rows x 6 columns]