turicreate.recommender.util.precision_recall_by_user

turicreate.recommender.util.precision_recall_by_user(observed_user_items, recommendations, cutoffs=[10])

Compute precision and recall at a given cutoff for each user. In information retrieval terms, precision represents the ratio of relevant, retrieved items to the number of relevant items. Recall represents the ratio of relevant, retrieved items to the number of relevant items.

Let \(p_k\) be a vector of the first \(k\) elements in the recommendations for a particular user, and let \(a\) be the set of items in observed_user_items for that user. The “precision at cutoff k” for this user is defined as

\[P(k) = \frac{ | a \cap p_k | }{k},\]

while “recall at cutoff k” is defined as

\[R(k) = \frac{ | a \cap p_k | }{|a|}\]

The order of the elements in the recommendations affects the returned precision and recall scores.

Parameters:

observed_user_items : SFrame

An SFrame containing observed user item pairs, where the first column contains user ids and the second column contains item ids.

recommendations : SFrame

An SFrame containing columns pertaining to the user id, the item id, the score given to that pair, and the rank of that item among the recommendations made for user id. For example, see the output of recommend() produced by any turicreate.recommender model.

cutoffs : list[int], optional

The cutoffs to use when computing precision and recall.

Returns:

out : SFrame

An SFrame containing columns user id, cutoff, precision, recall, and count where the precision and recall are reported for each user at each requested cutoff, and count is the number of observations for that user id.

Notes

The corner cases that involve empty lists were chosen to be consistent with the feasible set of precision-recall curves, which start at (precision, recall) = (1,0) and end at (0,1). However, we do not believe there is a well-known consensus on this choice.

Examples

Given SFrames train_data and test_data with columns user_id and item_id:

>>> from turicreate.toolkits.recommender.util import precision_recall_by_user
>>> m = turicreate.recommender.create(train_data)
>>> recs = m.recommend()
>>> precision_recall_by_user(test_data, recs, cutoffs=[5, 10])