Notes - Recommendation Metrics

Setting Goals and Choosing Metrics for Recommender System Evaluations

• Number of items that are either relevant or irrelevant and either contained in the recommendation set of a user or not
• How many of Top K contains relevant items

• If the recommendation list contains only relevant items, then the area under the curve is in fact zero
• Relevant items that are retrieved at the end of the list with no irrelevant items following do not add to the area under the limited curve.

• A top-k list that contains more relevant items will yield a higher score than a list with less relevant items
• How many of Top K contains relevant items. If the recommendation list contains only relevant items

Common metrics to evaluate recommendation systems

ROC Curve

A ROC curve plots recall (true positive rate) against fallout (false positive rate) for increasing recommendation set size

• True Positive items are therefore the items that you showed in your Top-N list that match what the user preferred in her held-out testing set
• False Positive are the items in your Top-N list that don't match her preferred items in her held-out testing set
• True Negative items are those you didn't include in your Top-N recommendations and are items the user didn't have in her preferred items in her held-out testing set.
• False Negative are items you didn't include in your Top-N recommendations but do match what the user preferred in her held-out testing set. 

Classification: ROC Curve and AUC

On Sampled Metrics for Item Recommendation

Keep Exploring!!!