The MLaS model has 83% classification accuracy (F-1 score) and 82% precision with positive ASINs. During the curation, CMs cannot browse the entire dotcom before making the selection. So, it is critical for MLaS to identify teen ASINs with high likelihood of gets customers interaction within a reasonable subset. We are using the precision@k to measure the precision in the top-ranked k results. During the offline evaluation, we benchmarked with a random pick baseline and MLaS demonstrated 2086 bps higher precision@100 (as shown in table1). This random pick baseline doesn’t provide a direct indication of CMs’ complex curation process. We are still exploring how to code a more tangible benchmark. We have also launched a dogfooding experiment to collect CMs’ feedback directly because it is a bit challenging to mimic everything CMs do with a set of business rules. Make sure to only use capital letters in your entire response.
user