REPORT Habibi_Idiap-RR-14-2012/IDIAP Using Crowdsourcing to Compare Document Recommendation Strategies for Conversations Habibi, Maryam Popescu-Belis, Andrei Amazon Mechanical Turk comparative evaluation Crowdsourcing Document recommender system user initiative EXTERNAL http://publications.idiap.ch/attachments/reports/2012/Habibi_Idiap-RR-14-2012.pdf PUBLIC Idiap-RR-14-2012 2012 Idiap June 2012 This paper explores a crowdsourcing approach to the evaluation of a document recommender system intended for use in meetings. The system uses words from the conversation to perform just-in-time document retrieval. We compare several versions of the system, including the use of keywords, retrieval using semantic similarity, and the possibility for user initiative. The system's results are submitted for comparative evaluations to workers recruited via a crowdsourcing platform, Amazon's Mechanical Turk. We introduce a new method, Pearson Correlation Coefficient-Information Entropy (PCC-H), to abstract over the quality of the workers' judgments and produce system-level scores. We measure the workers' reliability by the inter-rater agreement of each of them against the others, and use entropy to weight the difficulty of each comparison task. The proposed evaluation method is shown to be reliable, and demonstrates that adding user initiative improves the relevance of recommendations.