logo Idiap Research Institute        
 [BibTeX] [Marc21]
Using Crowdsourcing to Compare Document Recommendation Strategies for Conversations
Type of publication: Idiap-RR
Citation: Habibi_Idiap-RR-14-2012
Number: Idiap-RR-14-2012
Year: 2012
Month: 6
Institution: Idiap
Abstract: This paper explores a crowdsourcing approach to the evaluation of a document recommender system intended for use in meetings. The system uses words from the conversation to perform just-in-time document retrieval. We compare several versions of the system, including the use of keywords, retrieval using semantic similarity, and the possibility for user initiative. The system's results are submitted for comparative evaluations to workers recruited via a crowdsourcing platform, Amazon's Mechanical Turk. We introduce a new method, Pearson Correlation Coefficient-Information Entropy (PCC-H), to abstract over the quality of the workers' judgments and produce system-level scores. We measure the workers' reliability by the inter-rater agreement of each of them against the others, and use entropy to weight the difficulty of each comparison task. The proposed evaluation method is shown to be reliable, and demonstrates that adding user initiative improves the relevance of recommendations.
Keywords: Amazon Mechanical Turk, comparative evaluation, Crowdsourcing, Document recommender system, user initiative
Projects Idiap
Authors Habibi, Maryam
Popescu-Belis, Andrei
Crossref by Habibi_ACMRECOMMENDERSYSTEMS2012_2012
Added by: [ADM]
Total mark: 0
  • Habibi_Idiap-RR-14-2012.pdf (MD5: f8a761de0bac8f24b9df7295ab193ad7)