logo Idiap Research Institute        
 [BibTeX] [Marc21]
Using Crowdsourcing to Compare Document Recommendation Strategies for Conversations
Type of publication: Conference paper
Citation: Habibi_ACMRECOMMENDERSYSTEMS2012_2012
Publication status: Published
Booktitle: RecSys, Recommendation Utility Evaluation (RUE 2012)
Year: 2012
Month: September
Pages: 15-20
Location: Dublin, Ireland
Crossref: Habibi_Idiap-RR-14-2012:
Abstract: This paper explores a crowdsourcing approach to the evaluation of a document recommender system intended for use in meetings. The system uses words from the conversation to perform just-in-time document retrieval. We compare several versions of the system, including the use of keywords, retrieval using semantic similarity, and the possibility for user initiative. The system's results are submitted for comparative evaluations to workers recruited via a crowdsourcing platform, Amazon's Mechanical Turk. We introduce a new method, Pearson Correlation Coefficient-Information Entropy (PCC-H), to abstract over the quality of the workers' judgments and produce system-level scores. We measure the workers' reliability by the inter-rater agreement of each of them against the others, and use entropy to weight the difficulty of each comparison task. The proposed evaluation method is shown to be reliable, and the results show that adding user initiative improves the relevance of recommendations.
Projects Idiap
Authors Habibi, Maryam
Popescu-Belis, Andrei
Added by: [UNK]
Total mark: 0
  • Habibi_ACMRECOMMENDERSYSTEMS2012_2012.pdf