CONF Hajlaoui_CICLING-2013_2013/IDIAP Assessing the Accuracy of Discourse Connective Translations: Validation of an Automatic Metric Hajlaoui, Najeh Popescu-Belis, Andrei discourse connectives. Machine Translation MT evaluation EXTERNAL https://publications.idiap.ch/attachments/papers/2013/Hajlaoui_CICLING-2013_2013.pdf PUBLIC University of the Aegean - 14th International Conference on Intelligent Text Processing and Computational Linguistics Samos, Greece 7817 236-247 2013 Springer 10.1007/978-3-642-37256-8_20 doi Automatic metrics for the evaluation of machine translation (MT) compute scores that characterize globally certain aspects of MT quality such as adequacy and fluency. This paper introduces a reference-based metric that is focused on a particular class of function words, namely discourse connectives, of particular importance for text structuring, and rather challenging for MT. To measure the accuracy of connective translation (ACT), the metric relies on automatic word-level alignment between a source sentence and respectively the reference and candidate translations, along with other heuristics for comparing translations of discourse connectives. Using a dictionary of equivalents, the translations are scored automatically, or, for better precision, semi-automatically. The precision of the ACT metric is assessed by human judges on sample data for English/French and English/Arabic translations: the ACT scores are on average within 2% of human scores. The ACT metric is then applied to several commercial and research MT systems, providing an assessment of their performance on discourse connectives.