(21) Meta-Evaluation of Automatic Evaluation Methods for Machine Translation using Patent Translation Data in NTCIR-7
Hiroshi Echizen-ya, Terumasa Ehara, Sayori Shimohata, Atsushi Fujii, Masao Utiyama, Mikio Yamamoto, Takehito Utsuro, Noriko Kando
Proceedings of the 3rd Workshop on Patent Translation, pp.9-16, 2009-8
Herein, we describe a meta-evaluation based on various automatic evaluation methods for machine translation using patent translation data in NTCIR-7: we investigated the correlation between results obtained using automatic evaluation methods and human judges by particularly addressing sentence-level evaluation because the improvement of sentence-level evaluation is very important to realize high-quality automatic evaluation that is equivalent to that of human judges. Through this meta-evaluation, we confirmed that the automatic evaluation methods can yield high correlation in sentence-level adequacy (about 0.6-0.7). However, the correlation of sentence-level fluency in the rule-based machine translation systems was insufficient (about 0.5). These results indicate that automatic evaluation methods that can use grammatical knowledge effectively must be developed to improve sentence-level correlation.
>>
NEXT