Intrinsic and Extrinsic Automatic Evaluation Strategies for Paraphrase Generation Systems

Authors: Tulu Tilahun Hailu, Junqing Yu, Tessfu Geteye Fantaye

Paraphrase is an expression of a text with alternative words and orders to achieve a better clarity. Paraphrases have been found vital for augmenting training dataset, which aid to enhance performance of machine learning models that intended for various natural language processing (NLP) tasks. Thus, recently, automatic paraphrase generation has received increasing attention. However, evaluating quality of generated paraphrases is technically challenging. In the literature, the importance of generated paraphrases is tended to be determined by their impact on the performance of other NLP tasks. This kind of evaluation is referred as extrinsic evaluation, which requires high computational resources to train and test the models. So far, very little attention has been paid to the role of intrinsic evaluation in which quality of generated paraphrase is judged against predefined ground truth (reference paraphrases). In fact, it is also very challenging to find ideal and complete reference paraphrases. Therefore, in this study, we propose semantic or meaning oriented automatic evaluation metric that helps to evaluate quality of generated paraphrases against the original text, which is an intrinsic evaluation approach. Further, we evaluate quality of the paraphrases by assessing their impact on other NLP tasks, which is an extrinsic evaluation method. The goal is to explore the relationship between intrinsic and extrinsic evaluation methods. To ensure the effectiveness of proposed evaluation methods, extensive experiments are done on different publicly available datasets. The experimental results demonstrate that our proposed intrinsic and extrinsic evaluation strategies are promising. The results further reveal that there is a significant correlation between intrinsic and extrinsic evaluation approaches.


Journal: Journal of Computer and Communications
DOI: 10.4236/jcc.2020.82001 (PDF)
Paper Id: 98203 (metadata)

See also: Comments to Paper

About scirp

(SCIRP: is an academic publisher of open access journals. It also publishes academic books and conference proceedings. SCIRP currently has more than 200 open access journals in the areas of science, technology and medicine. Readers can download papers for free and enjoy reuse rights based on a Creative Commons license. Authors hold copyright with no restrictions. SCIRP calculates different metrics on article and journal level. Citations of published papers are shown based on Google Scholar and CrossRef. Most of our journals have been indexed by several world class databases. All papers are archived by PORTICO to guarantee their availability for centuries to come.
This entry was posted in JCC. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *