Evaluating ChatGPT’s performance in creative text translation for communication : a case study from English into German

Abstract

In recent years, there has been a growing interest in using generative artificial intelligence (AI) for machine translation (MT) tasks. One notable feature of generative AI systems is prompting, through which users can provide instructions to guide the output. Prompting presents both opportunities and challenges in the field of Translation Studies. Therefore, this study aims to evaluate the quality of MT output generated by ChatGPT. The impact of prompting techniques on translation quality is also analyzed in order to gain insights into optimizing the use of generative AI in translation workflows. The study employed three prompts – zero-shot prompt, context prompt, and few-shot with context prompt – to translate a creative text for the English-German language pair. The raw MT outputs were manually annotated, and the results of these annotations were then compared. The results revealed that most errors concern style. Interestingly, the translation produced using the simple zero-shot prompt outperformed those generated with the other two more complex prompts. However, the longer prompts led to a steady reduction in stylistic and fluency errors, while accuracy errors increased.