ChatGPT vs translation services: A comparative study

Apart from writing code or explaining concepts, ChatGPT can also be a useful tool for translating text. But how does it compare to sophisticated machine translation providers that do the same thing?

And especially those that can do so for free, such as Google Translate?

That’s what we’ll cover in this ChatGPT translation guide, as we explore:

• the technology that powers ChatGPT,
• what makes the artificial intelligence (AI) chatbot a potentially good translation option, and
• how it compares to typical machine translation providers.

We’ll also shed light on ways of improving ChatGPT’s translations and assessing their quality. Last but not least, we’ll advise on when using other translation tools may be more appropriate in certain situations.

Machine translation: an introduction

With ChatGPT being a form of machine translation, it’s worthwhile looking into the origins of this technology.

Also known as AI translation, machine translation is a subfield of computational linguistics that uses software to translate text, or speech, into different languages. It enables a computer to produce translations without human intervention, and quicker than humans can too.

The very first form of machine translation was developed in the 1970s. It was referred to as rule-based machine translation (RBMT) and used sets of grammatical rules to generate word-for-word text translations. RBMT was a rigid translation method that tended to produce poor-quality translations.

Subsequently came statistical machine translation (SMT). This technology analyzed human-provided translations, then used prediction algorithms to determine the best way of translating a source text. Such translations were more accurate than those produced by RBMT by leaps and bounds, but advancements in machine translation technology didn’t stop there!

The most sophisticated machine translation technology in the translation industry’s arsenal is currently neural machine translation (NMT).

NMT uses deep neural networks to analyze and identify patterns in the source text before producing an appropriate translation. The deep neural networks in NMT systems can have different structures, or “architecture,” which affect the nature of their output.

NMT is also what powers ChatGPT’s translation capabilities. ChatGPT is based on the transformer architecture – the Generative Pretrained Transformer (GPT) architecture, to be precise. GPT has been trained on large datasets and can generate human-like responses when provided with inputs.

The makers of ChatGPT, OpenAI, then packaged its GPT language model into a chat interface, allowing ChatGPT to hold entire conversations with humans with ease.

What are the advantages of machine translation?

There are reasons why humans have been steadily working on improving machine translation technology since the 1970s. Machine translation offers superior benefits in terms of:

• Speed: Machines can translate text faster than humans can, making it an appealing option for translating large volumes of text in a short time.
• Cost: Machine translation typically costs less to undertake than human translation. This is especially true for large-scale projects.
• Accessibility: Online machine translation tools like Google Translate and Microsoft Translator are readily available, making it possible for practically anyone in the world to translate text quickly and easily.
• Consistency: Given their systematic manner of operation, machine translation tools can reliably translate text following certain rules or in a certain style. This is ideal for ensuring consistency in the translation of terminology, or for maintaining brand voice across the translated text.
• Round-the-clock availability: As machines don’t require rest, machine translation services are available for use 24/7.

That said, conventional machine translation technology still has room for improvement. For example, the technology may face difficulty in accurately translating idiomatic language and accounting for context, compared to if a professional – and human – translator were to do the same job.

But ChatGPT isn’t your typical machine translation tool. Strictly speaking, it wasn’t even intentionally built for translation purposes! And such unorthodoxy is where it may have an edge in producing translations.

Translating text with ChatGPT

The nature of ChatGPT makes it a useful language translation tool in areas such as:

• Contextual understanding: As ChatGPT is trained to generate human-sounding text, it may be able to provide more nuanced translations in situations involving casual or conversational text. Machine translation systems that translate text more literally may not be able to replicate the original sentiment or tone of such source text in their translations as effectively.
• Multifunctionality: Apart from translation, ChatGPT can also perform other language-related tasks. For example, you can ask it to translate a sentence, then ask a follow-up question about that translation’s contents. These questions can range from asking whether the translation would be appropriate for a professional setting, or asking for the meaning of a certain word in it.

• Interactivity: The AI chatbot can interact with you as you work on your translations. For instance, if you aren’t satisfied with a translation or want it explained, you can have a back-and-forth conversation with ChatGPT to refine and clarify its translations.
• Informal and slang translation: Given how ChatGPT has been trained on a diverse range of Internet text, it may have more superior translation ability for translating informal language, Internet slang, and idioms compared to traditional machine translation systems.

Despite the translation prowess that ChatGPT brings to the table, there are situations where you may want to look for alternative translation solutions.

If you’re translating legal documents for high-stakes international business deals, for example, it may be prudent to delegate the work to human experts skilled in translating technical legal jargon.

Also, while ChatGPT can translate a couple of paragraphs just fine, the AI tool has a rough limit of 3,000 words per request. As a result, it isn’t an ideal tool for translating substantial amounts of text (say 10,000 words or even more). However, this may be the scale of the endeavor for efforts such as website translation.

How do the translation capabilities of machine translation providers compare with those of ChatGPT?

Given ChatGPT’s present limitations, website translation solutions like Weglot currently use machine translation software, such as Google Translate, Microsoft Translator, and DeepL, to translate website content at scale instead.

Here’s how the translation capabilities of these machine translation providers compare against those of ChatGPT:

Strengths of machine translation providers

• Machine translation providers are dedicated translation tools that have been specifically trained to handle translation tasks. As a result, they can dependably provide quality translations for common language pairs and text involving standard or formal language. In contrast, ChatGPT’s translations may not be the most accurate as it wasn’t designed specifically for translating text.
• These tools can also help translate text in many languages, including ones that aren’t widely spoken or written. For example, our Weglot website translation solution currently supports the translation of over 110 languages. ChatGPT can do a decent job at translating common languages, but it may have trouble when provided with text in a less commonly used language.
• Machine translation providers are also well-suited for translating large volumes of text within a short time. As mentioned above, however, ChatGPT can accommodate only roughly up to 3,000 words per request. Manually giving the chatbot new text to translate once you hit the word limit for a previous request can also be time-consuming.

Limitations of machine translation providers

• While their translation ability has improved over the years, machine translation providers can still struggle with translating idiomatic expressions and cultural references. They may also have trouble interpreting the context surrounding a given text and tailoring their translation accordingly.
• Machine translation providers typically don’t offer interactive features, but directly output a translation when fed a piece of text. Unlike ChatGPT, they therefore can’t discuss the contents of their translations with you or address questions about them.

Machine translation software can be used to translate website content, multilingual blog content, support documentation, user-generated content on social media, and customer testimonials. Machine translation providers in certain industries may also tailor their software to meet their customers’ specific needs.

Just take website localization, for example. Localizing a website generally involves first localizing its content, then publishing the localized output. If you’re using a non-specialized machine translation provider to translate and localize your website, you may need to work with your development team to implement your localized content.

However, we’ve developed our Weglot website translation solution to not only translate your website content, but also conveniently display the finalized translations on your website for you. You won’t have to delve into your website’s technical backend to get these up.

Weglot also provides other helpful features for helping your multilingual website rank well in online searches. These include automatic hreflang tag implementation, and the organizing of your translated content into dedicated language subdomains or subdirectories.

How to improve ChatGPT’s translation accuracy

So far, we’ve seen that ChatGPT is a potentially useful tool for translating smaller chunks of text presented in common languages. While its translations tend to be satisfactory, using pivot prompting and back translation tactics can help improve its translation performance.

Pivot prompting

Pivot prompting is a technique that involves first translating the source text into a language that ChatGPT knows well, then “pivoting” by translating the translation into your intended target language.

Let’s say we want to translate the Welsh sentence “Helo, sut ydych chi?” into English. Welsh isn’t an especially common language, so we may benefit from “pivoting” into a more commonly used language, such as French, first.

We’d therefore:

• First use ChatGPT to translate the phrase into French. (ChatGPT gave us the French sentence of “Bonjour, comment ça va?” when we did so.)
• After that, we’d get ChatGPT to translate the French sentence of “Bonjour, comment ça va?” into English. (And here, ChatGPT translated the sentence into “Hello, how are you?”)

Pivot prompting can improve ChatGPT’s translation accuracy as you don’t directly translate your text from a low-resource language on which the chatbot may not have been sufficiently trained. Instead, the chatbot translates the source text with reference to an intermediary language for which it can generate translations more easily.

That said, pivot prompting may introduce other translation errors if mistranslations occur during the intermediate translation step. That’s because ChatGPT will work off a different text to produce the translation in your target language rather than the original source text itself.

It’s just like playing a game of Telephone, where the initial message may inadvertently get modified as it travels from person to person!

Back translation

When using ChatGPT to translate text, you could also have it first translate your text into the target language, before translating it back to the source language.

Let’s say we want ChatGPT to translate the English sentence “Hello, how are you?” into Spanish. So we submitted our request, and ChatGPT returned the phrase “Hola, ¿cómo estás?”

We then got ChatGPT to translate “Hola, ¿cómo estás?” back into English. And here, it gave us our initial English sentence of “Hello, how are you?”

Carrying out such a back translation (or “reverse translation”) can help you double-check the accuracy of ChatGPT’s provided translation. This is especially true if you're more familiar with the source language than the target one.

But as with pivot prompting, back translation introduces extra steps in the translation process. Take care that you don’t introduce translation errors into each of these steps!

Overcoming limitations with traditional natural language prompts for machine translation

The general rule of thumb in machine translation is to provide input, or prompts, that are as clear, direct, and specific as possible. Doing so helps the natural language processing (NLP) tool understand what you’re asking of it and generate a more accurate translation.

But with a more context-aware and flexible translation tool like ChatGPT, following this general rule of thumb may lead to less effective translations in situations such as:

• Translating cultural nuances and slang: To provide clear and direct translation prompts, you could remove all cultural expressions, idioms, and slang from your source text. But doing so would precisely lead to ChatGPT producing translations that lack such nuance. You can preserve it by leaving such cultural references in your text, and requesting ChatGPT to translate its gist or overarching meaning (rather than doing a direct, word-for-word translation).
• Ambiguity in the meaning of words: ChatGPT may have difficulty producing an accurate translation if the source text in the prompt offers insufficient context as to the most appropriate meaning for a particular word. To overcome this issue, clarify your prompt to provide extra context on these ambiguous words, or information on their intended meaning.
• Creativity and style: Using a more traditional translation prompt might result in you getting a correct but bland translation. Try asking ChatGPT to provide a more creative translation, or one that matches a certain tone or style. Depending on the situation, for instance, you can ask ChatGPT to produce a translation that has a poetic style or is in a formal tone.

Experiments for assessing the quality of machine translation

After getting a text translation, how do you know whether it’s suitable for use? You don’t want a situation where you put resources toward obtaining a translation but fail to get the results you’re looking for.

This could be where, for example, your multilingual website is so poorly translated that your target customers can’t understand it.

There are various methods of assessing the machine translation quality (including that for ChatGPT translation), both via automatic and human means. We’ll take a look at some of them in this next section.

As you review these methods, consider which you may want to use to assess the quality of your machine translations. Undertaking a combination of automatic and human evaluation methods will tend to provide a more comprehensive assessment of translation quality rather than relying solely on automatic or human methods alone.

Automatic translation evaluation methods

In automatic translation evaluation, you’d use metrics and algorithms that automatically score the quality of the resulting translation. These include:

• BLEU (Bilingual Evaluation Understudy): This algorithm considers how closely a machine-translated text resembles that produced by a professional human translator, then provides a score between zero and one. The higher a translation’s BLEU score, the higher a translation’s quality. You can explore the BLEU algorithm here.
• METEOR (Metric for Evaluation of Translation with Explicit ORdering): The BLEU algorithm suffers from some limitations, such as having a limited ability to account for chunks of translated words that match those in a “gold standard” reference translation. METEOR therefore scores translations against a provided reference text with the aim of accounting for such limitations. For example, it imposes a “chunk penalty” for every chunk it finds in a translation (in the most ideal situation, the translation should have only one chunk). Access METEOR here.
• TER (Translation Edit Rate): This metric measures the number of human edits a machine translation needs before it matches a reference translation. The lower the TER score, the fewer edits a translation is deemed to need (and hence its higher quality). Try TER for your machine translations here.
• ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Among other metrics, ROUGE evaluates a machine translation against a set of reference translations based on the extent of recall and precision between the two. “Recall” measures the number of words in the machine translation that can be found in the reference translation. On the other hand, “precision” considers the proportion of such overlapping words against the total number of words in the machine translation. A ROUGE tool is available here.
• BERTScore: BERTScore compares a machine translation against a reference translation, and scores the former according to how similar its meaning is to that of the latter. Whether there is an exact match between the words in the machine and reference translations is less important. Get the code for BERTScore here.

Human translation evaluation methods

As the name suggests, human translation evaluation methods involve having human evaluators rate translations according to more subjective considerations. Such methods include:

• Adequacy assessment: The human evaluator rates the adequacy of translations on a scale based on how accurately it conveys the original text’s meaning.
• Fluency or readability assessment: Here, the evaluator considers the fluency of the translation – in other words, whether it reads as if it had been originally written in the target language.
• Error analysis: This method involves identifying errors in the translation, counting how many there are, and sorting each error into categories such as mistranslations, omissions, and additions.
• Ranking or preference tests: Evaluators receive multiple translations of the same text, such as those produced by different machine translation systems or software versions. They then rank them in order of highest to lowest quality.
• Post-editing effort: Human evaluators correct the machine translation and keep track of the number of edits required. They may also note the amount of time they needed to complete the edits.

Choose the right translation tool for your business

ChatGPT’s ability to understand context, and workshop its translations through back-and-forth interactions with you, can make it a good alternative to using a machine translation provider for simple translation requests. This is especially true when you use targeted prompts, and tactics such as pivot prompting, to refine its translations.

But if you have specialized translation needs, relying on a translation solution purpose-built for such needs may be the way to go. Take the situation of website translation, for instance, which typically involves translating large volumes of text for displaying online.

ChatGPT translation prompts are limited to about 3,000 words each – or about only one to three blog posts’ worth of words. If your website has many more words than that, repeatedly copying and pasting chunks of website copy into the AI chatbot for translation purposes can be tedious.

Meanwhile, machine translation providers may be able to translate all the content on a website at once, but there’s still the issue of moving the translations to your multilingual website.

The ideal solution here is to use Weglot, which will not only provide high-quality website translations but also add them to your website for you. Using the Visual Editor, you can preview how the translations will appear on your website and edit them – within the context of your website’s design – on the spot.

Weglot then displays them under language subdomains or subdirectories, depending on how you want to organize your content for search engine optimization (SEO) purposes. Other features, such as automatic hreflang tag implementation, also help give your website the best chances of getting seen by multilingual search engine users within your target audience.

Explore how Weglot can help you achieve seamless website translation by signing up for a 10-day free trial here.