My attempt at an AI writing assistant for Chinese

29 Feb, 2024

A dragon on display in Beijing to celebrate the new year

February 2024 marks the beginning of the Dragon year in the Chinese zodiac. It's a particularly significant year for me personally, as I was born in a Dragon year!

After living in Beijing for over 7 years, one of my wishes for the new year is to improve my Chinese skills. In this post I'll explain how I've been using AI to help me write better Chinese.

I've split the post into the following sections:

• Defining the problem
• My initial approach
• A successful solution
• Putting it all together
• Ideas for future work

Major disclaimer: I'm not an expert in linguistics or language processing. I've tried to minimize inaccuracies in this post, but it's possible that some have slipped through. If you have a correction, suggestion, or comment, I'd love to hear your feedback! You can drop me a message here.

Defining the problem

When I want to write or say something in Chinese, there's often a key piece of vocabulary or grammar that I'm missing. I can often get around my lack of knowledge by expressing myself in a different way, but otherwise:

• If I'm in a hurry, I write fully in English and ask a translation app to tell me the Chinese
• If I'm not in a hurry, I write in Chinese as best as I can, then use a dictionary to figure out the missing parts

Ideally there'd be a way to get the best of both worlds - the speed of a translation app combined with the self-improvement benefits of attempting to write in Chinese first.

What I really want is a tool that lets me write a mix of Chinese and English, then recommends a proper Chinese expression. I'm not aware of an existing tool, so I decided to build my own.

My core requirements

1. The tool can translate a mix of Chinese characters, pinyin, and English into a good Chinese expression
2. The Chinese expression includes accurate pinyin

Bonus requirements

3. The tool is as fast as Microsoft Translator, Google Translate, etc
4. My recurring costs are low, even if other people use the tool

Bumper bonus requirements

5. There's a way to get an explanation of the Chinese expression
6. There's a way to ask follow-up questions about the Chinese expression

My initial approach

I started by testing a couple of large language models (LLMs) to see how well they could generate Chinese text given a mix of Chinese and English. At the time, I only tested GPT-3.5 and GPT-4 from OpenAI. Google has since released Gemini Ultra, so I've also included that here for comparison.

I experimented with several mixed-language expressions to get a feel for quality, but the one I'll use throughout this post is "你 dei tie 你的 shoelaces". The intended meaning in English is "you need to tie your shoelaces". Here's the problem I posed to the LLMs:

I'm trying to say something like "你 dei tie 你的 shoelaces" in Chinese. Please provide a single recommendation for how I should say it using standard Mandarin. Respond with the Chinese expression first, then the corresponding pinyin transliteration. Do not provide anything else; no discussion is needed.

And here are the results:

All three Chinese expressions are slightly clunky because of the "你的" part, which means "your" as in "your shoelaces". In English, "your" is needed in this context, whereas in Chinese it's perfectly normal to omit it. To the best of my understanding, saying "你的鞋带" is not incorrect - but it can come across as overly formal/specific.

Since I included "你的" in my original expression, it's not surprising to see the LLMs include it too. But I was hoping that the LLMs would remove it. I also tried modifying my instructions to encourage the LLMs to be more colloquial, without much success.

A more significant issue with the LLM responses is that they use "xì" as the pinyin for "系". The character "系" has multiple meanings and pronunciations, and in this context "jì" is the correct pronunciation.

These issues left me with low confidence in the quality of the LLM responses. I'd need to adjust my approach.

A successful solution

Maybe the LLMs are able to output good Chinese, but the task I presented was too complex. Maybe I need to feed the LLMs an English interpretation of the mixed-language expression and ask for a Chinese translation.

I could manually provide an English interpretation. But if I need to write English, I might as well use a translation app from the beginning - defeating the entire point! So I modified my LLM instructions to ask for English:

I'm trying to say something like "你 dei tie 你的 shoelaces" in Chinese. Please provide an English interpretation of what I'm trying to say. Do not provide anything else; no discussion is needed.

The result from GPT-3.5 was:

You need to tie your shoelaces

Great! I tried a bunch more expressions and consistently got excellent results from GPT-3.5. This is good news, because GPT-3.5 is fast and cheap enough for me.

With an English interpretation in hand, I could ask the LLMs for a Chinese translation:

I'm trying to say "You need to tie your shoelaces" in Chinese. Please provide a single recommendation for how I should say it using standard Mandarin. Respond with the Chinese expression first, then the corresponding pinyin transliteration. Do not provide anything else; no discussion is needed.

I also included three popular translation apps in the comparison. Here are the results:

GPT-4, Gemini Ultra, and Microsoft Translator all performed well. Notably, GPT-4 and Gemini Ultra appear to have benefited from the task being less complex, as this time they provided pinyin that is correct for the context.

After experimenting with some more mixed-language expressions, my conclusion was that Microsoft Translator generally performs better than GPT-4 at direct translation. I haven't investigated Gemini Ultra further because it wasn't available at the time I did this work. Regardless, Microsoft Translator is significantly faster and cheaper than either GPT-4 or Gemini Ultra.

So, to solve my problem, I'm using a combination of GPT-3.5 and Microsoft Translator. GPT-3.5 generates an English version of what I'm trying to write, then Microsoft Translator translates the English version into Chinese.

Putting it all together

To automate my solution, I built a tool called Unscrambler. Here's what it looks like:

Unscrambler shows the English version of the mixed-language text that you write, so that you can verify how your input was interpreted by GPT-3.5, then Unscrambler shows the Chinese translation provided by Microsoft Translator. The pinyin version of the Chinese translation is hidden by default, giving you a chance to try reading the Chinese characters before consulting the pinyin!

If you'd like to understand the translation in more detail, click Copy AI help request to copy a conversation starter to your clipboard. You can then paste the conversation starter into an AI chatbot (e.g., ChatGPT) and have a chat about the translation. If possible, I recommend that you use GPT-4 or Gemini Advanced.

One reason you might want to have a follow-up chat is to ask about vocabulary differences between your input and the Chinese translation. For example, here's an excerpt from a chat with GPT-4 about "dei" vs "需要":

In the context you've provided, using "dei" instead of "需要" makes the sentence sound less formal and more akin to something you'd hear in daily conversation. Thus, "你得系好鞋带" would be a less formal way to say "You need to tie your shoelaces," maintaining the same overall meaning but in a more casual tone.

In this case, the info from GPT-4 is accurate. But please try to independently verify any info that you get from an AI-based tool - and that includes Unscrambler!

Ideas for future work

• Are there any Chinese language LLMs or dedicated multilingual LLMs that can generate good Chinese from a mix of Chinese and English? Will Gemini 1.5 be able to do it? (I can't access Gemini 1.5 yet)

• Can any LLMs smaller than GPT-3.5 generate good English from a mix of Chinese and English?