User:TJones (WMF)/Notes/N.O.R.M. MVP Design Notes/Georgian Transliteration

(September 2025) Despite the expectations in T127003, it turned out that given a search in Latin or Cyrillic script, it is not "fairly easy to render that search into Georgian script", alas!

Romanization

As noted in the enwiki article on the Romanization of Georgian, there isn't really a single standard for transliteration of Georgian into the Latin alphabet. The problem ultimately stems, I think, from the fact the Georgian has too many sounds for the Latin alphabet (to be fair, so does English).

Some letters are pretty consistent—a for ა, b for ბ, g for გ, etc.—others have many options across transliteration systems, including some fairly hard-to-type options, depending on your keyboard—like t, t', t̕, tʻ, T, or t̊ for თ, or zh, ž, J, j, or g̃ for ჟ. (Because Georgian doesn't use capital letters, some transliteration systems treat upper- and lowercase varieties as different letters!)

The "unofficial" system, being unofficial, isn't entirely consistent, either. Looking at various transliteration descriptions, transliteration tools on the internet, and what is used in our query logs, I see that normal people doing normal stuff on the internet almost never use any of the "official" or academic systems (which are designed for institutions to be able to losslessly and reversibly encode Georgian names into the Latin alphabet).

The informal system deals with the lack of enough Latin letters by either distinguishing upper- and lowercase letters (like t for ტ and T for თ), or using digraphs (like sh for შ or ts for წ—and one trigraph, tch for ჭ). Some choices seem to be influenced by other Latin-based writing systems (sh for შ == /ʃ/ and ch for ჩ == /t͡ʃ/ look like they are based on English (and not French, for example), while c for ც == /t͡s/ or წ == /t͡sʼ/ looks to be inspired by Czech, German, or Polish). And of course sometimes things are just ambiguous, like c for ც or წ, and t for თ or ტ. For native speakers, these are fine for informal communication; if you replaced every sh with s and ch with c above, it would not be too hard in context to figure out coices, Englis, Frenc, distinguising, Czec and Polis.

Interestingly, it looks like some people who use the system that distinguishes capital letters from lowercase will still uppercase the first letter of names transliterated into Latin!

Cyrillicization

I really couldn't find a lot of information on Cyrillic transliteration of Georgian. There is one article on Russian Wikipedia on "Georgian-Russian practical transliteration", which seems to be based on an academic paper from 1972—which is actually available, through the magic of the internet!

The intro text of the article says something like, "Georgian-Russian practical transliteration describes the rules of Russian transliteration of Georgian proper names for which there is no historically established (traditional) transliteration."

So, based on that and the actual mappings, it sounds like the goal is to translate Georgian names into something reasonably pronounceable by Russian speakers, not a lossless encoding of the original Georgian, or even something used by Georgian speakers. т can be თ or ტ, п can be პ or ფ, к can stand for all three of კ, ყ, and ქ, and more.

Playing the Odds

Autocomplete allows for two letters to differ in a title match, and I would suggest that Did You Mean should verify that its suggestions don't get zero results. This kind of flexibility either partially rehabilitates (autocomplete) or hides (Did You Mean) poor transliterations.

So, based on the frequency of letters, and—for Latin—looking to an input conversion script on Georgian Wikipedia for guidance—I made reasonable choices on how to map single Latin and Cyrillic letters to Georgian letters.

For example, while Cyrillic к has three Georgian letters that it could be mapped from, კ seems to be the best bet. (In my sample—კ:55.4%, ყ:11.8%, ქ:32.8%.)

These single-letter mappings are far from perfect, but they work pretty well on some example queries I pulled from our logs.

This left the Latin digraphs (sh, ch, ts, etc.) which are best when not treated as single letters—and two Cyrillic letters—т and п—for which there was no clear favorite.

Many of the digraphs seem unlikely as separate letters, like having /s/+/h/ in a word rather than /ʃ/ (though it can occur in English, in compounds like glasshouse), so I also evaluated mapping them to their equivalents in the style that uses upper- and lowercase.

To handle capitalized names and all uppercase words, I also looked at lowercasing everything that was left over after mixed-case mapping, and mapping the remnants.

In Context

As I mentioned before, there are no clear favorites when mapping Cyrillic т and п to Georgian, so I set out to see if contextual clues could help me decide.

I took all of the Georgian words in my sample queries, and looked for simple contexts that favor one mapping over another. For example, for т (t-like) mappings ტ is more likely to appear between ი and ა than თ is. So, if we've mapped all the other characters, and we have იтა, we map it to იტა (not ითა—though that combination does occur, იტა is 7x more likely).

In the end, I had about 165 contexts that had predominate mappings for Cyrillic т and п. Most were one letter before and one letter after (or a word boundary before or after), but I also had fallback contexts of one letter before or after, and a final context-free mapping of т to ტ because it is slightly more common.

Fortunately there were no contexts for т that relied on the mappings of п (პ, ფ), and only two very uncommon ones for п that relied on mappings of т (თ, ტ). So, I mapped т first, then п.

The long list of contextual mappings isn't necessarily super efficient, but first I was trying to see if they were even reasonably effective.

The Proof in the Pudding

I ran a test where I tried mapping Latin or Cyrillic queries into Georgian script and—while it is definitely a very crude measure—I looked to see how many total results the queries got as-is, and under the various transliteration mappings. I looked at four variants for Latin mappings, and one for Cyrillic.

The four Latin variants were:

Latin Case Sensitive (LCS)—Map everything in the query as-is using the mixed-case mapping suggested in the aforementioned input conversion script. This can leave a lot of capitalized Latin letters unconverted, which means the mapping failed.

Latin Case Insensitive (LCI)—Lowercase the query, then use the mixed-case mapping suggested in the input conversion script. This handles all-caps and typical Latin-case words, but loses a lot of distinctions between upper- and lowercase Latin letters.

Latin Digraphs, Case Sensitive (LDCS)—Map lowercase digraphs (zh, gh, sh, dz, ts, kh, ch, and trigraph tch) to their mixed-case mapping equivalents (e.g., sh to S) and then use the mixed-case mapping suggested in the input conversion script. This can also leave a lot of capitalized Latin letters unconverted.

Latin Digraphs, Case Insensitive (LDCI)—Map lowercase digraphs and uppercase digraphs at the start of words to their mixed-case mapping equivalents, then use the mixed-case mapping suggested in the input conversion script, then lowercase the result and re-map using the mixed-case mapping.

For Cyrillic (Cyr): Lowercase the query, then map a couple of digraphs (дж and дз), then map everything but т and п, then use the context patterns to map т and п.

Results

I had 4691 Latin and 144 Cyrillic queries (i.e., 4835 queries with no Georgian script in them), taken from a sample of recent Georgian Wikipedia queries.

For the LCS and LDCS case-sensitive mappings, 3312 Latin queries (70.6%) were incomplete conversions, meaning they had leftover uppercase Latin characters after the conversion. All 26 English lowercase Latin letters get mapped to something, so the case insensitive mappings usually had nothing leftover, unless the query had uncommon Latin characters or diacritics. (I'd already checked that the letters in the more academic transliteration systems were not commonly used.)

For the Cyrillic mapping, 79 Cyrillic queries (54.9%) were incomplete conversions. Since the Cyrillic input was lowercased before mapping, these queries had characters that were unmappable. (I also did a search for other mappings that might use some of the unmappable Cyrillic queries, but didn't find anything.)

So, plenty of Latin and Cyrillic queries aren't suitable for transliteration, which is not unexpected. I skimmed 900 Latin queries and 110 (12.2%) where obviously English, French, or Spanish, and plenty were names. They were sorted alphabetically, so not quite a random sample, but obviously there are lots of Latin queries that are not transliterated.

I tracked the number of queries that got any results (nonZRR) and the total number of hits all of those queries got.

In addition to the four Latin variants and the Cyrillic mapping, we have:

Orig—the original query.

Combo—the original query if it got any hits, or the LDCI query if it got any hits, or the Cyrillic query if it got any hits.

For all 4835 queries:

	nonZRR	total hits
Orig	709	41,337
LCS	81	37,022
LCI	265	100,955
LDCS	95	37,680
LDCI	340	111,090
Cyr	80	2,346
Combo	909	85,285

Orig and the transliteration variants overlap, so Combo isn't an obvious sum of any others. The Latin transliterated queries clearly are capable of generating a lot of results in Georgian. As I said before, it's a crude measure; it tells us that we aren't always generating gibberish—but we can't be sure we are getting things right, either. (For example, in English, bat, bet, bit, bot, and but are all fine words, but they aren't interchangeable. They are probably usually better than bqt, though.)

Out of 4126 queries that originally got zero results:

	nonZRR	total hits
Orig	0	0
LCS	37	18,462
LCI	127	36,368
LDCS	48	19,103
LDCI	190	43,838
Cyr	10	110
Combo	200	43,948

Since all the original queries in this sample got zero results, Combo is just LDCI + Cyrillic.

And just to see how "poor-performing" queries do when transliterated, I looked at the stats for the 385 original queries that got from 1–5 results:

	nonZRR	total hits
Orig	385	794
LCS	16	11,527
LCI	51	14,316
LDCS	16	11,527
LDCI	58	14,476
Cyr	12	943

The Combo line is omitted from this table because it would be the same as the originals, since Combo only replaces to zero-results original queries.

I think this gives us some hints as to how autocomplete and Did You Mean suggestions could be helped by transliteration (before Georgian wiki searchers catch on that transliteration is available and intentionally use it).

An Unexpected Revelation!

After looking at a lot of examples, I think that in addition to having a reasonable system for converting transliterated Georgian text back into Georgian script, these mappings also provide a reasonable approximation for transliterating non-Georgian names into Georgian—particularly names spelled according to typical English and Russian spelling rules—as if English spelling has any actual rules!

That's probably still helpful, though.

After running the experiment above and looking at more examples—always try to look at more examples!—I found some ways to improve both Latin and Cyrillic transliteration.

Latin

Both ignoring case and handling digraphs for Latin transliteration was a bit of a challenge, and I originally settled on mapping digraphs at the beginning of a word regardless of case, on the assumption that a word starting with Sh- or SH- is probably representing the /‍ʃ/ sound rather than /‍s/+/‍h/ or /‍ʃ/+/‍h/ (since S can be used for /‍ʃ/). After looking at the example queries I had, I decided that I should just map any digraph that starts with a capital letter, which picks up word-initial cases, all-caps cases, and camelCase cases.

Cyrillic

So I started with about 165 trigram and bigram rules for mapping Cyrillic т and п. I sorted them based on frequency and removed obvious ones that would bleed other rules, but I didn't look that carefully. To decide which rules were really necessary, I pulled all the Cyrillic queries I could for Georgian Wikipedia (754), and counted how often each rule fired when transliterating all of those queries.

Some rules never fired because earlier rules affected their context. Sometimes, more-specific rules turned out not to be necessary in practice, and a later less specific rule could handle multiple cases at once.

With a little trial and error, I was able to get the ~165 contexts used to map Cyrillic т and п down to just 40, which should be more efficient. Also, if we later find a systematic problem with Cyrillic mappings for other letters—which certainly have some deficiencies—we might be able to find (and afford) another ~20 rules per letter to improve the contextual mappings.

PHP Implementation—Replacements and Regexes

When re-implementing the algorithm in PHP, I tried to use strtr() rather than str_replace() or preg_replace() as much as possible for efficiency—it's certainly faster than a bunch of separate regex replacements, especially when most of the replacements are fixed strings. However, there were some complications... as there always are.

I had to split a single call to strtr() for Cyrillic mapping into two so that the shorter single-letter and digraph replacements could create the environments for the contextual mappings for т and п (strtr() prefers to map longer strings first, of course). I also re-ordered some of the contextual mappings for т and п so that I could group together the ones based on word-boundaries (which require regexes). I also accepted a handful of deviations from my original samples as a result of these reorderings, other reorderings strtr() did itself internally, and the fact that three-letter contextual mappings can't overlap in strtr() (which is rare, but does happen).

(Since the Cyrillic mappings were never really a lossless transliteration scheme, it's all a most likely guess anyway, so a few changes in <0.1% of cases doesn't matter much. For Latin, only one example each of tsh and dzh got changed (e.g., parsed as ts + h vs t + sh); both are rare and neither is part of any Georgian transliteration scheme I encountered. I'm not (yet!) trying to support general transliteration from any Latin alphabet into Georgian.)

I refactored the code a few times, testing which (surprisingly expensive!) checks for relevant characters (Georgian, Latin, or Cyrillic), which options to bail early, etc., had the most impact on speed. I think I've got a good final result, but if it is too slow, it'll probably be for Cyrillic queries (which are rare) and we can drop the contextual mappings for т and п (which may not be great anyway) and turn them into single-option mappings.. which will be more efficiently inaccurate and/or inaccurately efficient!