Content translation/Product Definition/Performance

In general, Content translation does not do any heavy processing internally. It heavily depend on Parsoid and Machine translation engines hosted outside like Apertium. Hence CX performance is mostly dependent on the performance of these systems. Most of the time the user spend will be on translating the content in the translation editor provided.

Server side processing
Saving draft translation and publishing the content by converting to wikitext with the help of Parsoid are the only two server side actions by the CX extensions server side. Saving the draft does not have any processing involved other than insert or replace actions in CX database tables. Publishing involves an API call to parsoid service instance to convert the HTML content to wikitext and edit api call to save the content. At user interface the potential delay in publishing is handled using waiting indicators when Publish or save button is pressed

cxserver
cxserver is a nodejs server and it act as a webservice wrapper, content preprocessor for source article, postprocessor for MT result, API bridge for third party MT instances.

It follows similar architecture of parsoid by creating a cluster of worker threads. It has node-express based web APIs.

For the source article, once the source articles HTML is recieved, cxserver does segmentation on the content to identify section and sentence boundaries. This is basically a DOM parsing and converting the DOM to a linear document model.

Machine Translation
Machine translation is happening in external applications and CX is a consumer using asynchrounous web API requests.

Apertium machine translation is provided by Apertium-APY, a python tornado based webservice wrapper on top of Apertium. It provides the concurrency and threading to support multiple machine translation requests. Wikimedia hosts an apertium-apy instance in a dedicated server. The network, processor load, memory usage were monitored for a week time to ensure nothing wrong happens there. But this monitoring will repeat when CX is opened up for wider audience.

Apertium machine translation is a processor intensive operation. It performs well with multi threaded machines. Internally apertium is a pipeline of multiple unix processes. A typical large sized section(MT is done per section in MT. A paragraph is a section) is targetted to get translated below one second server time. This potential delay is addressed in UX with the help of loading indicators and MT prefetching. When source article is loaded, MT request for next section will be fired, so that when user request MT, we can provide it early. Similarly when a section is translation, its header is also translated. MT results are cached in client side between translators edits- at any time translator can restore the MT output to the translated section.

If at all we support external MT providers like google, yandex or bing, we will be dependent on their processing time and network latency. MT prefetching will have more important role there.