Content translation/Product Definition/Performance

In general, Content translation does not do any heavy processing internally. It heavily depend on Parsoid and Machine translation engines hosted outside like Apertium. Hence CX performance is mostly dependent on the performance of these systems. Most of the time the user spend will be on translating the content in the translation editor provided.

Server side processing

Saving draft translation and publishing the content by converting to wikitext with the help of Parsoid are the only two server side actions by the CX extensions server side. Saving the draft does not have any processing involved other than insert or replace actions in CX database tables. Publishing involves an API call to parsoid service instance to convert the HTML content to wikitext and another call to the edit API call to save the content. At user interface the potential delay in publishing is handled using waiting indicators when Publish or save button is pressed

cxserver

cxserver is a nodejs server and it act as a webservice wrapper, content preprocessor for source article, postprocessor for MT result, API bridge for third party MT instances.

It follows similar architecture of parsoid by creating a cluster of worker threads. It has node-express based web APIs.

For the source article, once the source articles HTML is received, cxserver does segmentation on the content to identify section and sentence boundaries. This is basically a DOM parsing and converting the DOM to a linear document model.

Machine Translation

Machine translation is happening in external applications and CX is a consumer using asynchrounous web API requests.

Apertium machine translation is provided by Apertium-APY, a python tornado based webservice wrapper on top of Apertium. It provides the concurrency and threading to support multiple machine translation requests. Wikimedia hosts an apertium-apy instance in a dedicated server. The network, processor load, memory usage were monitored for a week time to ensure nothing wrong happens there. But this monitoring will repeat when CX is opened up for wider audience.

Apertium machine translation is a processor intensive operation. It performs well with multi threaded machines. Internally apertium is a pipeline of multiple unix processes. A typical large sized section(MT is done per section in MT. A paragraph is a section) is targeted to get translated below one second server time. This potential delay is addressed in UX with the help of loading indicators and MT prefetching. When source article is loaded, MT request for next section will be fired, so that when user request MT, we can provide it early. Similarly when a section is translation, its header is also translated. MT results are cached in client side between translators edits- at any time translator can restore the MT output to the translated section.

If at all we support external MT providers like google, yandex or bing, we will be dependent on their processing time and network latency. MT prefetching will have more important role there.

Caching

As per architecture and deployment plan, Varnish caching will be configured in front for cxserver to cache api requests to cxserver.

The API requests to source and target wikis will also be cached. For example, the link details will be requested only once and will be cached per translation section. All tools modules - Dictionary, MT, Links follow the same pattern of caching the api requests to external wikis.

Client side resources loading

The extension does not add any JavaScript or CSS to Wiki pages, except the entry point module-which checks if the article is missing an inter-language link in a potential language that user is interested. If missing, the module lazy loads a translation selector when clicked on the red interlanguage link it created.

The actual translation dashboard, tools etc are loaded only for Special:CX page.

Translation tools in user interface

The translation tools are lazy loaded in parallel with article loading so that we can present the Special:CX as fast as possible