Content translation/Developers/Segmentation
Background information
editSentence segmentation is the splitting of text into individual sentences, titles, captions etc. It is weakly language-specific; a single straightforward mechanism works reasonably well for many languages, but can be improved on with language-specific rules. For ContentTranslation we care most about source language segmentation.
Unicode TR29 gives a default algorithm for plain text: http://www.unicode.org/reports/tr29/#Sentence_Boundaries . For many languages including English, this works well but not perfectly (‘“Hello!” Mr Smith said’ versus ‘“Goodbye!” Mr Smith left.’). CLDR 1.4 is starting to collect language-specific segmentation rules: http://cldr.unicode.org/index/downloads/cldr-1-4/.
Further issues arise when segmenting HTML and other rich text. In particular, text features such as italics can span segments and start/end in the middle of a segment. This means that HTML’s tree structure interacts badly with segmentation: it can be hard to demarcate and manipulate a segment, as it does not exist as an HTML node.
Functional requirements for segmentation
edit- Handle plain text (generic algorithm)
- Handle plain text (language-specific extensions)
- Handle markup that
- can appear in/across segments, e.g. <i>
- causes a segment boundary, e.g. <br>
- wraps segments, e.g. <p> and <h1>
- is not part of any segment, e.g. <table>
- can create a segment within a segment, e.g. <img alt=xxx>
Segmentation in the and content translation workflow
edit- Identify and demarcate segments
- Render segments for the user to see
- Allow user to enter a translated version of the segment
- Reconstruct the translated HTML
Worked example: segments with unbalanced tags
editSample document:
<table>
<tr><td>The participants were <i>“in fear of their lives. Nobody knew what was happening”</i>, even though the researchers were present.</td>
<td>Nobody understood</td>
</tr>
</table>
Segmented structure:
<table>
<tr><td>[The participants were <i>“in fear of their lives.] [Nobody knew what was happening”</i>, even though the researchers were present.]</td>
<td>[Nobody understood]</td>
</tr>
</table>
TODO: Inter-segment space (between the first two segments above) may need localizing
- In Chinese, it would need to become a full-width space
- In other languages, space may need to be removed (or added).
Segments
edit- The participants were
- <i>“in fear of their lives.'''</i><i>'''Nobody knew what was happening”</i>, even though the researchers were present.
- Nobody understood
Note: to render a segment individually, the tags need rebalancing (bold above)
Non-segmented HTML with placeholders:
<table><tr><td>[placeholder1] [placeholder2]</td><td>[placeholder3]</td></tr></table>
Translated segments:
- Roedd y cyfranogwyr yn <i>“ofni am eu bywydau.”<i>
- Er fod yr ymchwilwyr yn bresennol, <i>“Ni wyr neb beth a ddigwyddai”</i>.
- Doedd neb yn deall.
Note: in this case, the italic tags in the second translated segment are no longer at the start.
Reconstructed document:
<table><tr><td>Roedd y cyfranogwyr yn <i>“ofni am eu bywydau.”</i> Er fod yr ymchwilwyr yn bresennol, <i>“Ni wyr neb beth a ddigwyddai”</i>.</td><td>Doedd neb yn deall.</td></tr></table>
The translation has two ... sections, whereas the source only had one. The quotation has been split in half, for grammatical/stylistic reasons.
Different possible approaches to Segmentation
editSegment plain text
editThis is the approach used in most texts that discuss the linguistic aspects of segmentation, e.g. TR29. It is not sufficient to define rich text segmentation fully.
Segment an HTML string
editThis approach involves augmenting plaintext segmentation rules (e.g. TR29) to cope with HTML tags. This is probably not a sensible approach in the long run: a pre-existing HTML parser should be used.
Segment a DOM tree
editApply plaintext segmentation rules (e.g. TR29) within text nodes (counting multiple consecutive text nodes as one). Apply specific rules for each HTML element.
This is the most obvious approach, but working with the segmented structure is challenging: segment start/end pairs may not be balanced at a tree level, and so some segments cannot form branches. Therefore, you cannot wrap each segment in a ..., say, without rewriting the document with re-balanced tags.
Segment from a SAX event stream; output HTML fragments
editApply SAX non-validating parsing on the html fragment and attach listeners to the interesting contexts, i.e.:
- text
- Paragraph, headers
- links
- (may be) alt text or title attributes for images, videos etc
Within the text handling, build on algorithms like TR29 to do language-specific segmentation.
SAX Based Segmenter: https://gerrit.wikimedia.org/r/#/c/117406/
The HTML content(need not be valid or complete) is passed to sax parser. The sax parser used is the parser from node-sax node module. The sax parser is subclassed(inherited) to customize the needs of CX.
CXSegmenter uses CXParser. CXParser inherits SAXParser
The sax parser emits events as and when it see start tags, close tags, text. The received content is reconstructed to form the segmented content. The segmented content will have segment marks. Segment marks are nothing but <span>
nodes with a class and segmenteId. We are interested in sentences and links. So every sentence and link will have this information. Sentences will be surrounded by span and links will have the class and segmentId in the a tag itself. Following example illustrates this.
Source HTML | <p>
<a href="#">Hydrogen</a> is a gas
</p>
|
Segmented content | <p>
<span class="cx-segment" data-segmentid="0">
<a class="cx-link" data-linkid="1" href="#">Hydrogen</a>
is a gas</span>
</p>
|
Identifying links are easy, we just need to listen for “opentag” events with tag as “a”. But identifying sentences require language specific algorithms.
In the case of English, sentence boundaries are often punctuations like period, question mark and exclamation marks. A header like h1,h2 etc are also sentences in the sense of segments A paragraph is a collection of sentences. Punctuations need not mark as end of sentences in special cases like abbreviations. Example: Dr. D. John should not be split at periods. Same is the case of 1000.000 The rules can be sophisticated depending on the level of accuracy we need in sentence segmentation. But it is nearly impossible at least in practice because of multiple reasons
- The content may not follow linguistic correctness about punctuation. It is very well possible that the content editor did not use the punctuations properly. For example, a space followed by period is the correct style of writing English, but we cannot expect that always
- A 100% accurate sentence segmentation system will consider quotes inside the sentences. Eg: He said-”I will be back soon. Please wait”. This is a complex case, because quotes contains full sentences. In practice it is possible to have a link containing multiple sentences. There could be cases like italics spanning from half of one sentence to half of next sentence.
Because of all these, the CX functionality should be fault tolerant about sentence segmentation. The sentences segmentation is used for
- Split the whole article content in a semantic way so that we can pass the parts to external service providers
- Translation memories often works with full sentences
- To create a source-translation pair to be used for machine learning purpose. We need to know how the user translated sentence A to Translation A so that we can better serve another user.
- In the UX, to give a visual sync up for the source-translation pairs
In all of the above cases it is ok to have wrong segments. They does not break any functionality, but reduces the quality of experience in varying levels.
Sentence segmentation rules for languages
editThe theoretical way to do this is to implement TR29 of Unicode standard. At the same time TR29 recommends tailoring of the algorithm to meet the practical pupose.
It is also possible to write language specific rules ourself, just to meet our need. What is attempted in https://gerrit.wikimedia.org/r/#/c/117406/ is a segmentation implementation written by ourself. From testing, it is found that it meets the requirements.
The segmentation rules for other languages can either fallback to English or can have their own subclasses overriding text splitting logic.
https://gerrit.wikimedia.org/r/#/c/117406/ contains CXParserHi which inherits CXParserEn and override ontext method. Hindi sentences are terminated at Devanagari Danda signs
Hindi Segmentation - Regular expression:
text.replace( /([a-zA-Zअ-ह]*)([।|!|?][\s])/g, textSplit );
Segment a linear model
Use a tree transformation to change the tags that can occur within segments into character annotations, to make a more linear representation (like VisualEditor does: http://www.mediawiki.org/wiki/VisualEditor/API/Data_Model/Surface ). Denote segments by ranges in the linear model.
This option is more appealing if we can re-use VisualEditor’s code to transform the HTML to/from the linear model. The resulting linear model text sequences have a structure similar to the following:
[‘H’, ‘e’, ‘l’, ‘l’, ‘o’, ‘ ’, [‘w’, ‘bold’], [‘o’, ‘bold’], [‘r’, ‘bold’], [‘l’, ‘bold’], [‘d’, ‘bold’], ‘.’]
This is similar to a plain text character array, and so is readily amenable to TR29 etc. The main disadvantage is that JavaScript's regular expression engine cannot be used on the array (though it can be used on any character individually, e.g. to test whether it is in the Malayalam block [\u0d00-\u0d7f]).
More related pages
edit- Content translation
- Content translation/Announcement-January2015
- Content translation/Announcements
- Content translation/Announcements/Announcement-July2014
- Content translation/Announcements/Announcement-November2014
- Content translation/Announcements/Announcement-October2014
- Content translation/Announcements/Announcement-October2014/ca
- Content translation/Announcements/Announcement-October2014/cy
- Content translation/Announcements/Announcement-October2014/de
- Content translation/Announcements/Announcement-October2014/en
- Content translation/Announcements/Announcement-October2014/es
- Content translation/Announcements/Announcement-October2014/ja
- Content translation/Announcements/Announcement-October2014/mk
- Content translation/Announcements/Announcement-October2014/nl
- Content translation/Announcements/Announcement-October2014/oc
- Content translation/Announcements/Announcement-October2014/pl
- Content translation/Announcements/Announcement-October2014/pt-br
- Content translation/Announcements/Announcement-October2014/ru
- Content translation/Announcements/Announcement-October2014/scn
- Content translation/Announcements/Announcement-October2014/sd
- Content translation/Announcements/Announcement-October2014/tr
- Content translation/Announcements/Announcement-October2014/vls
- Content translation/Announcements/Announcement-October2014/zh
- Content translation/Apertium/Deployment
- Content translation/CX3
- Content translation/Caching
- Content translation/Community Consultation June 2016
- Content translation/Community Consultation June 2016/Announcement draft
- Content translation/Community Consultation June 2016/Announcement draft/bg
- Content translation/Community Consultation June 2016/Announcement draft/de
- Content translation/Community Consultation June 2016/Announcement draft/en
- Content translation/Community Consultation June 2016/Announcement draft/es
- Content translation/Community Consultation June 2016/Announcement draft/fr
- Content translation/Community Consultation June 2016/Announcement draft/he
- Content translation/Community Consultation June 2016/Announcement draft/kn
- Content translation/Community Consultation June 2016/Announcement draft/nb
- Content translation/Community Consultation June 2016/Announcement draft/pt
- Content translation/Community Consultation June 2016/Announcement draft/ru
- Content translation/Community Consultation June 2016/Announcement draft/tr
- Content translation/Community Consultation June 2016/Announcement draft/uk
- Content translation/Community Consultation June 2016/Announcement draft/zh
- Content translation/Created pages comparison
- Content translation/Deletion statistics comparison
- Content translation/Deletion statistics comparison/tr
- Content translation/Deployment Plan/Archive2
- Content translation/Deployment Plan/Archive3
- Content translation/Deployment Plan/Current Status
- Content translation/Deployment Plan/Current Status/Archive September2014
- Content translation/Deployment Plan/Dev Checklist
- Content translation/Deployments
- Content translation/Deployments/Beta
- Content translation/Deployments/Checklist
- Content translation/Deployments/Deployment Plan
- Content translation/Deployments/Deployment Plan/Archive1
- Content translation/Deployments/Deployment Plan/Archive4
- Content translation/Deployments/Deployment Plan/Current Status/Archive October2014
- Content translation/Deployments/Done
- Content translation/Deployments/How-to
- Content translation/Deployments/How-to/TPA
- Content translation/Deployments/Labs
- Content translation/Developers
- Content translation/Developers/Front end
- Content translation/Developers/GWT-Definitions
- Content translation/Developers/GWT Test Cases
- Content translation/Developers/Markup
- Content translation/Developers/Markup/ar
- Content translation/Developers/Markup/de
- Content translation/Developers/Markup/en
- Content translation/Developers/Markup/fr
- Content translation/Developers/Markup/hi
- Content translation/Developers/Markup/id
- Content translation/Developers/Markup/nl
- Content translation/Developers/Markup/tr
- Content translation/Developers/Progress calculation
- Content translation/Developers/Segmentation
- Content translation/Developers/Setup
- Content translation/Development Plan/Roadmap
- Content translation/Development Plan/Roadmap/CX01Release
- Content translation/Development Plan/Roadmap/CX02Release
- Content translation/Development Plan/Roadmap/CX03Release
- Content translation/Development Plan/Roadmap/MVPRelease
- Content translation/Documentation
- Content translation/Documentation/All pages
- Content translation/Documentation/CX event
- Content translation/Documentation/Comparison with the Translate extension
- Content translation/Documentation/FAQ
- Content translation/Documentation/FAQ/cs
- Content translation/Documentation/FAQ/en
- Content translation/Documentation/FAQ/fr
- Content translation/Documentation/FAQ/hu
- Content translation/Documentation/FAQ/ja
- Content translation/Documentation/FAQ/nl
- Content translation/Documentation/FAQ/pl
- Content translation/Documentation/FAQ/ru
- Content translation/Exploratory Testing Tools
- Content translation/Infographic
- Content translation/Infographic/af
- Content translation/Infographic/ar
- Content translation/Infographic/ast
- Content translation/Infographic/be-tarask
- Content translation/Infographic/bg
- Content translation/Infographic/bn
- Content translation/Infographic/ca
- Content translation/Infographic/cs
- Content translation/Infographic/da
- Content translation/Infographic/de
- Content translation/Infographic/en
- Content translation/Infographic/es
- Content translation/Infographic/fa
- Content translation/Infographic/fr
- Content translation/Infographic/ha
- Content translation/Infographic/hi
- Content translation/Infographic/hu
- Content translation/Infographic/id
- Content translation/Infographic/ja
- Content translation/Infographic/kn
- Content translation/Infographic/ko
- Content translation/Infographic/mni
- Content translation/Infographic/nl
- Content translation/Infographic/oc
- Content translation/Infographic/pl
- Content translation/Infographic/pt
- Content translation/Infographic/pt-br
- Content translation/Infographic/ru
- Content translation/Infographic/scn
- Content translation/Infographic/sd
- Content translation/Infographic/th
- Content translation/Infographic/tr
- Content translation/Infographic/vi
- Content translation/Infographic/zh
- Content translation/Language anomalies
- Content translation/Languages
- Content translation/Machine Translation/Apertium
- Content translation/Machine Translation/Apertium/Beta
- Content translation/Machine Translation/Apertium/Debug
- Content translation/Machine Translation/Apertium/Packaging
- Content translation/Machine Translation/Apertium/Packaging/binNMU
- Content translation/Machine Translation/Apertium/Service
- Content translation/Machine Translation/Apertium/Update
- Content translation/Machine Translation/Configuration
- Content translation/Machine Translation/Google Translate
- Content translation/Machine Translation/Google Translate/anp
- Content translation/Machine Translation/Google Translate/bn
- Content translation/Machine Translation/Google Translate/de
- Content translation/Machine Translation/Google Translate/en
- Content translation/Machine Translation/Google Translate/es
- Content translation/Machine Translation/Google Translate/fr
- Content translation/Machine Translation/Google Translate/gu
- Content translation/Machine Translation/Google Translate/hu
- Content translation/Machine Translation/Google Translate/id
- Content translation/Machine Translation/Google Translate/ja
- Content translation/Machine Translation/Google Translate/ko
- Content translation/Machine Translation/Google Translate/nl
- Content translation/Machine Translation/Google Translate/pl
- Content translation/Machine Translation/Google Translate/pt
- Content translation/Machine Translation/Google Translate/ru
- Content translation/Machine Translation/Google Translate/tr
- Content translation/Machine Translation/Google Translate/vi
- Content translation/Machine Translation/Google Translate/zh
- Content translation/Machine Translation/LingoCloud
- Content translation/Machine Translation/MT Clients
- Content translation/Machine Translation/MT Clients/ar
- Content translation/Machine Translation/MT Clients/de
- Content translation/Machine Translation/MT Clients/en
- Content translation/Machine Translation/MT Clients/fr
- Content translation/Machine Translation/MT Clients/hu
- Content translation/Machine Translation/MT Clients/id
- Content translation/Machine Translation/MT Clients/ja
- Content translation/Machine Translation/MT Clients/luz
- Content translation/Machine Translation/MT Clients/nl
- Content translation/Machine Translation/MT Clients/oc
- Content translation/Machine Translation/MT Clients/pl
- Content translation/Machine Translation/MT Clients/ru
- Content translation/Machine Translation/MT Clients/tr
- Content translation/Machine Translation/Matxin
- Content translation/Machine Translation/Matxin/cs
- Content translation/Machine Translation/Matxin/da
- Content translation/Machine Translation/Matxin/de
- Content translation/Machine Translation/Matxin/en
- Content translation/Machine Translation/Matxin/es
- Content translation/Machine Translation/Matxin/fr
- Content translation/Machine Translation/Matxin/hu
- Content translation/Machine Translation/Matxin/id
- Content translation/Machine Translation/Matxin/ja
- Content translation/Machine Translation/Matxin/ko
- Content translation/Machine Translation/Matxin/nl
- Content translation/Machine Translation/Matxin/pl
- Content translation/Machine Translation/Matxin/ru
- Content translation/Machine Translation/Matxin/tr
- Content translation/Machine Translation/Matxin/zh
- Content translation/Machine Translation/MinT
- Content translation/Machine Translation/MinT/ar
- Content translation/Machine Translation/MinT/azb
- Content translation/Machine Translation/MinT/bn
- Content translation/Machine Translation/MinT/cs
- Content translation/Machine Translation/MinT/de
- Content translation/Machine Translation/MinT/en
- Content translation/Machine Translation/MinT/fr
- Content translation/Machine Translation/MinT/id
- Content translation/Machine Translation/MinT/it
- Content translation/Machine Translation/MinT/ja
- Content translation/Machine Translation/MinT/ky
- Content translation/Machine Translation/MinT/mni
- Content translation/Machine Translation/MinT/nl
- Content translation/Machine Translation/MinT/ru
- Content translation/Machine Translation/MinT/sd
- Content translation/Machine Translation/MinT/uk
- Content translation/Machine Translation/MinT/ur
- Content translation/Machine Translation/MinT/vi
- Content translation/Machine Translation/MinT/zh
- Content translation/Machine Translation/NLLB-200
- Content translation/Machine Translation/Yandex
- Content translation/Machine Translation/Yandex/cs
- Content translation/Machine Translation/Yandex/de
- Content translation/Machine Translation/Yandex/en
- Content translation/Machine Translation/Yandex/es
- Content translation/Machine Translation/Yandex/fr
- Content translation/Machine Translation/Yandex/hu
- Content translation/Machine Translation/Yandex/id
- Content translation/Machine Translation/Yandex/ja
- Content translation/Machine Translation/Yandex/kn
- Content translation/Machine Translation/Yandex/nl
- Content translation/Machine Translation/Yandex/pt
- Content translation/Machine Translation/Yandex/ru
- Content translation/Machine Translation/Yandex/th
- Content translation/Machine Translation/Yandex/tr
- Content translation/Machine Translation/Yandex/zh
- Content translation/Machine Translation/Youdao
- Content translation/Machine Translation/Youdao/cs
- Content translation/Machine Translation/Youdao/de
- Content translation/Machine Translation/Youdao/en
- Content translation/Machine Translation/Youdao/es
- Content translation/Machine Translation/Youdao/fr
- Content translation/Machine Translation/Youdao/hu
- Content translation/Machine Translation/Youdao/id
- Content translation/Machine Translation/Youdao/ja
- Content translation/Machine Translation/Youdao/nl
- Content translation/Machine Translation/Youdao/pl
- Content translation/Machine Translation/Youdao/sw
- Content translation/Machine Translation/Youdao/tr
- Content translation/Machine Translation/Youdao/zh
- Content translation/Product Definition
- Content translation/Product Definition/AbuseFilter
- Content translation/Product Definition/AbuseFilter/de
- Content translation/Product Definition/AbuseFilter/en
- Content translation/Product Definition/AbuseFilter/es
- Content translation/Product Definition/AbuseFilter/fa
- Content translation/Product Definition/AbuseFilter/fr
- Content translation/Product Definition/AbuseFilter/id
- Content translation/Product Definition/AbuseFilter/ja
- Content translation/Product Definition/AbuseFilter/lo
- Content translation/Product Definition/AbuseFilter/nl
- Content translation/Product Definition/AbuseFilter/pl
- Content translation/Product Definition/AbuseFilter/pt
- Content translation/Product Definition/AbuseFilter/pt-br
- Content translation/Product Definition/AbuseFilter/ru
- Content translation/Product Definition/AbuseFilter/th
- Content translation/Product Definition/AbuseFilter/tr
- Content translation/Product Definition/AbuseFilter/zh
- Content translation/Product Definition/Abuse prevention
- Content translation/Product Definition/Campaigns
- Content translation/Product Definition/Database
- Content translation/Product Definition/Dictionaries
- Content translation/Product Definition/Entry points
- Content translation/Product Definition/Images
- Content translation/Product Definition/LinearDoc
- Content translation/Product Definition/MT API Proof of Concept
- Content translation/Product Definition/Mobile exploration
- Content translation/Product Definition/Performance
- Content translation/Product Definition/Round Trip
- Content translation/Product Definition/Section alignment
- Content translation/Product Definition/Templates
- Content translation/Product Definition/Translation view
- Content translation/Product Definition/analytics
- Content translation/Product Definition/analytics/ar
- Content translation/Product Definition/analytics/de
- Content translation/Product Definition/analytics/en
- Content translation/Product Definition/analytics/fr
- Content translation/Product Definition/analytics/id
- Content translation/Product Definition/analytics/ja
- Content translation/Product Definition/analytics/nl
- Content translation/Product Definition/analytics/pl
- Content translation/Product Definition/analytics/tr
- Content translation/Product Definition/analytics/zh
- Content translation/Published pages
- Content translation/Published translations
- Content translation/Published translations/en