VisualEditor/Design/Bidirectional text requirements

UNFINISHED

This is a very, very rough draft. Any comment is welcome.

Preamble
Since MediaWiki in particular and Wikimedia projects in particular are used in right-to-left languages, such as Arabic, Pashto, Persian and Hebrew, the projected visual editor must support writing and displaying them easily.

"Easily" in this context means that any user must be able to write mixed right-to-left and left-to-right text as easily as he would do it with pen and paper and see the results while editing.

Although this is true for any text in right-to-left languages, proper support for mixing right-to-left and left-to-right is of particular importance for Wikimedia projects, which are heavily multilingual and frequently cite foreign names.

This document tries to be more functional - to describe what the software should do, rather than technical - to describe how it should be implemented. Some implementation suggestions may creep into the document nevertheless.

The direction property
It must be possible to apply the direction property at several levels:
 * the level of a content page
 * a block-level element (usually a paragraph, but possibly also a table cell, and possible a template etc.)
 * an inline element (effectively , but probably set using a template).

Page-level
The page-level direction property is inherited from the direction of the page content language (see Title::getPageLanguage), which is by default mostly the wiki content language but can be overridden by using the hook. As an example, this causes Donor_policy/ar to be displayed correctly RTL because the Translate extension makes use of this hook.

However, it is not possible to override the language through the wiki itself, which is usually addressed by manually applying  to the whole page. (See bug 28970.) Having the ability to set it per-page through the wiki is particularly useful for current Wikimedia sites like Meta, Commons, Outreach, Strategy etc., for "embassy" pages in various projects, and for other purposes.

Thoughts
UI idea: this can be an option selectable from a dropdown list placed somewhere in "page properties" in the editing mode (is there such a thing now?)

Another barely related idea: everybody hates that stuff like categories, interlanguage links, DEFAULTSORT etc. are part of the page source. These can also be changed in something like "page properties".

Implementation: This property should probably be stored in the database as a column in the page table. If it is defined, 'mw-content-ltr' or 'mw-content-rtl' should be used accordingly.

Language: It is not directly related to directionality, but it may be useful to set the language (lang attribute) at the same levels. Direction may also be set automatically according to the language.

Paragraph-level
The paragraph-level direction property is inherited from the direction of the page, but it must be possible to override it.

In existing software there are usually three ways to do it:
 * 1) Pressing Ctrl-Shift (see ).
 * 2) Pressing a button in the toolbar. It usually looks more or less like this: ¶◄ (see screenshot).
 * 3) Selecting an option in a "Format paragraph" dialog (see screenshot).

As far as the internal implementation goes, adding a dir attribute to the  element should be enough.

Several paragraphs at once
Pending question: Should the setting of direction for several paragraphs at once exist? Something like: It would be comparable to a section in OpenDocument.

For display purposes setting direction per paragraph is enough, but applying this to several paragraphs may be useful for semantic markup, for example to indicate a long quote.

Inline
This is needed for writing a string with directionality X inside a paragraph with directionality Y.

Common scenarios:

Most likely this would be an HTML , but it can also be (in HTML5),. Since the Visual editor is supposed to be able to handle templates, this will require very little coding. However, this would not be completely robust, because the templates must be actually created manually after installing MediaWiki with the Visual editor, while it would be better to have it installed automatically. A structured way to manage templates, for example by global transclusion or a "standard template library" - ideas that are occasionally discussed at wikitech-l - would make this more robust.

Direction setting
The usual keyboard shortcut in Windows and Linux for setting direction is Ctrl-Shift. Left Ctrl-Shift sets the direction to LTR and right Ctrl-Shift sets the direction to LTR.

In rich text editors such as LibreOffice and Microsoft Word it sets the direction of the current paragraph or of all the selected paragraphs. In plain text editors such as Notepad and browser textareas it sets the direction of the whole text, as there's no robust way to set the direction of a single paragraph in plain text (RLE/PDF is not quite robust).

Notable exceptions:
 * Firefox: Ctrl-Shift does nothing. Ctrl-Shift-X, no matter at which side, switches the current direction (bidi.browser.ui must be set to true so this would actually work). It seems to be so because DOM doesn't define precisely what to do with Ctrl-Shift by itself. There's a very old request to change it to Ctrl-Shift (Mozilla Bug 98160 - Replace Accel+Shift+X (Switches text direction) with more intuitive keyboard shortcuts: Ctrl+Left/Right Shift on Windows).
 * Notepad++: Ctrl-Alt-L, Ctrl-Alt-R.
 * The commonly used rich text editor in GMail, which is actually a complete HTML iframe, doesn't provide any way to change paragraph direction with the keyboard. It can be changed using a toolbar button (which only appears if the interface language in RTL).
 * In supporting text editors on Mac OS X the shortcut is Ctrl-Command-←/→. (Must be turned on in system keyboard settings.)

Since Ctrl-Shift is the common shortcut, the MediaWiki visual editor should use it in a fashion similar to LibreOffice and Word - to set the direction of the current selected paragraphs. It should also be possible to interact with the client system as much as possible and allow the user to use the shortcut to which she is accustomed in the software.

Important question: Should Ctrl-Shift be used to set the direction of a selected inline element? The intuition says that it should only work for paragraphs, but maybe user testing will prove that users like it. (And maybe Ctrl-Shift-X should be used for that?..)

Movement
It must be possible to use the right and left arrow keys intuitively. While the cursor is in words in the main language of a paragraph, the cursor should move in the direction of the arrow. The hardest decision to make is how should the cursor move in a text in the other direction, for example in a Hebrew word within an English paragraph. In different programs it works differently:

Firefox notes:
 * 1) In Firefox this could be changed by setting bidi.controlstextmode in about:config, but this was disabled.
 * 2) The movement is visual also when using the visible caret (enabled in the accessibility settings).

Text selection always works in the logical order, because visual selection will cause non-contiguous text to be selected.

The movement order should probably be logical, since it is consistent with the selection order and more common (and probably easier to implement, too).

Another consideration is Ctrl-Arrow, which makes the cursor jump from word to word. It should be obvious that its direction must be identical to that of the regular movement arrows, but in Chrome it's implemented incorrectly (Chromium Issue 10741: Ctrl + Right/Left combination moves cursor to opposite direction between words in RTL text box).

(Also Ctrl-Shift-Arrow, Ctrl-Del etc.)

Values for direction
The possible values for direction in HTML are "ltr" and "rtl". HTML5 adds "auto", which, according to the specification, applies the direction of the first character with strong directionality the the whole element. It already works in Webkit (and even used in some elements that MediaWiki outputs) and will probably work in Firefox in mid-2012. It doesn't work in IE9.

HTML4 prohibits automatically setting the direction according to language. In standard HTML the default direction is "ltr" and it will remain so even if the language is explicitly defined to a code of an RTL language - is "ltr". HTML5 is not different in this regard. It is not clear why is this so, but the probable reason is that there are thousand of language codes, their relationships with scripts is not completely defined and so it's impossible to rely upon webmasters to use correct codes and upon browsers to determine correct direction.

Despite this, there is no problem for MediaWiki to determine the direction according to the language and write it explicitly to the output HTML. In fact, this is done already, especially since 1.18. MediaWiki also strives to use standard codes (although there are deviations from this rule in the current projects). Furthermore, the Visual Editor should encourage specifying the language using the lang attribute, by making it possible and easy (see Visual editor/Internationalization requirements). The Visual Editor should, therefore, provide the following values for direction:
 * ltr: dir="ltr"
 * rtl: dir="rtl"
 * auto: dir="auto" (?)
 * by language: let MediaWiki determine the direction according to the language tag.

It's possible that "by language" should be the default value, but this requires some thought.

Isolation
Bidirectional isolation prevents the directionality of an element from interfering with the surrounding content. This often happens, for example, when an English name appears before a person's birth date in an RTL wiki and when a footnote number appears after a Hebrew word in an LTR wiki.

In Wikimedia projects the foreign text is usually not related to the surrounding content, and it is therefore worth considering making an element bidi-isolated by default (but when - when the language is different? when the directionality is different?).

This can be done both by putting the content in a element or by applying the CSS rule "unicode-bidi: isolate".

Enabling and disabling the controls
It should be possible to disable the bidirectional controls - buttons, dropdowns, dialogs -, as well as the relevant keyboard shortcuts (Ctrl-Shift?). It is likely that many people will never use them. For example, someone who writes in an LTR project and never deals with Middle Eastern themes will probably never use them, so they may be unnecessary clutter in the toolbar.

LibreOffice, Microsoft Word, GMail and Google Docs only enable these bidirectional controls if the interface language is RTL or if the user specifically requested for it. MediaWIki Visual editor can take a similar path: This should be one of editing preferences, enabled by default in RTL wikis and for users who chose an RTL language in the preferences. It must be possible to enable it in any setup (GMail is buggy in this regard last time i checked).

Usage of control characters
Several transparent control characters are used for controlling directionality:
 * RLM / LRM - transparent characters with strong directionality. It's easy to understand them as Hebrew and Latin letters, respectively, but invisible. They are used for isolating inline pieces of text with ambiguous directionality, for example a Latin letter from a following number in a right-to-left paragraph. Equivalent to the HTML entities &amp;rlm; and &amp;lrm;.
 * RLE / LRE - start an embedded inline block of right-to-left or left-to-right text that runs until a PDF character. Similar to /.
 * RLO / LRO - starts a section in which the Unicode bidirectional algorithm doesn't apply. That is, Hebrew letters will appear left-to-right. Equivalent to the HTML tag.
 * PDF - pop directional formatting. Ends a block of text started by RLE, LRE, RLO or LRO.

There's no practical use for RLE, LRE, RLO, LRO and PDF in text that can be marked up with HTML.

RLM and LRM are currently frequently used in MediaWiki, as there is no other common way to isolate pieces of text with ambiguous directionality. They are often inserted as a template (such as w:he:Template:כ). Implementation of the HTML5 tag may diminish the need for them, but currently they are needed.

Other considerations

 * Offline and PDF export: These features shouldn't be directly affected by this if parsing and rendering stays the same, but they must be tested anyway.
 * Web fonts.

Current bidirectional features in MediaWiki

 * Directionality support
 * Existing and resolved bugs related to right-to-left text

Related standards

 * Unicode Bidirectional Algorithm
 * Language information and text direction - Existing HTML4 standard.
 * Additional Requirements for Bidi in HTML - Prospective bidirectional features in HTML5.
 * Open Document Format for Office Applications (OpenDocument) v1.1 - a.k.a. ODF and ODT, used in LibreOffice, OpenOffice and other applications.
 * Bidi-related about:config settings in Mozilla