Enabling Sinhala in GNU/Linux HOWTO

3.5.1.1. Xorg 6.9+

The aforementioned layout is already included in Xorg 6.9 and above and distributions that ship with xkeyboard-config 0.6 and above.

Debian Etch:

/usr/share/X11/xkb/symbols/lk

Ubuntu 5.10:

/etc/X11/xkb/symbols/lk

Fedora Core 5:

/usr/share/X11/xkb/symbols/lk

The latest version of the keyboard layout can be downloaded from CVS:

• lk

Read the comments in the lk file to see how to create rakaaranshaya, yansaya and repaya.

The window manager should come with a program which allows the user to choose multiple keyboard layouts.

In the example below I have chosen the SHIFT keys to switch between the Sinhala phonetic layout and the US QWERTY layout. Hold one of the SHIFT keys down and then press the other SHIFT key, this should toggle between the layouts.

Using the GUI in GNOME:

1. Run:

   gnome-keyboard-properties

2. Choose the “Layout” tab and click on the “Add” button. This will open a new window which contains a list of layouts ordered by country.

3. Scroll down the list till you find “Sri Lanka” and then highlight it by clicking on it. The Sinhala layout is the default in the Sri Lanka layouts file, so you do NOT need to click the expand triangle icon. Then press “OK”.

4. Choose the “Layout Options” tab and click on the text “Group Shift/Lock behavior”. A list will expand below this text

5. Scroll down the list till you find the text “Both Shift keys together change group”. Click on the corresponding checkbox.

6. If you wish to use an LED to indicate the toggling of keyboard layouts, click on the text "Use keyboard LED to show alternative group". A list will expand below this text

7. Scroll down the list till you find the text “ScrollLock LED shows alternative group”.

Using the command line in X:

1. In an xterm do:

   setxkbmap -layout "us,lk" -option "grp:shifts_toggle,grp_led:scroll"

Alternately, you can directly modify /etc/X11/xorg.conf:

1. To add the new lk keyboard layout, look for this line:

   Section "InputDevice"

   There will probably be two such lines, one for the keyboard and another for the mouse. Go to the keyboard related line.

2. Then add 'lk' to a line that looks like:

   Option "XkbLayout" "us,lk"

3. Also add a mechanism to switch between 'us' and 'lk' and indicate which LED should be used:

   Option "XkbOptions" "grp:shifts_toggle,grp_led:scroll"

4. If asked by the window manager, reset keyboard defaults to the X defaults.

3.5.1.2. XFree86 4.3+ or Xorg 6.7+

In the example below I have chosen the ALT keys to switch between the Sinhala phonetic layout and the US QWERTY layout. Hold one of the ALT keys down and then press the other ALT key, this should toggle between the layouts.

1. Download the keyboard layout from:

   • sin

2. Copy the keyboard layout to:

   /etc/X11/xkb/symbols/pc/

3. There are two options:

   1. In an xterm do:

      setxkbmap -layout "sin,us" -option "grp:alts_toggle,grp_led:scroll"

   2. Or alternately, edit the /etc/X11/XF86Config or /etc/X11/xorg.conf file.

      1. To add the new 'sin' keyboard layout, look for this line:

         Section "InputDevice"

         There will probably be two such lines, one for the keyboard and another for the mouse. Go to the keyboard related line.

      2. Then add 'sin' to a line that looks like:

         Option "XkbLayout" "sin,us"

      3. Also add a mechanism to switch between 'us' and 'sin' and indicate which LED should be used:

         Option "XkbOptions" "grp:alts_toggle,grp_led:scroll"

      4. If asked by the window manager, reset keyboard defaults to the X defaults.

4.2.2.1. Patches:

• ICU 2.6 Sinhala patch http://cvs.savannah.nongnu.org/viewcvs/*checkout*/sinhala/patches/icu-2.6-oo-sinhala1.patch?root=sinhala

• ICU 3.4 Sinhala patch http://cvs.savannah.nongnu.org/viewcvs/*checkout*/sinhala/patches/icu-3.4-sinhala1.patch?root=sinhala

• ICU 3.6 Sinhala patch http://cvs.savannah.nongnu.org/viewcvs/*checkout*/sinhala/patches/icu-3.6-sinhala1.patch?root=sinhala

4.2.2.2. Split dependent vowel modifier (diga o) issue

There's an issue with U+0DDD (dependent vowel diga o) that can cause Open Office to crash. Opening this text file will crash Open Office and ICU 3.6:

• http://cvs.savannah.nongnu.org/viewcvs/*checkout*/sinhala/patches/icu-crash-testcase-dv-oo.txt?root=sinhala

The worstCaseExpansion for Sinhala was set to 3 when it should have been set to 4. The dependent vowel 'oo' (U+0DDD) consists of (kombuva)(dotted-circle)(aela-pilla)(al-lakuna) which are 4 glyphs. As a result of the worstCaseExpansion being 3, memory was probably being allocated for 3 glyphs when memory was required for 4 glyphs. The actual crash occurred when unallocated memory was being freed.

Caolan McNamara also found and fixed this bug. [10]

4.2.2.3. Call Tree

1. source/layoutex/ParagraphLayout.cpp

   ParagraphLayout::ParagraphLayout(const LEUnicode chars[], le_int32 count, const FontRuns *fontRuns, const ValueRuns *levelRuns, const ValueRuns *scriptRuns, const LocaleRuns *localeRuns, UBiDiLevel paragraphLevel, le_bool vertical, LEErrorCode &status)

   1. source/layout/LayoutEngine.cpp:

      LayoutEngine *LayoutEngine::layoutEngineFactory(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, LEErrorCode &success)

   2. LayoutEngine *LayoutEngine::layoutEngineFactory(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, le_int32 typoFlags, LEErrorCode &success)

      • IndicOpenTypeLayoutEngine::IndicOpenTypeLayoutEngine(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, le_int32 typoFlags, const GlyphSubstitutionTableHeader *gsubTable)

      • IndicOpenTypeLayoutEngine::IndicOpenTypeLayoutEngine(const LEFontInstance *fontInstance, le_int32 scriptCode, le_int32 languageCode, le_int32 typoFlags)

2. source/layout/LayoutEngine.cpp

   le_int32 LayoutEngine::layoutChars(const LEUnicode chars[], le_int32 offset, le_int32 count, le_int32 max, le_bool rightToLeft, float x, float y, LEErrorCode &success)

   1. le_int32 LayoutEngine::computeGlyphs(const LEUnicode chars[], le_int32 offset, le_int32 count, le_int32 max, le_bool rightToLeft, LEGlyphStorage &glyphStorage, LEErrorCode &success)

      1. source/layout/IndicLayoutEngine.cpp

         le_int32 IndicOpenTypeLayoutEngine::characterProcessing(const LEUnicode chars[], le_int32 offset, le_int32 count, le_int32 max, le_bool rightToLeft, LEUnicode *&outChars, LEGlyphStorage &glyphStorage, LEErrorCode &success)

         1. source/layout/IndicReordering.cpp

            le_int32 IndicReordering::reorder(const LEUnicode *chars, le_int32 charCount, le_int32 scriptCode, LEUnicode *outChars, LEGlyphStorage &glyphStorage, MPreFixups **outMPreFixups)

3. engine->getGlyphs(fStyleRunInfo[run].glyphs, layoutStatus);

4. engine->getGlyphPositions(fStyleRunInfo[run].positions, layoutStatus);

5. engine->getCharIndices(&fGlyphToCharMap[glyphBase], runStart, layoutStatus);

4.3.2.1. Ubuntu 5.10

Ubuntu 5.10 enabled Pango by default. Have a look at:

/usr/bin/mozilla-firefox 

which contains the code:

##
## Set MOZ_ENABLE_PANGO
##
MOZ_ENABLE_PANGO=1
export MOZ_ENABLE_PANGO

4.3.2.2. Ubuntu 6.06

On the other hand Ubuntu 6.06, decided to disable Pango in Firefox by default except for a pre-determined list of locales. The extensive discussion can be found here:

• https://launchpad.net/distros/ubuntu/+source/firefox/+bug/32561

Have a look at:

/usr/bin/mozilla-firefox 

which contains the code:

if [ "x${MOZ_DISABLE_PANGO}" = x ]; then
    if egrep '^(bn|gu|hi|kn|ml|mr|ne|pa|ta|te)_' \
        /var/lib/locales/supported.d/*[^~] >/dev/null 2>&1; then
        MOZ_DISABLE_PANGO=0
    else
        MOZ_DISABLE_PANGO=1
    fi
    export MOZ_DISABLE_PANGO
fi
if [ "x${MOZ_DISABLE_PANGO}" = x0 ]; then
    unset MOZ_DISABLE_PANGO
fi

This means that Ubuntu 6.06 users that need Pango enabled in Firefox need to set an environment variable

MOZ_DISABLE_PANGO=0

You can see the difference by running Firefox at the command line like so:

# MOZ_DISABLE_PANGO=0 mozilla-firefox

4.3.3.1. Fedora Core 4 [11]

##
## Set MOZ_ENABLE_PANGO is no longer used because Pango is enabled by default
## you may use MOZ_DISABLE_PANGO=1 to force disabling of pango
##
#MOZ_DISABLE_PANGO=1
#export MOZ_DISABLE_PANGO

4.3.3.2. Fedora Core 5 [12]

##
## In order to better support certain scripts (such as Indic and some CJK 
## scripts), Fedora builds its Firefox, with permission from the Mozilla 
## Corporation, with the Pango system as its text renderer.  This change 
## is known to break rendering of MathML, and may negatively impact 
## performance on some pages.  To disable the use of Pango, set 
## MOZ_DISABLE_PANGO=1 in your environment before launching Firefox.
##
#
# MOZ_DISABLE_PANGO=1
# export MOZ_DISABLE_PANGO
#

4.6.1.1. Terminology

• ci = case insensitive

• cs = case sensitive

• bin = binary

4.6.1.2. Useful Commands

• SHOW CHARACTER SET;

• SHOW COLLATION;

• SHOW COLLATION like 'ucs%';

• SHOW COLLATION like 'utf8%';

• SET NAMES 'utf8'; // after connecting to server if the server has NOT set 'skip-character-set-client-handshake'

• SHOW CREATE TABLE <table-name>

• SHOW VARIABLES;

• \s

4.6.1.3. Setup MySQL Server

Edit the file /etc/mysql/my.cnf and add to the [mysqld] section:

• default-character_set=utf8

• skip-character-set-client-handshake

This is done to ensure that UTF-8 is the default encoding for the server and client.

4.6.1.4. Files requiring modification

• mysql/config/ac-macros/character_sets.m4

• mysql/mysys/charset-def.c

• mysql/strings/ctype-uca.c

• mysql/configure (generated)

4.6.1.5. Code

• mysql/strings/ctype-uca.c

/*
  Collation language is implemented according to
  subset of ICU Collation Customization (tailorings):
  http://icu.sourceforge.net/userguide/Collate_Customization.html
  
  Collation language elements:
  Delimiters:
    space   - skipped
  
  <char> :=  A-Z | a-z | \uXXXX
  
  Shift command:
    <shift>  := &       - reset at this letter. 
  
  Diff command:
    <d1> :=  <     - Identifies a primary difference.
    <d2> :=  <<    - Identifies a secondary difference.
    <d3> := <<<    - Idenfifies a tertiary difference.
  
  
  Collation rules:
    <ruleset> :=  <rule>  { <ruleset> }
    
    <rule> :=   <d1>    <string>
              | <d2>    <string>
              | <d3>    <string>
              | <shift> <char>
    
    <string> := <char> [ <string> ]
  An example, Polish collation:
  
    &A < \u0105 <<< \u0104
    &C < \u0107 <<< \u0106
    &E < \u0119 <<< \u0118
    &L < \u0142 <<< \u0141
    &N < \u0144 <<< \u0143
    &O < \u00F3 <<< \u00D3
    &S < \u015B <<< \u015A
    &Z < \u017A <<< \u017B    
*/

• mysql/include/m_ctype.h

typedef struct charset_info_st
{
  uint      number;
  uint      primary_number;
  uint      binary_number;
  uint      state;
  const char *csname;
  const char *name;
  const char *comment;
  const char *tailoring;
  uchar    *ctype;
  uchar    *to_lower;
  uchar    *to_upper;
  uchar    *sort_order;
  uint16   *contractions;
  uint16   **sort_order_big;
  uint16      *tab_to_uni;
  MY_UNI_IDX  *tab_from_uni;
  MY_UNICASE_INFO **caseinfo;
  uchar     *state_map;
  uchar     *ident_map;
  uint      strxfrm_multiply;
  uchar     caseup_multiply;
  uchar     casedn_multiply;
  uint      mbminlen;
  uint      mbmaxlen;
  uint16    min_sort_char;
  uint16    max_sort_char; /* For LIKE optimization */
  uchar     pad_char;
  my_bool   escape_with_backslash_is_dangerous;
  
  MY_CHARSET_HANDLER *cset;
  MY_COLLATION_HANDLER *coll;
  
} CHARSET_INFO;

• mysys/charset.c: CHARSET_INFO *all_charsets[256]

• mysys/charset.c

  init_available_charsets()

  • mysys/charset-def.c

    init_compiled_charsets()

    • mysys/charset.c

      add_compiled_collation()

  • init_state_maps()

• mysys/charset.c

  get_charset_by_name()

  • get_collation_number()

    • get_collation_number_internal()

  • get_internal_charset()

• strings/ctype-uca.c

  create_tailoring()

4.6.1.6. Testing

1. Test Swedish (MySQL is a Swedish company) collation algorithm:

   http://en.wikipedia.org/wiki/Swedish_alphabet

   CREATE TABLE ts (
           id SERIAL PRIMARY KEY,
           letter VARCHAR(10) NOT NULL
   ) CHARACTER SET utf8;

   Running this collation handler:

   SELECT * FROM ts ORDER BY letter COLLATE utf8_swedish_ci;

   results in non-English alphabet characters appearing at the end of the sorted alphabet as expected.

2. Apply the Sinhala collation patch to MySQL:

   • mysql-dfsg-5.0-5.0.27-sinhala1.patch

3. Test new Sinhala collation algorithm:

   http://en.wikipedia.org/wiki/Sinhala_alphabet

   CREATE TABLE tl (
           id SERIAL PRIMARY KEY,
           letter VARCHAR(10) NOT NULL
   ) CHARACTER SET utf8;

   Load the data from this file:

   • mysql-data-sinhala.txt

   Running this collation handler:

   SELECT * FROM tl ORDER BY letter COLLATE utf8_sinhala_ci;

   results match the Sinhala Character Code for Information Interchange Part 1 : Collation Sequence.

   Download the output file from:

   • mysql-output-utf8_sinhala_ci.txt

4. Need to add a Sinhala collation test to the MySQL test suite:

   • mysql/mysql-test/t/ctype_utf8.test

   • mysql/mysql-test/r/ctype_utf8.result