Python Module for Myanmar Language Processing

I am writing a crawler, to crawl Burmese phrases from the Web, in python currently. Since non-unicode encodings — such as wininnwa and zawgyi — are also used commonly in Myanmar websites, I needed to convert those texts into unicode.

So, I ported Thanlwinsoft's Myanmar converter, written in javascript, to python. Since, converting those encodings is a common task in Myanmar language processing, I am releasing it here.

I have also written a python script, which can be used to convert those encodings from command line.

Please note that I haven't tested it throughly, and this is an alpha release. So, don't use it without backing up first.

Install

For debian/ubuntu users, you can just grab the deb package and install. For other platforms, download the archive and run this command in extracted folder.

python setup.py install

Usage

Using myanmar python module

   >>> import myanmar.converter as converter
   >>> converter.get_available_encodings ()
   ['zawgyi', 'wwin_burmese', 'wininnwa', 'unicode']   
   >>> print converter.convert (u'ydkifoGefy½dk*&rfrif;', 'wininnwa', 'unicode')
   ပိုင်​သွန်​ပ​ရို​ဂ​ရမ်​မင်း

Using myanmar-converter script

trhura @ ~ $ myanmar-converter -h
Usage: myanmar-converter [OPTIONS...] [FILE...]

Convert between Myanmar legacy encodings and unicode.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -l, --list            List supported encodings.
  -f ENCODING, --from=ENCODING
                        Convert characters from ENCODING
  -t ENCODING, --to=ENCODING
                        Convert characters to ENCODING
  -o FILE, --output=FILE
                        Write output to FILE

trhura @ ~ $ echo 'wmrDe,frStokH;ðyykH' |  \
 myanmar-converter -f 'wwin_burmese' -t 'unicode'
တာ​မီ​နယ်​မှ​အသုံးပြု​ပုံ

Unicode Character Boundary Segmentation for Burmese

        Character boundary segmentation for Burmese is still not working properly in all platforms supporting Myanmar unicode (Windows , OS X, ICU, Pango). Although the default character segmentation specification, described in Unicode Standard Annex #29, works well for most Burmese text, there are, however, still some cases, where it needs to be tailored be work properly for Burmese.

         Normally, Burmese (user-perceived) characters are a combination for consonants + marks (medials, vowel, tones). However, this simple rule doesn't work when final (devowelized) consonants are involved. Final consonants are consonants with their inherent vowel killed. There are three ways final consonants appear in Burmese –

1. Followed by U103A (ASAT SIGN) (က်)
2. Stacked together with succeeding consonants (က္က)
3. As Kinzi (င်္)

         Two separate rules for segmentation of those final consonants need to be considered.

1. When followed by ASAT — (က်, မ်, င်, ည် …)

         They are segmented as separate characters in all platforms currently supporting Myanmar unicode (ICU, Windows8, pango). For example, မောင် is segmented into two separate characters – မော and င်. Although they seem visually separate, မောင် is considered as one character in Burmese. Let's say — we want to display “ဒိန်ချဉ်ဆိုင်” vertically, as in a signboard. This will be displayed as:

ဒိ

န်

ချ

ဉ်

ဆို

င်,

where it should be displayed as:

ဒိန်

ချဉ်

ဆိုင်

2. Kinzi and Stacked Consonants — (အင်္ဂါ, အက္ခရာ…)
 

        The default unicode algorithm segments each consonant + mark combination as a character. However, since two consonants are conjoined in kinzis and stacked consonants, “one consonant, one character” rule makes it confusing for selecting text and cursor movement. Currently, pango and ICU segment those conjoined consonants as two separate characters.

အင်္ဂါ = အ + င်္ + ဂါ
အက္ခရာ = အ + က ္ + ခ + ရာ

        Windows 8 segments those conjoined characters as one character – which is the proper approach in my opinion.

အင်္ဂါ = အ + င်္ဂါ
အက္ခရာ = အ + က္ခ + ရာ

        There is another issue regarding U104E (၎). Myanmar symbol ၎ (U104E), unlike other symbols, is not an independent symbol. It is always written together with Nga, Asat, Visarga. So, ၎င်း should be counted as one character, rather two characters (၎ + င်း).

Simple Burmese OCR

As my graduation project, I am working on a simple Burmese OCR (only alphabets & digits) using ANN. I have already finished the demo program. Here are some screenshots for the curious.