• Created: 2006-01-03 by Atie

• Priority: NeedsPriority

• People: NeedsLead, NeedsSecond

• Contributors: Atie, JunKobayashi, AbelCheung4, Zero0w

• Interested:

• Status: UbzSpecification, BrainDump (then DraftSpecification then EditedSpecification then ApprovedSpecification), DistroSpecification

• Branch:

• Malone bug: 29185, 31208, 31149, 38595

• Packages affected:

• Depends: This specification is a sub specficication of BetterCJKSupportSpecification.

• Dependents:
  
  Display context of search results
  Case-sensitive searching
• BoF sessions: none yet

Summary

This sub specification aims at improving Font configuration for CJK support in Ubuntu.

Rationale

Usually Chinese, Japanese and Korean (CJK) users make their own font configurations in ~/.fonts.conf so that they see the best looking, highest quality, ttf fonts. This is necessary due to lack of support by fontconfig; we don't currently have any method to simplify this process for CJK users.

Use cases

• Chulsoo installed ttf-alee without ubuntu-desktop and found the fonts look fuzzy and didn't match with his English font while web browsing. He asked Linux forum about font configuration and finally got a good looking setup. But it was not a simple way to set it up because he didn't understand the xml tags and values.
• Joe wants to learn Chinese and he is happy with ubuntu-desktop because it already has Chinese ttf fonts, but he is not satisfied with the display quality and his Chinese classmate tells him it looks poor. But he doesn't know how to improve the appearance.

Background

We believe we have established the following facts.

• For CJK languages, users have different preferences as to their preferred font and it is not acceptable to provide only one option; or to put it another way, users must be able to choose between a few options without editing config files. Currently this is not possible.
• The current defaults are suboptimal; they result (in some languages) in the use of fonts which are not a reasonable choice.

• fontconfig's properties features can be used to provide different behaviour for different languages. In particular, the <edit> stanzas which are the preferred approach for selecting CJK fonts can be limited using <test> to particular languages.

• All of the relevant ttf-* font packages are installed by default as dependencies of ubuntu-desktop.
• There are 5 languages involved: Korean, Jananese, mainland Chinese, Chinese for Taiwan and Chinese for Hong Kong. Each language will have approximately two sensible choices of font, so we will need approximately 10 fontconfig XML stanzas.

Approach

The default choice of the font system must be improved. We would like the user to be able to make the choice of font using the normal font configuration mechanism (System / Preferences / Font and the corresponding Firefox preferences).

This will be achieved by providing language-specific fontconfig XML stanzas in the default install which (i) map sans-serif, the default font, to the "best" relevant CJK font (ii) provide a mapping for suitable name(s) for the plausible alternatives.

These XML stanzas will be provided in /etc/fonts/conf.d by the ttf-* packages. Providing them there means that, since all systems will have them installed, any bugs which make the changes break other languages will be detected; if they were in the language packs then such bugs would strike systems with multiple language support but be otherwise undetected. The other alternative would be to provide this information from the fontconfig package itself but since it's likely to be fairly large, and at least initially fairly volatile, and to some extent coupled to the ttf font packages' contents, this would be less than ideal.

Open Questions

The information here is currently based mainly on the knowledge of a Korean-language expert. The situation for the other affected languages needs to be checked.

There have been reports of trouble to do with ordering of the various fontconfig stanzas. fontconfig is sensitive to ordering and somewhat fragile. Certainly none of the language-specific changes to fontconfig files should involve reordering any existing configuration. We need to establish that writing appropriate XML to /etc/fonts/conf.d is sufficient, and whether there is any specific ordering constraint which affects the appropriate choice of filename. (Files in conf.d are read in lexicographical order, and recent practice is to prepend a sequence number to the filename.)

We currently have no confirmation that language-specific XML works correctly in practice. If it does not then we must either fix it (if this is feasible) or opt for a different approach (which will definitely be suboptimal for multilingual installs).

Exact syntax of the appropriate fontconfig XML stanzas need to be established and tested. We have examples for Korean (see below).

Implemented solution

• language-selector-common provides font configuration files for CJK as below. Check out the DapperL10nSprint wiki for details.

   /usr/share/language-selector/fontconfig/ja_JP
   /usr/share/language-selector/fontconfig/ko_KR
   /usr/share/language-selector/fontconfig/zh_CN
   /usr/share/language-selector/fontconfig/zh_HK
   /usr/share/language-selector/fontconfig/zh_SG
   /usr/share/language-selector/fontconfig/zh_TW

• One of fontconfig/<language> above is linked to /etc/fonts/language-selector.conf for CJK by choice of language at installation. Users can use gnome-language-selector or qt-language-selector to do same linking with the GUI.

• 1) and 2) in the Notes below had been implemented by language-selector and the embedded bitmap patch to fontconfig. 3) and 4) are also requested as patch of fontconfig for CJK. (2006-04-24)

Notes

These notes predate the discussions and decision documented above but are retained because they contain information that may still be useful.

• You can also use fontforge to open and explore the ttf files and determine whether or not they carry embedded bitmaps, and at what sizes.

  • Status on 18/3: It is decided fontconfig 2.3.9x won't go into dapper. Instead, 2.3.2 will be used. I isolated the necessary part of patch for fontconfig here: http://abelcheung.org/~deaddog/ubuntu/fontconfig-2.3.2-embedded-bitmap.patch. This patch is supposed to go into ubuntu soon. For libXft, Ubuntu already included version 2.1.8.2 so that's not a problem.

  • There is also another way to enable embedded bitmap globally without the need to match certain range of font size over a particular font name. It is to patch the fontconfig and libXft package directly, here are the patches:

    • fontconfig-2.2.3-add_extra_option-20041127-01.patch

    • http://firefly.idv.tw/setfont-xft/patches/libXft/2.1.6/libXft-2.1.6-CJK-extra-20041128.patch

  • With these patches, a new fontconfig parameter called embeddedbitmap can be turned on, which will manage the process globally and only target the font with embedded bitmaps. This is better than the antialias parameter because antialias will turn off anti-aliasing even at those font sizes that are without embedded bitmaps. And that's the reason we need to limit it to certain range as we have been doing now.

  • That's right. fontconfig 2.4 is NOT going to come out soon enough for the Dapper Drake release. Either we use the patch above as suggested by Status on 18/3, or we have to resort to the old way for enabling embedded bitmap font by using the antialias attribute. For now it's likely we will choose the path to patch fontconfig 2.3.2 directly.

    • Update Mar 20 - Note: When slant (italic) style is used, embeddedbitmap mode must be turned off - since bitmap glyph cannot be italicized - they must be reverted back to vector (non-bitmap) glyph by turning off embeddedbitmap mode (just for italic style). We will need to do this manually ourselves in the /etc/fonts/fonts.conf.

    • More discussions can be found in this Chinese Debian forum.

• If there will be no problem to touch the configuration by fontconfig directly, I am going to add unfonts for Korean to the configuration as Chinese and Japanese in below. However, I am afraid confliction by order as seen below between Chinese and Japanese.
• Chinese
  1. ttf-arphic-uming/ttf-arphic-ukai (truetype, bitmap for basic zh_CN/zh_TW/zh_HK)
  2. xfonts-wqy (bitmap for full GBK coverage) (ttf-newsung not needed as ttf-arphic-uming includes it)
     • For Chinese, fonts in the following order looks best. However, this may conflict with Japanese and/or Korean, because CJK truetype fonts have some glyphs in common, and for each language, their own fonts are preferred.

         <alias>
           <family>serif</family>
           <prefer>
             <family>Bitstream Vera Serif</family>
             <family>DejaVu Serif</family>
             <family>Times New Roman</family>
             <family>Thorndale AMT</family>
             <family>Luxi Serif</family>
             <family>Nimbus Roman No9 L</family>
             <family>Times</family>
             <family>Frank Ruehl</family>
             <family>MgOpen Canonica</family>
             <family>AR PL ShanHeiSun Uni</family>
             <family>AR PL ZenKai Uni</family>
             <family>IPAMonaPMincho</family>
             <family>IPAPMincho</family>
             <family>Sazanami Mincho</family>
             <family>Kochi Mincho</family>
             <family>ＭＳ 明朝</family>
             <family>Baekmuk Batang</family>
             <family>FreeSerif</family>
           </prefer>
         </alias>
         <alias>
           <family>sans-serif</family>
           <prefer>
             <family>Bitstream Vera Sans</family>
             <family>DejaVu Sans</family>
             <family>Verdana</family>
             <family>Arial</family>
             <family>Albany AMT</family>
             <family>Luxi Sans</family>
             <family>Nimbus Sans L</family>
             <family>Helvetica</family>
             <family>Nachlieli</family>
             <family>MgOpen Moderna</family>
             <family>AR PL ShanHeiSun Uni</family>
             <family>AR PL ZenKai Uni</family>
             <family>IPAMonaPGothic</family>
             <family>IPAPGothic</family>
             <family>Sazanami Gothic</family>
             <family>Kochi Gothic</family>
             <family>ＭＳ ゴシック</family>
             <family>Baekmuk Dotum</family>
             <family>SimSun</family>
             <family>FreeSans</family>
           </prefer>
         </alias>
         <alias>
           <family>monospace</family>
           <prefer>
             <family>Bitstream Vera Sans Mono</family>
             <family>DejaVu Sans Mono</family>
             <family>Andale Mono</family>
             <family>Courier New</family>
             <family>Cumberland AMT</family>
             <family>Luxi Mono</family>
             <family>Nimbus Mono L</family>
             <family>Courier</family>
             <family>Miriam Mono</family>
             <family>AR PL ShanHeiSun Uni</family>
             <family>AR PL ZenKai Uni</family>
             <family>IPAMonaGothic</family>
             <family>IPAGothic</family>
             <family>Sazanami Gothic</family>
             <family>Kochi Gothic</family>
             <family>AR PL KaitiM GB</family>
             <family>Baekmuk Dotum</family>
             <family>FreeMono</family>
           </prefer>
         </alias>

• Japanese
  1. ttf-kochi-gothic
  2. ttf-kochi-mincho
  3. ttf-sazanami-gothic
  4. ttf-sazanami-mincho
  5. ipafont
  6. ipamonafont

     Note: ipafont and ipamonafont package can be found in ubuntu-ja breezy-extra.

     Also see the possible licensing issue.

  7. The following configuration including:

     • To get display Japanese correctly, dropping FreeSans and FreeSerif priority(29185).

     • If Sazanami font, IPA font and IPA Mona Font are installed, display them.
     If this configuration don't have bad effects on other languages, I think it may be good to merge the default /etc/fonts/fonts.conf.

     •   <alias>
           <family>serif</family>
           <prefer>
             <family>Bitstream Vera Serif</family>
             <family>DejaVu Serif</family>
             <family>Times New Roman</family>
             <family>Thorndale AMT</family>
             <family>Luxi Serif</family>
             <family>Nimbus Roman No9 L</family>
             <family>Times</family>
             <family>Frank Ruehl</family>
             <family>MgOpen Canonica</family>
             <family>IPAMonaPMincho</family>
             <family>IPAPMincho</family>
             <family>Sazanami Mincho</family>
             <family>Kochi Mincho</family>
             <family>AR PL SungtiL GB</family>
             <family>AR PL Mingti2L Big5</family>
             <family>ＭＳ 明朝</family>
             <family>Baekmuk Batang</family>
             <family>FreeSerif</family>
           </prefer>
         </alias>
         <alias>
           <family>sans-serif</family>
           <prefer>
             <family>Bitstream Vera Sans</family>
             <family>DejaVu Sans</family>
             <family>Verdana</family>
             <family>Arial</family>
             <family>Albany AMT</family>
             <family>Luxi Sans</family>
             <family>Nimbus Sans L</family>
             <family>Helvetica</family>
             <family>Nachlieli</family>
             <family>MgOpen Moderna</family>
             <family>IPAMonaPGothic</family>
             <family>IPAPGothic</family>
             <family>Sazanami Gothic</family>
             <family>Kochi Gothic</family>
             <family>AR PL KaitiM GB</family>
             <family>AR PL KaitiM Big5</family>
             <family>ＭＳ ゴシック</family>
             <family>Baekmuk Dotum</family>
             <family>SimSun</family>
             <family>FreeSans</family>
           </prefer>
         </alias>
         <alias>
           <family>monospace</family>
           <prefer>
             <family>Bitstream Vera Sans Mono</family>
             <family>DejaVu Sans Mono</family>
             <family>Andale Mono</family>
             <family>Courier New</family>
             <family>Cumberland AMT</family>
             <family>Luxi Mono</family>
             <family>Nimbus Mono L</family>
             <family>Courier</family>
             <family>Miriam Mono</family>
             <family>IPAMonaGothic</family>
             <family>IPAGothic</family>
             <family>Sazanami Gothic</family>
             <family>Kochi Gothic</family>
             <family>AR PL KaitiM GB</family>
             <family>Baekmuk Dotum</family>
             <family>FreeMono</family>
           </prefer>
         </alias>

• Korean
  1. ttf-baekmuk ( Fonts in this package are used by fontconfig. By a survey, 5% of voters like this - due to lower quality for displaying than the others. )
  2. ttf-unfonts ( 63% of voters prefer this. Has several font-sets, even bold font-sets. )
     • To give preference to Unfonts than Baekmuk

         <alias>
           <family>serif</family>
           <prefer>
              <family>UnBatang</family>
              <family>Baekmuk Batang</family>
           </prefer>
         </alias>
         <alias>
           <family>sans-serif</family>
           <prefer>
             <family>UnDotum</family>
             <family>Baekmuk Dotum</family>
            </prefer>
         </alias>

     • To get better looking

          <match target="font" >
             <test name="family" qual="any" >
                <string>UnPen</string>
                <string>UnBatang</string>
                <string>UnDotum</string>
                <string>UnGungseo</string>
                <string>UnPilgi</string>
                <string>UnBom</string>
                <string>UnYetgul</string>
                <string>UnPenheulim</string>
                <string>UnGraphic</string>
                <string>UnSinmun</string>
             </test>
             <edit mode="assign" name="hinting" ><bool>false</bool></edit>
             <edit mode="assign" name="autohint" ><bool>true</bool></edit>
          </match>

  3. ttf-alee ( 30% of voters, includes bitmap font and good for screen display. )
     • To get best displaying

          <match target="font">
              <test name="family"><string>EunGuseul</string></test>
                  <edit mode="assign" name="autohint"><bool>true</bool></edit>
          </match>
          <match target="font">
              <test name="family"><string>EunGuseul</string></test>
              <test name="pixelsize" compare="more"><int>11</int></test>
              <test name="pixelsize" compare="less"><int>16</int></test>
                  <edit mode="assign" name="autohint"><bool>false</bool></edit>
                  <edit mode="assign" name="antialias"><bool>false</bool></edit>
          </match>

     • To match with Bitsream Vera fonts

          <match target="pattern">
              <test name="family"><string>EunGuseul</string></test>
                  <edit mode="append" binding="strong" Name="family">
                     <string>Bitstream Vera Sans</string>
                 </edit>
          </match>
          <match target="pattern">
              <test name="family"><string>EunGuseul Mono</string></test>
                  <edit mode="append" binding="strong" name="family">
                     <string>Bitstream Vera Sans Mono</string>
                  </edit>
          </match> 

2. Setting up a .conf file to /etc/fonts/conf.d when a ttf-font package installed, is it preferred way to solve this?

• I think this may solve the problem of each CJK languages , but how can we configure it for defaultly using ? And also the configureation of each languages need more test , avoid affect other languages such as greek, English ,etc.

• I think if fontsconf support this:

  • <test name='locale'><String>zh_CN.UTF-8</String></test> or <test name='locale'><String>zh_CN.*</String></test> now, Chinese,Japanese,Korean and other can do it by self.

• There are lang, familylang, stylelang and fullnamelang properties of fontconfig, try them instead of <test name='locale'> above. ( fonts-conf ) To list up installed ttf fonts per lang, use fc-list command like this, fc-list :lang=ko .

• I found the name lang used on SuSE Linux 9 Japanese Edition. The following code is one part of suse-pre-user.conf. I'll report about other distributions when I check them up.

• Fontconfig setting samples, used by CJK users of other distros (only for reference):

  • Fedora (compiled freetype 2.1.10 and fontconfig 2.3.2 by himself)

  • Debian (first example, second example)

3. The below configration for spacing needs to be a patch of Dapper's fontconfig for CJK langs. The ko_KR fontconfig file contains same configuration already, but better to be used for all CJK.

•         <match target="font">
                  <test name="lang" compare="contains">
                          <string>ja</string>
                          <string>zh</string>
                          <string>ko</string>
                  </test>
                  <test name="spacing" compare="eq">
                          <const>dual</const>
                  </test>
                  <edit name="spacing">
                          <const>proportional</const>
                  </edit>
                  <edit name="globaladvance" binding="strong">
                          <bool>false</bool>
                  </edit>
          </match>

4.To display semibold as bold for embolden glyphes, the following configuration required. Problem and test results are described in the bug 38595 Applying this configuration to fontconfig with "CJK lang test" is a boon to most of CJK fonts which don't have semibold and bold glyphes.

•         <match target="font">
                  <test name="weight" compare="less_eq">
                          <const>medium</const>
                  </test>
                  <test target="pattern" name="weight" compare="more">
                          <const>medium</const>
                  </test>
                  <edit name="embolden" mode="assign">
                          <bool>true</bool>
                  </edit>
                  <edit name="weight" mode="assign">
                          <const>bold</const>
                  </edit>
           </match>