Subversion Repositories openrisc_2011-10-31

@node Iconv

@chapter Encoding conversions (@file{iconv.h})

This chapter describes the Newlib iconv library.

The iconv functions declarations are in

@file{iconv.h}.

@menu

* iconv::                           Encoding conversion routines

* Introduction::                    Introduction to iconv and encodings

* Supported encodings::             The list of currently supported encodings

* iconv design decisions::          General iconv library design issues

* iconv configuration::             iconv-related configure script options

* Encoding names::                  How encodings are named.

* CCS tables::                      CCS tables format and 'mktbl.pl' Perl script

* CES converters::                  CES converters description

* The encodings description file::  The 'encoding.deps' file and 'mkdeps.pl'

* How to add new encoding::         The steps to add new encoding support

* The locale support interfaces::   Locale-related iconv interfaces

* Contact::                         The author contact

@end menu

@page

@include iconv/iconv.def

@page

@node Introduction

@section Introduction

@findex encoding

@findex character set

@findex charset

@findex CES

@findex CCS

@*

The iconv library is intended to convert characters from one encoding to

another. It implements iconv(), iconv_open() and iconv_close()

calls, which are defined by the Single Unix Specification.

@*

In addition to these user-level interfaces, the iconv library also has

several useful interfaces which are needed to support coding

capabilities of the Newlib Locale infrastructure.  Since Locale

support also needs to

convert various character sets to and from the @emph{wide characters

set}, the iconv library shares it's capabilities with the Newlib Locale

subsystem. Moreover, the iconv library supports several features which are

only needed for the Locale infrastructure (for example, the MB_CUR_MAX value).

@*

The Newlib iconv library was created using concepts from another iconv

library implemented by Konstantin Chuguev (ver 2.0). The Newlib iconv library

was rewritten from scratch and contains a lot of improvements with respect to

the original iconv library.

@*

Terms like @dfn{encoding} or @dfn{character set} aren't well defined and

are often used with various meanings. The following are the definitions of terms

which are used in this documentation as well as in the iconv library

implementation:

@itemize @bullet

@item

@dfn{encoding} - a machine representation of characters by means of bits;

@item

@dfn{Character Set} or @dfn{Charset} - just a collection of

characters, i.e. the encoding is the machine representation of the character set;

@item

@dfn{CCS} (@dfn{Coded Character Set}) - a mapping from an character set to a

set of integers @dfn{character codes};

@item

@dfn{CES} (@dfn{Character Encoding Scheme}) - a mapping from a set of character

codes to a sequence of bytes;

@end itemize

@*

Users usually deal with encodings, for example, KOI8-R, Unicode, UTF-8,

ASCII, etc. Encodings are formed by the following chain of steps:

@enumerate

@item

User has a set of characters which are specific to his or her language (character set).

@item

Each character from this set is uniquely numbered, resulting in an CCS.

@item

Each number from the CCS is converted to a sequence of bits or bytes by means

of a CES and form some encoding. Thus, CES may be considered as a

function of CCS which produces some encoding. Note, that CES may be

applied to more than one CCS.

@end enumerate

@*

Thus, an encoding may be considered as one or more CCS + CES.

@*

Sometimes, there is no CES and in such cases encoding is equivalent

to CCS, e.g. KOI8-R or ASCII.

@*

An example of a more complicated encoding is UTF-8 which is the UCS

(or Unicode) CCS plus the UTF-8 CES.

@*

The following is a brief list of iconv library features:

@itemize

@item

Generic architecture;

@item

Locale infrastructure support;

@item

Automatic generation of the program code which handles

CES/CCS/Encoding/Names/Aliases dependencies;

@item

The ability to choose size- or speed-optimazed

configuration;

@item

The ability to exclude a lot of unneeded code and data from the linking step.

@end itemize

@page

@node Supported encodings

@section Supported encodings

@findex big5

@findex cp775

@findex cp850

@findex cp852

@findex cp855

@findex cp866

@findex euc_jp

@findex euc_kr

@findex euc_tw

@findex iso_8859_1

@findex iso_8859_10

@findex iso_8859_11

@findex iso_8859_13

@findex iso_8859_14

@findex iso_8859_15

@findex iso_8859_2

@findex iso_8859_3

@findex iso_8859_4

@findex iso_8859_5

@findex iso_8859_6

@findex iso_8859_7

@findex iso_8859_8

@findex iso_8859_9

@findex iso_ir_111

@findex koi8_r

@findex koi8_ru

@findex koi8_u

@findex koi8_uni

@findex ucs_2

@findex ucs_2_internal

@findex ucs_2be

@findex ucs_2le

@findex ucs_4

@findex ucs_4_internal

@findex ucs_4be

@findex ucs_4le

@findex us_ascii

@findex utf_16

@findex utf_16be

@findex utf_16le

@findex utf_8

@findex win_1250

@findex win_1251

@findex win_1252

@findex win_1253

@findex win_1254

@findex win_1255

@findex win_1256

@findex win_1257

@findex win_1258

@*

The following is the list of currently supported encodings. The first column

corresponds to the encoding name, the second column is the list of aliases,

the third column is its CES and CCS components names, and the fourth column

is a short description.

@multitable @columnfractions .20 .26 .24 .30

@item

Name

@tab

Aliases

@tab

CES/CCS

@tab

Short description

@item

@tab

@tab

@tab

@item

big5

@tab

csbig5, big_five, bigfive, cn_big5, cp950

@tab

table_pcs / big5, us_ascii

@tab

The encoding for the Traditional Chinese.

@item

cp775

@tab

ibm775, cspc775baltic

@tab

table / cp775

@tab

The updated version of CP 437 that supports the balitic languages.

@item

cp850

@tab

ibm850, 850, cspc850multilingual

@tab

table / cp850

@tab

IBM 850 - the updated version of CP 437 where several Latin 1 characters have been

added instead of some less-often used characters like the line-drawing

and the greek ones.

@item

cp852

@tab

ibm852, 852, cspcp852

@tab

@tab

IBM 852 - the updated version of CP 437 where several Latin 2 characters have been added

instead of some less-often used characters like the line-drawing and the greek ones.

@item

cp855

@tab

ibm855, 855, csibm855

@tab

table / cp855

@tab

IBM 855 - the updated version of CP 437 that supports Cyrillic.

@item

cp866

@tab

866, IBM866, CSIBM866

@tab

table / cp866

@tab

IBM 866 - the updated version of CP 855 which follows more the logical Russian alphabet

ordering of the alternative variant that is preferred by many Russian users.

@item

euc_jp

@tab

eucjp

@tab

euc / jis_x0208_1990, jis_x0201_1976, jis_x0212_1990

@tab

EUC-JP - The EUC for Japanese.

@item

euc_kr

@tab

euckr

@tab

euc / ksx1001

@tab

EUC-KR - The EUC for Korean.

@item

euc_tw

@tab

euctw

@tab

euc / cns11643_plane1, cns11643_plane2, cns11643_plane14

@tab

EUC-TW - The EUC for Traditional Chinese.

@item

iso_8859_1

@tab

iso8859_1, iso88591, iso_8859_1:1987, iso_ir_100, latin1, l1, ibm819, cp819, csisolatin1

@tab

table / iso_8859_1

@tab

ISO 8859-1:1987 - Latin 1, West European.

@item

iso_8859_10

@tab

iso_8859_10:1992, iso_ir_157, iso885910, latin6, l6, csisolatin6, iso8859_10

@tab

table / iso_8859_10

@tab

ISO 8859-10:1992 - Latin 6, Nordic.

@item

iso_8859_11

@tab

iso8859_11, iso885911

@tab

table / iso_8859_11

@tab

ISO 8859-11 - Thai.

@item

iso_8859_13

@tab

iso_8859_13:1998, iso8859_13, iso885913

@tab

table / iso_8859_13

@tab

ISO 8859-13:1998 - Latin 7, Baltic Rim.

@item

iso_8859_14

@tab

iso_8859_14:1998, iso885914, iso8859_14

@tab

table / iso_8859_14

@tab

ISO 8859-14:1998 - Latin 8, Celtic.

@item

iso_8859_15

@tab

iso885915, iso_8859_15:1998, iso8859_15,

@tab

table / iso_8859_15

@tab

ISO 8859-15:1998 - Latin 9, West Europe, successor of Latin 1.

@item

iso_8859_2

@tab

iso8859_2, iso88592, iso_8859_2:1987, iso_ir_101, latin2, l2, csisolatin2

@tab

table / iso_8859_2

@tab

ISO 8859-2:1987 - Latin 2, East European.

@item

iso_8859_3

@tab

iso_8859_3:1988, iso_ir_109, iso8859_3, latin3, l3, csisolatin3, iso88593

@tab

table / iso_8859_3

@tab

ISO 8859-3:1988 - Latin 3, South European.

@item

iso_8859_4

@tab

iso8859_4, iso88594, iso_8859_4:1988, iso_ir_110, latin4, l4, csisolatin4

@tab

table / iso_8859_4

@tab

ISO 8859-4:1988 - Latin 4, North European.

@item

iso_8859_5

@tab

iso8859_5, iso88595, iso_8859_5:1988, iso_ir_144, cyrillic, csisolatincyrillic

@tab

table / iso_8859_5

@tab

ISO 8859-5:1988 - Cyrillic.

@item

iso_8859_6

@tab

iso_8859_6:1987, iso_ir_127, iso8859_6, ecma_114, asmo_708, arabic, csisolatinarabic, iso88596

@tab

table / iso_8859_6

@tab

ISO i8859-6:1987 - Arabic.

@item

iso_8859_7

@tab

iso_8859_7:1987, iso_ir_126, iso8859_7, elot_928, ecma_118, greek, greek8, csisolatingreek, iso88597

@tab

table / iso_8859_7

@tab

ISO 8859-7:1987 - Greek.

@item

iso_8859_8

@tab

iso_8859_8:1988, iso_ir_138, iso8859_8, hebrew, csisolatinhebrew, iso88598

@tab

table / iso_8859_8

@tab

ISO 8859-8:1988 - Hebrew.

@item

iso_8859_9

@tab

iso_8859_9:1989, iso_ir_148, iso8859_9, latin5, l5, csisolatin5, iso88599

@tab

table / iso_8859_9

@tab

ISO 8859-9:1989 - Latin 5, Turkish.

@item

iso_ir_111

@tab

ecma_cyrillic, koi8_e, koi8e, csiso111ecmacyrillic

@tab

table / iso_ir_111

@tab

ISO IR 111/ECMA Cyrillic.

@item

koi8_r

@tab

cskoi8r, koi8r, koi8

@tab

table / koi8_r

@tab

RFC 1489 Cyrillic.

@item

koi8_ru

@tab

koi8ru

@tab

table / koi8_ru

@tab

The obsolete Ukrainian.

@item

koi8_u

@tab

koi8u

@tab

table / koi8_u

@tab

RFC 2319 Ukrainian.

@item

koi8_uni

@tab

koi8uni

@tab

table / koi8_uni

@tab

KOI8 Unified.

@item

ucs_2

@tab

ucs2, iso_10646_ucs_2, iso10646_ucs_2, iso_10646_ucs2, iso10646_ucs2, iso10646ucs2, csUnicode

@tab

ucs_2 / (UCS)

@tab

ISO-10646-UCS-2. Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).

@item

ucs_2_internal

@tab

ucs2_internal, ucs_2internal, ucs2internal

@tab

ucs_2_internal / (UCS)

@tab

ISO-10646-UCS-2 in system byte order.

NBSP is always interpreted as NBSP (BOM isn't supported).

@item

ucs_2be

@tab

ucs2be

@tab

ucs_2 / (UCS)

@tab

Big Endian version of ISO-10646-UCS-2 (in fact, equivalent to ucs_2).

Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).

@item

ucs_2le

@tab

ucs2le

@tab

ucs_2 / (UCS)

@tab

Little Endian version of ISO-10646-UCS-2.

Little Endian, NBSP is always interpreted as NBSP (BOM isn't supported).

@item

ucs_4

@tab

ucs4, iso_10646_ucs_4, iso10646_ucs_4, iso_10646_ucs4, iso10646_ucs4, iso10646ucs4

@tab

ucs_4 / (UCS)

@tab

ISO-10646-UCS-4. Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).

@item

ucs_4_internal

@tab

ucs4_internal, ucs_4internal, ucs4internal

@tab

ucs_4_internal / (UCS)

@tab

ISO-10646-UCS-4 in system byte order.

NBSP is always interpreted as NBSP (BOM isn't supported).

@item

ucs_4be

@tab

ucs4be

@tab

ucs_4 / (UCS)

@tab

Big Endian version of ISO-10646-UCS-4 (in fact, equivalent to ucs_4).

Big Endian, NBSP is always interpreted as NBSP (BOM isn't supported).

@item

ucs_4le

@tab

ucs4le

@tab

ucs_4 / (UCS)

@tab

Little Endian version of ISO-10646-UCS-4.

Little Endian, NBSP is always interpreted as NBSP (BOM isn't supported).

@item

us_ascii

@tab

ansi_x3.4_1968, ansi_x3.4_1986, iso_646.irv:1991, ascii, iso646_us, us, ibm367, cp367, csascii

@tab

us_ascii / (ASCII)

@tab

7-bit ASCII.

@item

utf_16

@tab

utf16

@tab

utf_16 / (UCS)

@tab

RFC 2781 UTF-16. The very first NBSP code in stream is interpreted as BOM.

@item

utf_16be

@tab

utf16be

@tab

utf_16 / (UCS)

@tab

Big Endian version of RFC 2781 UTF-16.

NBSP is always interpreted as NBSP (BOM isn't supported).

@item

utf_16le

@tab

utf16le

@tab

utf_16 / (UCS)

@tab

Little Endian version of RFC 2781 UTF-16.

NBSP is always interpreted as NBSP (BOM isn't supported).

@item

utf_8

@tab

utf8

@tab

utf_8 / (UCS)

@tab

RFC 3629 UTF-8.

@item

win_1250

@tab

cp1250

@tab

@tab

Win-1250 Croatian.

@item

win_1251

@tab

cp1251

@tab

table / win_1251

@tab

Win-1251 - Cyrillic.

@item

win_1252

@tab

cp1252

@tab

table / win_1252

@tab

Win-1252 - Latin 1.

@item

win_1253

@tab

cp1253

@tab

table / win_1253

@tab

Win-1253 - Greek.

@item

win_1254

@tab

cp1254

@tab

table / win_1254

@tab

Win-1254 - Turkish.

@item

win_1255

@tab

cp1255

@tab

table / win_1255

@tab

Win-1255 - Hebrew.

@item

win_1256

@tab

cp1256

@tab

table / win_1256

@tab

Win-1256 - Arabic.

@item

win_1257

@tab

cp1257

@tab

table / win_1257

@tab

Win-1257 - Baltic.

@item

win_1258

@tab

cp1258

@tab

table / win_1258

@tab

Win-1258 - Vietnamese7 that supports Cyrillic.

@end multitable

@page

@node iconv design decisions

@section iconv design decisions

@findex CCS table

@findex CES converter

@findex Speed-optimized tables

@findex Size-optimized tables

@*

The first iconv library design issue arises when considering the

following two design approaches:

@enumerate

@item

Have modules which implement conversion from the encoding A to the encoding B

and vice versa i.e., one conversion module relates to any two encodings.

@item

Have modules which implement conversion from the encoding A to the fixed

encoding C and vice versa i.e., one conversion module relates to any

one encoding A and one fixed encoding C. In this case, to convert from

the encoding A to the encoding B, two modules are needed (in order to convert

from A to C and then from C to B).

@end enumerate

@*

It's obvious, that we have tradeoff between commonality/flexibility and

efficiency: the first method is more efficient since it converts

directly; however, it isn't so flexible since for each

encoding pair a distinct module is needed.

@*

The Newlib iconv model uses the second method and always converts through the 32-bit

UCS but its design also allows one to write specialized conversion

modules if the conversion speed is critical.

@*

The second design issue is how to break down (decompose) encodings.

The Newlib iconv library uses the fact that any encoding may be

considered as one or more CCS plus a CES. It also decomposes its

conversion modules on @dfn{CES converter} plus one or more @dfn{CCS

tables}. CCS tables map CCS to UCS and vice versa; the CES converters

map CCS to the encoding and vice versa.

@*

As the example, let's consider the conversion from the big5 encoding to

the EUC-TW encoding. The big5 encoding may be decomposed to the ASCII and BIG5

CCS-es plus the BIG5 CES. EUC-TW may be decomposed on the CNS11643_PLANE1, CNS11643_PLANE2,

and CNS11643_PLANE14 CCS-es plus the EUC CES.

@*

The euc_jp -> big5 conversion is performed as follows:

@enumerate

@item

The EUC converter performs the EUC-TW encoding to the corresponding CCS-es

transformation (CNS11643_PLANE1, CNS11643_PLANE2 and CNS11643_PLANE14

CCS-es);

@item

The obtained CCS codes are transformed to the UCS codes using the CNS11643_PLANE1,

CNS11643_PLANE2 and CNS11643_PLANE14 CCS tables;

@item

The resulting UCS codes are transformed to the ASCII and BIG5 codes using

the corresponding CCS tables;

@item

The obtained CCS codes are transformed to the big5 encoding using the corresponding

CES converter.

@end enumerate

@*

Analogously, the backward conversion is performed as follows:

@enumerate

@item

The BIG5 converter performs the big5 encoding to the corresponding CCS-es transformation

(the ASCII and BIG5 CCS-es);

@item

The obtained CCS codes are transformed to the UCS codes using the ASCII and BIG5 CCS tables;

@item

The resulting UCS codes are transformed to the ASCII and BIG5 codes using

the corresponding CCS tables;

@item

The obtained CCS codes are transformed to the EUC-TW encoding using the corresponding

CES converter.

@end enumerate

@*

Note, the above is just an example and real names (which are implemented

in the Newlib iconv) of the CES converters and the CCS tables are slightly different.

@*

The third design issue also relates to flexibility. Obviously, it isn't

desirable to always link all the CES converters and the CCS tables to the library

but instead, we want to be able to load the needed converters and tables

dynamically on demand. This isn't a problem on "big" machines such as

a PC, but it may be very problematical within "small" embedded systems.

@*

Since the CCS tables are just data, it is possible to load them

dynamically from external files.  The CES converters, on the other hand

are algorithms with some code so a dynamic library loading

capability is required.

@*

Apart from possible restrictions applied by embedded systems (small

RAM for example), Newlib itself has no dynamic library support and

therefore, all the CES converters which will ever be used must be linked into

the library.   However, loading of the dynamic CCS tables is possible and is

implemented in the Newlib iconv library.  It may be enabled via the Newlib

configure script options.

@*

The next design issue is fine-tuning the iconv library

configuration.  One important ability is for iconv to not link all it's

converters and tables (if dynamic loading is not enabled) but instead,

enable only those encodings which are specified at configuration

time (see the section about the configure script options).

@*

In addition, the Newlib iconv library configure options distinguish between

conversion directions. This means that not only are supported encodings

selectable, the conversion direction is as well. For example, if user wants

the configuration which allows conversions from UTF-8 to UTF-16 and

doesn't plan using the "UTF-16 to UTF-8" conversions, he or she can

enable only

this conversion direction (i.e., no "UTF-16 -> UTF-8"-related code will

be included) thus, saving some memory (note, that such technique allows to

exclude one half of a CCS table from linking which may be big enough).

@*

One more design aspect are the speed- and size- optimized tables. Users can

select between them using configure script options. The

speed-optimized CCS tables are the same as the size-optimized ones in

case of 8-bit CCS (e.g.m KOI8-R), but for 16-bit CCS-es the size-optimized

CCS tables may be 1.5 to 2 times less then the speed-optimized ones. On the

other hand, conversion with speed tables is several times faster.

@*

Its worth to stress that the new encoding support can't be

dynamically added into an already compiled Newlib library, even if it

needs only an additional CCS table and iconv is configured to use

the external files with CCS tables (this isn't the fundamental restriction

and the possibility to add new Table-based encoding support dynamically, by

means of just adding new .cct file, may be easily added).

@*

Theoretically, the compiled-in CCS tables should be more appropriate for

embedded systems than dynamically loaded CCS tables.  This is because the compiled-in tables are read-only and can be placed in ROM

whereas dynamic loading requires RAM.  Moreover, in the current iconv

implementation, a distinct copy of the dynamic CCS file is loaded for each opened iconv descriptor even in case of the same encoding.

This means, for example, that if two iconv descriptors for

"KOI8-R -> UCS-4BE" and "KOI8-R -> UTF-16BE" are opened, two copies of

koi8-r .cct file will be loaded (actually, iconv loads only the needed part

of these files).  On the other hand, in the case of compiled-in CCS tables, there will always be only one copy.

@page

@node iconv configuration

@section iconv configuration

@findex iconv configuration

@findex --enable-newlib-iconv-encodings

@findex --enable-newlib-iconv-from-encodings

@findex --enable-newlib-iconv-to-encodings

@findex --enable-newlib-iconv-external-ccs

@findex NLSPATH

@*

To enable an encoding, the @emph{--enable-newlib-iconv-encodings} configure

script option should be used. This option accepts a comma-separated list

of @emph{encodings} that should be enabled. The option enables each encoding in both

("to" and "from") directions.

@*

The @option{--enable-newlib-iconv-from-encodings} configure script option enables

"from" support for each encoding that was passed to it.

@*

The @option{--enable-newlib-iconv-to-encodings} configure script option enables

"to" support for each encoding that was passed to it.

@*

Example: if user plans only the "KOI8-R -> UTF-8", "UTF-8 -> ISO-8859-5" and

"KOI8-R -> UCS-2" conversions, the most optimal way (minimal iconv

code and data will be linked) is to configure Newlib with the following

options:

@*

@code{--enable-newlib-iconv-encodings=UTF-8

--enable-newlib-iconv-from-encodings=KOI8-R

--enable-newlib-iconv-to-encodings=UCS-2,ISO-8859-5}

@*

which is the same as

@*

@code{--enable-newlib-iconv-from-encodings=KOI8-R,UTF-8

--enable-newlib-iconv-to-encodings=UCS-2,ISO-8859-5,UTF-8}

@*

User may also just use the

@*

@code{--enable-newlib-iconv-encodings=KOI8-R,ISO-8859-5,UTF-8,UCS-2}

@*

configure script option, but it isn't so optimal since there will be

some unneeded data and code.

@*

The @option{--enable-newlib-iconv-external-ccs} option enables iconv's

capabilities to work with the external CCS files.

@*

The @option{--enable-target-optspace} Newlib configure script option also affects

the iconv library. If this option is present, the library uses the size

optimized CCS tables. This means, that only the size-optimized CCS

tables will be linked or, if the

@option{--enable-newlib-iconv-external-ccs} configure script option was used,

the iconv library will load the size-optimized tables. If the

@option{--enable-target-optspace}configure script option is disabled,

the speed-optimized CCS tables are used.

@*

Note: .cct files are searched by iconv_open in the $NLSPATH/iconv_data/ directory.

Thus, the NLSPATH environment variable should be set.

@page

@node Encoding names

@section Encoding names

@findex encoding name

@findex encoding alias