@Go to the ‰ŠP�”›first@�”˜, ‰ŠP�”›previous@�”˜, ‰ŠP�”›next@�”˜, ‰ŠP�”›last@�”˜ section, ‰ŠP�”›table of contents@�”˜.
Œ

ˆ‰SEC311Št�”›Extended Characters

@�”˜A number of languages use character sets that are larger than the range of values of type Hchar@. Japanese and Chinese are probably the most familiar examples.
The GNU C library includes support for two mechanisms for dealing with extended character sets: multibyte characters and wide characters. This chapter describes how to use these mechanisms, and the functions for converting between them. ‰IDX1539Š
@�”˜The behavior of the functions in this chapter is affected by the current locale for character classification--the HLC_CTYPE@ category; see section ‰ŠP�”›Categories of Activities that Locales Affect@�”˜. This choice of locale selects which multibyte code is used, and also controls the meanings and characteristics of wide character codes.
ˆ‰SEC312Št�”›Introduction to Extended Characters

@�”˜You can represent extended characters in either of two ways: þÂ
• As Bmultibyte characters@ which can be embedded in an ordinary string, an array of Hchar@ objects. Their advantage is that many programs and operating systems can handle occasional multibyte characters scattered among ordinary ASCII characters, without any change.
• ‰IDX1540Š@�”˜As Bwide characters@, which are like ordinary characters except that they occupy more bits. The wide character data type, Hwchar_t@, has a range large enough to hold extended character codes as well as old-fashioned ASCII codes. An advantage of wide characters is that each character is a single data object, just like ordinary ASCII characters. There are a few disadvantages: Ä
• Each existing program must be modified and recompiled to make it use wide characters.
• Files of wide characters cannot be read by programs that expect ordinary characters.Â Àð
Typically, you use the multibyte character representation as part of the external program interface, such as reading or writing text to files. However, it's usually easier to perform internal manipulations on strings containing extended characters on arrays of Hwchar_t@ objects, since the uniform representation makes most editing operations easier. If you do use multibyte characters for files and wide characters for internal operations, you need to convert between them when you read and write data.
If your system supports extended characters, then it supports them both as multibyte characters and as wide characters. The library includes functions you can use to convert between the two representations. These functions are described in this chapter.
ˆ‰SEC313Št�”›Locales and Extended Characters

@�”˜A computer system can support more than one multibyte character code, and more than one wide character code. The user controls the choice of codes through the current locale for character classification (see section ‰ŠP�”›Locales and Internationalization@�”˜). Each locale specifies a particular multibyte character code and a particular wide character code. The choice of locale influences the behavior of the conversion functions in the library.
Some locales support neither wide characters nor nontrivial multibyte characters. In these locales, the library conversion functions still work, even though what they do is basically trivial.
If you select a new locale for character classification, the internal shift state maintained by these functions can become confused, so it's not a good idea to change the locale while you are in the middle of processing a string.
ˆ‰SEC314Št�”›Multibyte Characters

‰IDX1541Š
@�”˜In the ordinary ASCII code, a sequence of characters is a sequence of bytes, and each character is one byte. This is very simple, but allows for only 256 distinct characters.
In a Bmultibyte character code@, a sequence of characters is a sequence of bytes, but each character may occupy one or more consecutive bytes of the sequence.
‰IDX1542Š@�”˜There are many different ways of designing a multibyte character code; different systems use different codes. To specify a particular code means designating the Bbasic@ byte sequences--those which represent a single character--and what characters they stand for. A code that a computer can actually use must have a finite number of these basic sequences, and typically none of them is more than a few characters long.
These sequences need not all have the same length. In fact, many of them are just one byte long. Because the basic ASCII characters in the range from H0@ to H0177@ are so important, they stand for themselves in all multibyte character codes. That is to say, a byte whose value is H0@ through H0177@ is always a character in itself. The characters which are more than one byte must always start with a byte in the range from H0200@ through H0377@.
The byte value H0@ can be used to terminate a string, just as it is often used in a string of ASCII characters.
Specifying the basic byte sequences that represent single characters automatically gives meanings to many longer byte sequences, as more than one character. For example, if the two byte sequence H0205 049@ stands for the Greek letter alpha, then H0205 049 065@ must stand for an alpha followed by an H`A'@ (ASCII code 065), and H0205 049 0205 049@ must stand for two alphas in a row.
If any byte sequence can have more than one meaning as a sequence of characters, then the multibyte code is ambiguous--and no good. The codes that systems actually use are all unambiguous.
In most codes, there are certain sequences of bytes that have no meaning as a character or characters. These are called Binvalid@.
The simplest possible multibyte code is a trivial one:
Â
The basic sequences consist of single bytes.À

This particular code is equivalent to not using multibyte characters at all. It has no invalid sequences. But it can handle only 256 different characters.
Here is another possible code which can handle 9376 different characters:
Â
The basic sequences consist of þÄ
• single bytes with values in the range H0@ through H0237@.
• two-byte sequences, in which both of the bytes have values in the range from H0240@ through H0377@.Âð À

This code or a similar one is used on some systems to represent Japanese characters. The invalid sequences are those which consist of an odd number of consecutive bytes in the range from H0240@ through H0377@.
Here is another multibyte code which can handle more distinct extended characters--in fact, almost thirty million:
Â
The basic sequences consist of þÄ
• single bytes with values in the range H0@ through H0177@.
• sequences of up to four bytes in which the first byte is in the range from H0200@ through H0237@, and the remaining bytes are in the range from H0240@ through H0377@.Âð À

In this code, any sequence that starts with a byte in the range from H0240@ through H0377@ is invalid.
And here is another variant which has the advantage that removing the last byte or bytes from a valid character can never produce another valid character. (This property is convenient when you want to search strings for particular characters.)
Â
The basic sequences consist of þÄ
• single bytes with values in the range H0@ through H0177@.
• two-byte sequences in which the first byte is in the range from H0200@ through H0207@, and the second byte is in the range from H0240@ through H0377@.
• three-byte sequences in which the first byte is in the range from H0210@ through H0217@, and the other bytes are in the range from H0240@ through H0377@.
• four-byte sequences in which the first byte is in the range from H0220@ through H0227@, and the other bytes are in the range from H0240@ through H0377@.Âð À

The list of invalid sequences for this code is long and not worth stating in full; examples of invalid sequences include H0240@ and H0220 0300 065@.
The number of Bpossible@ multibyte codes is astronomical. But a given computer system will support at most a few different codes. (One of these codes may allow for thousands of different characters.) Another computer system may support a completely different code. The library facilities described in this chapter are helpful because they package up the knowledge of the details of a particular computer system's multibyte code, so your programs need not know them.
You can use special standard macros to find out the maximum possible number of bytes in a character in the currently selected multibyte code with HMB_CUR_MAX@, and the maximum for Bany@ multibyte code supported on your computer with HMB_LEN_MAX@.

AMacro:@ int DMB_LEN_MAX
Â‰IDX1543Š@�”˜This is the maximum length of a multibyte character for any supported locale. It is defined in H`limits.h'@. ‰IDX1544ŠÀ

A�”˜Macro:@ int DMB_CUR_MAX
Â‰IDX1545Š@�”˜This macro expands into a (possibly non-constant) positive integer expression that is the maximum number of bytes in a multibyte character in the current locale. The value is never greater than HMB_LEN_MAX@.
‰IDX1546ŠH�”˜MB_CUR_MAX@ is defined in H`stdlib.h'@.À

Normally, each basic sequence in a particular character code stands for one character, the same character regardless of context. Some multibyte character codes have a concept of Bshift state@; certain codes, called Bshift sequences@, change to a different shift state, and the meaning of some or all basic sequences varies according to the current shift state. In fact, the set of basic sequences might even be different depending on the current shift state. See section ‰ŠP�”›Multibyte Codes Using Shift Sequences@�”˜, for more information on handling this sort of code.
What happens if you try to pass a string containing multibyte characters to a function that doesn't know about them? Normally, such a function treats a string as a sequence of bytes, and interprets certain byte values specially; all other byte values are "ordinary". As long as a multibyte character doesn't contain any of the special byte values, the function should pass it through as if it were several ordinary characters.
For example, let's figure out what happens if you use multibyte characters in a file name. The functions such as Hopen@ and Hunlink@ that operate on file names treat the name as a sequence of byte values, with H`/'@ as the only special value. Any other byte values are copied, or compared, in sequence, and all byte values are treated alike. Thus, you may think of the file name as a sequence of bytes or as a string containing multibyte characters; the same behavior makes sense equally either way, provided no multibyte character contains a H`/'@.
ˆ‰SEC315Št�”›Wide Character Introduction

B�”˜Wide characters@ are much simpler than multibyte characters. They are simply characters with more than eight bits, so that they have room for more than 256 distinct codes. The wide character data type, Hwchar_t@, has a range large enough to hold extended character codes as well as old-fashioned ASCII codes.
An advantage of wide characters is that each character is a single data object, just like ordinary ASCII characters. Wide characters also have some disadvantages: þÂ
• A program must be modified and recompiled in order to use wide characters at all.
• Files of wide characters cannot be read by programs that expect ordinary characters.Àð
Wide character values H0@ through H0177@ are always identical in meaning to the ASCII character codes. The wide character value zero is often used to terminate a string of wide characters, just as a single byte with value zero often terminates a string of ordinary characters.

AData Type:@ Dwchar_t
Â‰IDX1547Š@�”˜This is the "wide character" type, an integer type whose range is large enough to represent all distinct values in any extended character set in the supported locales. See section ‰ŠP�”›Locales and Internationalization@�”˜, for more information about locales. This type is defined in the header file H`stddef.h'@. ‰IDX1548ŠÀ

@�”˜If your system supports extended characters, then each extended character has both a wide character code and a corresponding multibyte basic sequence.
‰IDX1549Š‰IDX1550Š@�”˜In this chapter, the term Bcode@ is used to refer to a single extended character object to emphasize the distinction from the Hchar@ data type.
ˆ‰SEC316Št�”›Conversion of Extended Strings

‰IDX1551Š‰IDX1552Š
‰IDX1553Š@�”˜The Hmbstowcs@ function converts a string of multibyte characters to a wide character array. The Hwcstombs@ function does the reverse. These functions are declared in the header file H`stdlib.h'@.
In most programs, these functions are the only ones you need for conversion between wide strings and multibyte character strings. But they have limitations. If your data is not null-terminated or is not all in core at once, you probably need to use the low-level conversion functions to convert one character at a time. See section ‰ŠP�”›Conversion of Extended Characters One by One@�”˜.

AFunction:@ size_t Dmbstowcs@ B(wchar_t *Bwstring@, const char *Bstring@, size_t Bsize@)
Â‰IDX1554Š@�”˜The Hmbstowcs@ ("multibyte string to wide character string") function converts the null-terminated string of multibyte characters Bstring@ to an array of wide character codes, storing not more than Bsize@ wide characters into the array beginning at Bwstring@. The terminating null character counts towards the size, so if Bsize@ is less than the actual number of wide characters resulting from Bstring@, no terminating null character is stored.
The conversion of characters from Bstring@ begins in the initial shift state.
If an invalid multibyte character sequence is found, this function returns a value of H-1@. Otherwise, it returns the number of wide characters stored in the array Bwstring@. This number does not include the terminating null character, which is present if the number is less than Bsize@.
Here is an example showing how to convert a string of multibyte characters, allocating enough space for the result. wchar_t *
mbstowcs_alloc (const char *string)
{
size_t size = strlen (string) + 1;
wchar_t *buf = xmalloc (size * sizeof (wchar_t));

size = mbstowcs (buf, string, size);
if (size == (size_t) -1)
return NULL;
buf = xrealloc (buf, (size + 1) * sizeof (wchar_t));
return buf;
} À

AFunction:@ size_t Dwcstombs@ B(char *Bstring@, const wchar_t Bwstring@, size_t Bsize@)
Â‰IDX1555Š@�”˜The Hwcstombs@ ("wide character string to multibyte string") function converts the null-terminated wide character array Bwstring@ into a string containing multibyte characters, storing not more than Bsize@ bytes starting at Bstring@, followed by a terminating null character if there is room. The conversion of characters begins in the initial shift state.
The terminating null character counts towards the size, so if Bsize@ is less than or equal to the number of bytes needed in Bwstring@, no terminating null character is stored.
If a code that does not correspond to a valid multibyte character is found, this function returns a value of H-1@. Otherwise, the return value is the number of bytes stored in the array Bstring@. This number does not include the terminating null character, which is present if the number is less than Bsize@.À

ˆ‰SEC317Št�”›Multibyte Character Length

‰IDX1556Š‰IDX1557Š
@�”˜This section describes how to scan a string containing multibyte characters, one character at a time. The difficulty in doing this is to know how many bytes each character contains. Your program can use Hmblen@ to find this out.

AFunction:@ int Dmblen@ B(const char *Bstring@, size_t Bsize@)
Â‰IDX1558Š@�”˜The Hmblen@ function with a non-null Bstring@ argument returns the number of bytes that make up the multibyte character beginning at Bstring@, never examining more than Bsize@ bytes. (The idea is to supply for Bsize@ the number of bytes of data you have in hand.)
The return value of Hmblen@ distinguishes three possibilities: the first Bsize@ bytes at Bstring@ start with valid multibyte character, they start with an invalid byte sequence or just part of a character, or Bstring@ points to an empty string (a null character).
For a valid multibyte character, Hmblen@ returns the number of bytes in that character (always at least H1@, and never more than Bsize@). For an invalid byte sequence, Hmblen@ returns H-1@. For an empty string, it returns H0@.
If the multibyte character code uses shift characters, then Hmblen@ maintains and updates a shift state as it scans. If you call Hmblen@ with a null pointer for Bstring@, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See section ‰ŠP�”›Multibyte Codes Using Shift Sequences@�”˜.
‰IDX1559Š@�”˜The function Hmblen@ is declared in H`stdlib.h'@.À

ˆ‰SEC318Št�”›Conversion of Extended Characters One by One

‰IDX1560Š‰IDX1561Š
‰IDX1562Š@�”˜You can convert multibyte characters one at a time to wide characters with the Hmbtowc@ function. The Hwctomb@ function does the reverse. These functions are declared in H`stdlib.h'@.

AFunction:@ int Dmbtowc@ B(wchar_t *Bresult@, const char *Bstring@, size_t Bsize@)
Â‰IDX1563Š@�”˜The Hmbtowc@ ("multibyte to wide character") function when called with non-null Bstring@ converts the first multibyte character beginning at Bstring@ to its corresponding wide character code. It stores the result in H*Jresult@.
Hmbtowc@ never examines more than Bsize@ bytes. (The idea is to supply for Bsize@ the number of bytes of data you have in hand.)
Hmbtowc@ with non-null Bstring@ distinguishes three possibilities: the first Bsize@ bytes at Bstring@ start with valid multibyte character, they start with an invalid byte sequence or just part of a character, or Bstring@ points to an empty string (a null character).
For a valid multibyte character, Hmbtowc@ converts it to a wide character and stores that in H*Jresult@, and returns the number of bytes in that character (always at least H1@, and never more than Bsize@).
For an invalid byte sequence, Hmbtowc@ returns H-1@. For an empty string, it returns H0@, also storing H0@ in H*Jresult@.
If the multibyte character code uses shift characters, then Hmbtowc@ maintains and updates a shift state as it scans. If you call Hmbtowc@ with a null pointer for Bstring@, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See section ‰ŠP�”›Multibyte Codes Using Shift Sequences@�”˜.À

AFunction:@ int Dwctomb@ B(char *Bstring@, wchar_t Bwchar@)
Â‰IDX1564Š@�”˜The Hwctomb@ ("wide character to multibyte") function converts the wide character code Bwchar@ to its corresponding multibyte character sequence, and stores the result in bytes starting at Bstring@. At most HMB_CUR_MAX@ characters are stored.
Hwctomb@ with non-null Bstring@ distinguishes three possibilities for Bwchar@: a valid wide character code (one that can be translated to a multibyte character), an invalid code, and H0@.
Given a valid code, Hwctomb@ converts it to a multibyte character, storing the bytes starting at Bstring@. Then it returns the number of bytes in that character (always at least H1@, and never more than HMB_CUR_MAX@).
If Bwchar@ is an invalid wide character code, Hwctomb@ returns H-1@. If Bwchar@ is H0@, it returns H0@, also storing H0@ in H*Jstring@.
If the multibyte character code uses shift characters, then Hwctomb@ maintains and updates a shift state as it scans. If you call Hwctomb@ with a null pointer for Bstring@, that initializes the shift state to its standard initial value. It also returns nonzero if the multibyte character code in use actually has a shift state. See section ‰ŠP�”›Multibyte Codes Using Shift Sequences@�”˜.
Calling this function with a Bwchar@ argument of zero when Bstring@ is not null has the side-effect of reinitializing the stored shift state Bas well as@ storing the multibyte character H0@ and returning H0@.À

ˆ‰SEC319Št�”›Character-by-Character Conversion Example

@�”˜Here is an example that reads multibyte character text from descriptor Hinput@ and writes the corresponding wide characters to descriptor Houtput@. We need to convert characters one by one for this example because Hmbstowcs@ is unable to continue past a null character, and cannot cope with an apparently invalid partial character by reading more input. int
file_mbstowcs (int input, int output)
{
char buffer[BUFSIZ + MB_LEN_MAX];
int filled = 0;
int eof = 0;

while (!eof)
{
int nread;
int nwrite;
char *inp = buffer;
wchar_t outbuf[BUFSIZ];
wchar_t *outp = outbuf;

/* Fill up the buffer from the input file. */
nread = read (input, buffer + filled, BUFSIZ);
if (nread < 0)
{
perror ("read");
return 0;
}
/* If we reach end of file, make a note to read no more. */
if (nread == 0)
eof = 1;

/* Hfilled@ is now the number of bytes in Hbuffer@. */
filled += nread;

/* Convert those bytes to wide characters--as many as we can. */
while (1)
{
int thislen = mbtowc (outp, inp, filled);
/* Stop converting at invalid character;
this can mean we have read just the first part
of a valid character. */
if (thislen == -1)
break;
/* Treat null character like any other,
but also reset shift state. */
if (thislen == 0) {
thislen = 1;
mbtowc (NULL, NULL, 0);
}
/* Advance past this character. */
inp += thislen;
filled -= thislen;
outp++;
}

/* Write the wide characters we just made. */
nwrite = write (output, outbuf,
(outp - outbuf) * sizeof (wchar_t));
if (nwrite < 0)
{
perror ("write");
return 0;
}

/* See if we have a Breal@ invalid character. */
if ((eof && filled > 0) || filled >= MB_CUR_MAX)
{
error ("invalid multibyte character");
return 0;
}

/* If any characters must be carried forward,
put them at the beginning of Hbuffer@. */
if (filled > 0)
memcpy (inp, buffer, filled);
}
}

return 1;
}
ˆ‰SEC320Št�”›Multibyte Codes Using Shift Sequences

@�”˜In some multibyte character codes, the Bmeaning@ of any particular byte sequence is not fixed; it depends on what other sequences have come earlier in the same string. Typically there are just a few sequences that can change the meaning of other sequences; these few are called Bshift sequences@ and we say that they set the Bshift state@ for other sequences that follow.
To illustrate shift state and shift sequences, suppose we decide that the sequence H0200@ (just one byte) enters Japanese mode, in which pairs of bytes in the range from H0240@ to H0377@ are single characters, while H0201@ enters Latin-1 mode, in which single bytes in the range from H0240@ to H0377@ are characters, and interpreted according to the ISO Latin-1 character set. This is a multibyte code which has two alternative shift states ("Japanese mode" and "Latin-1 mode"), and two shift sequences that specify particular shift states.
When the multibyte character code in use has shift states, then Hmblen@, Hmbtowc@ and Hwctomb@ must maintain and update the current shift state as they scan the string. To make this work properly, you must follow these rules: þÂ
• Before starting to scan a string, call the function with a null pointer for the multibyte character address--for example, Hmblen (NULL, 0)@. This initializes the shift state to its standard initial value.
• Scan the string one character at a time, in order. Do not "back up" and rescan characters already scanned, and do not intersperse the processing of different strings.Àð
Here is an example of using Hmblen@ following these rules: void
scan_string (char *s)
{
int length = strlen (s);

/* Initialize shift state. */
mblen (NULL, 0);

while (1)
{
int thischar = mblen (s, length);
/* Deal with end of string and invalid characters. */
if (thischar == 0)
break;
if (thischar == -1)
{
error ("invalid multibyte character");
break;
}
/* Advance past this character. */
s += thischar;
length -= thischar;
}
}
The functions Hmblen@, Hmbtowc@ and Hwctomb@ are not reentrant when using a multibyte code that uses a shift state. However, no other library functions call these functions, so you don't have to worry that the shift state will be changed mysteriously.
Œ
Go to the ‰ŠP�”›first@�”˜, ‰ŠP�”›previous@�”˜, ‰ŠP�”›next@�”˜, ‰ŠP�”›last@�”˜ section, ‰ŠP�”›table of contents@�”˜.