Wsrtjtyk

As we know, different encodings map different representations to same characters. Using setlocale we can specify the encoding of strings that are read from input, but does this apply to string literals as well? I'd find this surprising since these are compile-time!

This matters for tasks as simple as, for example, determining whether a string read from input contains a specific character. When reading strings from input it seems sensible to set the locale to to the user's locale (setlocale("LC_ALL", "");) so that the string is read and processed correctly. But when we're comparing this string with a character literal, won't problems arise due to mismatched encoding?

In other words: The following snippet seems to work for me. But doesn't it work only because of coincidence? Because - for example? - the source code happened to be saved in the same encoding that is used on the machine during runtime?

#include <stdio.h>

#include <wchar.h>

#include <stdlib.h>

#include <locale.h>



int main()

{

        setlocale(LC_ALL, "");



        // Read line and convert it to wide string so that wcschr can be used

        // So many lines! And that's even though I'm omitting the necessary

        // error checking for brevity. Ah I'm also omitting free's

        char *s = NULL; size_t n = 0;

        getline(&s, &n, stdin);

        mbstate_t st = {0}; const char* cs = s;

        size_t wn = mbsrtowcs(NULL, &cs, 0, &st);

        wchar_t *ws = malloc((wn+1) * sizeof(wchar_t));

        st = (mbstate_t){0};

        mbsrtowcs(ws, &cs, (wn+1), &st);



        int contains_guitar = (wcschr(ws, L'🎸') != NULL);

        if(contains_guitar)

                printf("Let's rock!n");

        else

                printf("Let's not.n");

        return 0;

}

How to do this correctly?

Using setlocale we can specify the encoding of strings that are read from input, but does this apply to string literals as well?

No. String literals use the execution character set, which is defined by your compiler at compile time.

Execution character set does not have to be the same as the source character set, the character set used in the source code. The C compiler is responsible for the translation, and should have options for choosing/defining them. The default depends on the compiler, but on Linux and most current POSIXy systems, is usually UTF-8.

The following snippet seems to work for me. But doesn't it work only because of coincidence?

The example works because the character set of your locale, the source character set, and the execution character set used when the binary was constructed, all happen to be UTF-8.

How to do this correctly?

Two options. One is to use wide characters and string literals. The other is to use UTF-8 everywhere.

For wide input and output, see e.g. this example in another answer here.

Do note that getwline() and getwdelim() are not in POSIX.1, but in C11 Annex K. This means they are optional, and as of this writing, not widely available at all. Thus, a custom implementation around fgetwc() is recommended instead. (One based on fgetws(), wcslen(), and/or wcscspn() will not be able to handle embedded nuls, L'', correctly.)

In a typical wide I/O program, you only need mbstowcs() to convert command-line arguments and environment variables to wide strings.

Using UTF-8 everywhere is also a perfectly valid practical approach, at least if it is well documented, so that users know the program inputs and outputs UTF-8 strings, and developers know to ensure their C compiler uses UTF-8 as the execution character set when compiling those binaries.

Your program can even use e.g.

    if (!setlocale(LC_ALL, ""))

        fprintf(stderr, "Warning: Your C library does not support your current locale.n");

    if (strcmp("UTF-8", nl_langinfo(CODESET)))

        fprintf(stderr, "Warning: Your locale does not use the UTF-8 character set.n");

to verify the current locale uses UTF-8.

I have used both approaches, depending on the circumstances. It is difficult to say which one is more portable in practice, because as usual, both work just fine on non-Windows OSes without issues.

If you're willing to assume UTF-8,

strstr(s,"🎸")

Or:

strstr(s,u8"🎸")

The latter avoids some assumptions but requires a C11 compiler. If you want the best of both and can sacrifice readability:

strstr(s,"360237216270")