It may not be clear till you read all the comments on the bug that the ９ which isn't matched is a FULLWIDTH DIGIT NINE.
If you're using, e.g., [a-fA-F0-9] in any locale other than C/POSIX, you're going to have a bad time.
This is going to cause problems, and it's a shame the developers note this, but then seem to wash their hands of it. Perhaps glibc should do what the programmer means, not what they say. It's obvious the regex above intends to validate hex digits, so glibc should do that, standards (and locale) be damned, unless the programmer explictly opts in to the technically correct behavior. The documentation even says:
> Therefore, using [a-z] does not make much sense except in the C/POSIX locale.
If that range doesn't make sense except in the C/POSIX locale, then why interpret it in any other locale? Come on glibc... help us out here.
This suggests the only safe portable way to represent hex is one of these? [1234567890abcdefABCDEF] or [\dabcdefABCDEF]
That sucks. Fortunately I don't use C, but I'm worried about this finding its way into other languages...
Javascript defines regex ranges in terms of UTF-16 code units w/no consideration for locale, so you can use [a-fA-F0-9] and it will work as expected.
The issue discussed in this bug is only relevant to POSIX regular expression ranges.
Perhaps you're lamenting that there are so many different flavors of regular expression. I agree. Just the other day I had to give up on using a regex in a CloudFormation template to validate input because I could not get it to work as documented.
This is bizarre. I thought [0-9] was supposed to match only 0-9 (the 10 digits), while \d was meant to match all digits including the various Unicode variations.
This is true for many regular expression dialects.
But this bug about POSIX regular expressions. POSIX doesn't define \d, and it defines ranges such as [0-9] only in the POSIX locale.
Historically, glibc considered such ranges in the context of locale's collation order. For example, in Estonian locale, [a-z] doesn't include the letters tuvwxy, because that's how Estonian alphabet works: https://en.wikipedia.org/wiki/Estonian_orthography#Alphabet
This (somewhat surprising) semantics of character ranges is not new, but the recent changes in glibc made it more spectacular.
Yes, [0-9] is supposed to be locale independent, which is related to the fact that ISO C requires the digit characters '0' through '9' to be consecutive.
The same is not true of [A-Z], which doesn't necessarily include just the upper-case English letters from ASCII. It is locale-specific, in fact.
The POSIX digit matcher is [[:digit:]]. It is related to the C isdigit function from <ctype.h>. Those functions have regex counterparts, like isalnum -> [[:alnum:]] and so on.
The way those functions have locale-dependent behavior is linked to how the corresponding regex classes have locale-dependent behavior.
So where are glibc regexps used in the most typical practical cases?
At least grep does not use them (at least not in unmodified form) as a comment in the bug report notes.
I think changing such behavior is unacceptable, regardless of what the spec says. Here I would to propose the API stability promise of the Linux kernel: We don't break existing programs.
If somebody notes that the previous behavior is not correct according to the spec a new posixly correct mode can be introduced. But it should not magically become default.
Does this "pass through" to the regex implementations in any other languages, or do those tend to implement their own parser or integrate different libraries?
Few languages use glibc for their regex implementation. PCRE is probably the most commonly used third-party regex lib, but it depends. Python and a bunch of languages have their own implementations.
In all the scripted tests we write, one of the first lines will be:
export LANG=C
A similar construct is also used in most places where we call sort (except where we're actually sorting strings of text that will be read by an end user), eg:
This works till you want at least minimal UTF-8 support, such as when you're working with actual English text as opposed to human-readable computerese.
Sticking to the "C" locale will work even if you want UTF-8 support. You just roll your own UTF-8 support.
I built the TXR language entirely without any of the harmful garbage that is the ISO C/POSIX localization. It handles UTF-8 just fine.
The C localization stuff was developed too early, at a time when nobody had any real experience with localization. Before Unicode, before the Internet.
Before threads! How the do you set it up so that one thread runs in one locale and another in another? That's important if a global server is servicing two different requests simultaneously from users in two different locales. The idiotic C locale stuff relies on global variables.
You set some magic variables and, poof; numerous functions in your entire image change their behavior, whether they are working with internationalized data or not.
The setlocale function might as well be called fuck_my_program_please.
> I guess we should all just force-set our C programs to the C locale.
Ah, but you are in luck: this is not required!
A C program begins execution in the "C" locale, regardless of the environment variables.
The locale variables do not affect a C program until it calls the "fuck this program function" called setlocale, with LC_ALL, and "". The empty string argument means "use the environment variable".
The problem is that many modern utilities in GNU/Linux land all make this setlocale call, in order to support internationalization. So then they are affected by whatever hell breaks loose.
Your own program is safe, if neither it, nor a third party library, calls setlocale, then your strcmp and isalpha and whatever else behave in good old naive manner.
It may not be clear till you read all the comments on the bug that the ９ which isn't matched is a FULLWIDTH DIGIT NINE.
If you're using, e.g., [a-fA-F0-9] in any locale other than C/POSIX, you're going to have a bad time.
This is going to cause problems, and it's a shame the developers note this, but then seem to wash their hands of it. Perhaps glibc should do what the programmer means, not what they say. It's obvious the regex above intends to validate hex digits, so glibc should do that, standards (and locale) be damned, unless the programmer explictly opts in to the technically correct behavior. The documentation even says:
> Therefore, using [a-z] does not make much sense except in the C/POSIX locale.
If that range doesn't make sense except in the C/POSIX locale, then why interpret it in any other locale? Come on glibc... help us out here.
$0.02.
I agree. I actually thought this bug report was a late April fools joke.
At least make the new behavior optional and off by default. Iirc, some languages do this by requiring a modifier to the regexp, like /[0-9]/u
Absolutely. Using a 'range' should be an actual range, not a category.
This suggests the only safe portable way to represent hex is one of these? [1234567890abcdefABCDEF] or [\dabcdefABCDEF] That sucks. Fortunately I don't use C, but I'm worried about this finding its way into other languages...
For a POSIX regex, the correct way to match a hex digit is to use is [[:xdigit:]].
Right - would be great if more languages supported that, specifically Javascript.
Javascript defines regex ranges in terms of UTF-16 code units w/no consideration for locale, so you can use [a-fA-F0-9] and it will work as expected.
The issue discussed in this bug is only relevant to POSIX regular expression ranges.
Perhaps you're lamenting that there are so many different flavors of regular expression. I agree. Just the other day I had to give up on using a regex in a CloudFormation template to validate input because I could not get it to work as documented.
Followup. The glibc developers came up with a solution that meets the standards without violating the principle of least surprise.
See https://sourceware.org/bugzilla/show_bug.cgi?id=23393#c28 and https://www.sourceware.org/ml/libc-alpha/2018-07/msg00620.ht...
This is bizarre. I thought [0-9] was supposed to match only 0-9 (the 10 digits), while \d was meant to match all digits including the various Unicode variations.
0-9 is defined as a [ascii] range, so you are correct.
This is true for many regular expression dialects.
But this bug about POSIX regular expressions. POSIX doesn't define \d, and it defines ranges such as [0-9] only in the POSIX locale.
Historically, glibc considered such ranges in the context of locale's collation order. For example, in Estonian locale, [a-z] doesn't include the letters tuvwxy, because that's how Estonian alphabet works: https://en.wikipedia.org/wiki/Estonian_orthography#Alphabet
This (somewhat surprising) semantics of character ranges is not new, but the recent changes in glibc made it more spectacular.
\d is a Perlism.
Yes, [0-9] is supposed to be locale independent, which is related to the fact that ISO C requires the digit characters '0' through '9' to be consecutive.
The same is not true of [A-Z], which doesn't necessarily include just the upper-case English letters from ASCII. It is locale-specific, in fact.
The POSIX digit matcher is [[:digit:]]. It is related to the C isdigit function from <ctype.h>. Those functions have regex counterparts, like isalnum -> [[:alnum:]] and so on.
The way those functions have locale-dependent behavior is linked to how the corresponding regex classes have locale-dependent behavior.
Reminds me of Ruby's regexp behaviour at one point.
With ignorecase, !\W didn't match 'k' and 's' (only!)
https://bugs.ruby-lang.org/issues/4044
So where are glibc regexps used in the most typical practical cases?
At least grep does not use them (at least not in unmodified form) as a comment in the bug report notes.
I think changing such behavior is unacceptable, regardless of what the spec says. Here I would to propose the API stability promise of the Linux kernel: We don't break existing programs.
If somebody notes that the previous behavior is not correct according to the spec a new posixly correct mode can be introduced. But it should not magically become default.
Does this "pass through" to the regex implementations in any other languages, or do those tend to implement their own parser or integrate different libraries?
Few languages use glibc for their regex implementation. PCRE is probably the most commonly used third-party regex lib, but it depends. Python and a bunch of languages have their own implementations.
I guess we should all just force-set our C programs to the C locale.
In all the scripted tests we write, one of the first lines will be:
A similar construct is also used in most places where we call sort (except where we're actually sorting strings of text that will be read by an end user), eg:
Setting LANG=C is not sufficient, because LANG can be overridden by various LC_* variables.
You should set LC_ALL=C instead.
LC_COLLATE is the critical category according to the bug report. No need to set anything else.
It's not enough if LC_ALL is already set in environment, sice that would override LC_COLLATE.
So just set LC_ALL and be done with it. It's bulletproof.
This works till you want at least minimal UTF-8 support, such as when you're working with actual English text as opposed to human-readable computerese.
Sticking to the "C" locale will work even if you want UTF-8 support. You just roll your own UTF-8 support.
I built the TXR language entirely without any of the harmful garbage that is the ISO C/POSIX localization. It handles UTF-8 just fine.
The C localization stuff was developed too early, at a time when nobody had any real experience with localization. Before Unicode, before the Internet.
Before threads! How the do you set it up so that one thread runs in one locale and another in another? That's important if a global server is servicing two different requests simultaneously from users in two different locales. The idiotic C locale stuff relies on global variables.
You set some magic variables and, poof; numerous functions in your entire image change their behavior, whether they are working with internationalized data or not.
The setlocale function might as well be called fuck_my_program_please.
> I guess we should all just force-set our C programs to the C locale.
Ah, but you are in luck: this is not required!
A C program begins execution in the "C" locale, regardless of the environment variables.
The locale variables do not affect a C program until it calls the "fuck this program function" called setlocale, with LC_ALL, and "". The empty string argument means "use the environment variable".
The problem is that many modern utilities in GNU/Linux land all make this setlocale call, in order to support internationalization. So then they are affected by whatever hell breaks loose.
Your own program is safe, if neither it, nor a third party library, calls setlocale, then your strcmp and isalpha and whatever else behave in good old naive manner.