With bitmasks, if they just assumed bit 5 (whatever) as zero=whitespace, they would have saved billions of cpu cycles per device per day. With 32 bit unicode, that gives you 32 unique whitespace chars.
Should have got guys that grew up with the 8-bit micros to write up a spec.
This code is very simple but I still don't really get it. Why is the hashing needed and how is it faster than a regex? Can someone smarter than me explain?
No branching (if statements), no heavy CPU operations, just a multiplication by a constant, a shift and a lookup in a small table that will get cached fast
You need a hashtable to check whether this character is part of a given set. And the perfect hashing-based implementation is the most efficient form of a hashtable since you only need a lookup and an equality check. I included a small visualisation in the article that illustrates how it works, so if you try clicking on the elements, it will show you how the calculations are made.
Whitespace depends on the context. For example, you could have characters that are not counted as whitespace in Unicode, but end up rendering as whitespace when displayed in a particular context. The only real way to check for this is to actually try to render it and see what happens. There are some other ways to approximate it, like checking the font that you use for rendering, or the bounds of the glyph, or other things like that, but the only way to really know for sure in all cases is to just render it and see what happens.
One example is a field that clips its contents to contain zalgo, but inadvertantly to the point where some glyphs can render completely out of bounds. Sometimes the clipping is so egregious that even descenders of normal letters like 'g' get cut off. (This is more common than you'd think.)
With font.color = white even an ordinary "W" could count as whitespace...
Something to remember if you see whitespace with an extraordinary width out there in the wild...
The \s character class in JavaScript regular expressions includes 25 characters. I don’t know if that perfectly matches the Unicode list of 25 characters, but it does make identifying white space simplistic.
White space characters are challenging to differentiate in human readable text, but JavaScript makes that perfectly clear to humans by splitting a string on every character, which is helpful for debugging.
The odd one out above is NEXT LINE (U+0085). Unicode explicitly includes it in White_Space, and Python accepts it too, but JavaScript doesn't. Instead, JavaScript recognises ZERO WIDTH NO-BREAK SPACE (U+FEFF), which Unicode doesn't include in the set.
With bitmasks, if they just assumed bit 5 (whatever) as zero=whitespace, they would have saved billions of cpu cycles per device per day. With 32 bit unicode, that gives you 32 unique whitespace chars.
Should have got guys that grew up with the 8-bit micros to write up a spec.
I think horizontal white spacing should be distinguished, as that's what most care about with splitting etc.
Have a look at how GNU coreutils classifies each type of spacing (in a cross platform consistent way) at https://github.com/coreutils/coreutils/blob/master/src%2Fsys...
This code is very simple but I still don't really get it. Why is the hashing needed and how is it faster than a regex? Can someone smarter than me explain?
No branching (if statements), no heavy CPU operations, just a multiplication by a constant, a shift and a lookup in a small table that will get cached fast
You need a hashtable to check whether this character is part of a given set. And the perfect hashing-based implementation is the most efficient form of a hashtable since you only need a lookup and an equality check. I included a small visualisation in the article that illustrates how it works, so if you try clicking on the elements, it will show you how the calculations are made.
> Why is the hashing needed and how is it faster than a regex? Can someone smarter than me explain?
It's a perfect hash (i.e. no collisions) that hash to an entry in a 32-element array.
Very similar to how we usually do quick lookups of strings for `switch()` statements: https://github.com/lelanthran/libxtree/blob/878cb4fc11ffd617...
Whitespace depends on the context. For example, you could have characters that are not counted as whitespace in Unicode, but end up rendering as whitespace when displayed in a particular context. The only real way to check for this is to actually try to render it and see what happens. There are some other ways to approximate it, like checking the font that you use for rendering, or the bounds of the glyph, or other things like that, but the only way to really know for sure in all cases is to just render it and see what happens.
My tab renders as: " > ". I still consider it whitespace.
What turns into whitespace due to rendering? Just because my font lacks that symbol doesn't mean it is whitespace.
One example is a field that clips its contents to contain zalgo, but inadvertantly to the point where some glyphs can render completely out of bounds. Sometimes the clipping is so egregious that even descenders of normal letters like 'g' get cut off. (This is more common than you'd think.)
I get the desire to filter that out, not sure whether I'd classify it as whitespace though.
With font.color = white even an ordinary "W" could count as whitespace... Something to remember if you see whitespace with an extraordinary width out there in the wild...
The \s character class in JavaScript regular expressions includes 25 characters. I don’t know if that perfectly matches the Unicode list of 25 characters, but it does make identifying white space simplistic.
White space characters are challenging to differentiate in human readable text, but JavaScript makes that perfectly clear to humans by splitting a string on every character, which is helpful for debugging.
The article covers that: