>I actually wrote a sed utility to collapse arbitrary whitespace
Can be done with tr, without writing a utility (if by "collapse" you mean what I think you do):
This file t:
$ cat t
the quick brown fox jumped
over
the lazy dog
containing many combinations of spaces, tabs and newlines (whitespace) can be changed to this file t2:
$ cat t2
the
quick
brown
fox
jumped
over
the
lazy
dog
by this tr command:
tr -cs "[a-zA-Z]" "\012" < t > t2
That also makes the output more amenable to further processing, including common tasks like finding the frequencies of the words in the input, as mentioned in the "More shell, less egg" post mentioned in this post:
The Bentley-Knuth problem and solutions:
https://jugad2.blogspot.com/2012/07/the-bentley-knuth-proble...
Does that work with utf8 files? I hardly ever work with us ascii files anymore.
Don't know. It was a while back, and in Python 2 (if you mean the Python version I wrote). I didn't take any special steps to support Unicode, so likely not. Same for the shell version I wrote.
Just realized you may have meant the tr command I used - whether it supports UTF-8. Don't know about that either.
I should've been more explicit, but was thinking about: tr -cs "[a-zA-Z]" "\012" < t > t2
(which I somehow managed to read as an awk invocation).
Re: python - I believe if using things like \w, \d or \s you should be Unicode safe.
Come to think of it, I seem to recall gnu tools should also have Unicode aware pattern/character classes, eg:
https://www.gnu.org/software/gawk/manual/html_node/Bracket-E...
> For example, before the POSIX standard, you had to write /[A-Za-z0-9]/ to match alphanumeric characters. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/ to match the alphabetic and numeric characters in your character set.
https://docs.python.org/3/library/re.html#re-syntax
[ed:
Apparently gnu tr is still out in the cold re:unicode;
https://www.gnu.org/software/coreutils/manual/html_node/tr-i...
> Currently tr fully supports only single-byte characters. Eventually it will support multibyte characters; when it does, the -C option will cause it to complement the set of characters, whereas -c will cause it to complement the set of values. This distinction will matter only when some values are not characters, and this is possible only in locales using multibyte encodings when the input contains encoding errors.
]
Got it. Good info, thanks.