Show HN: Mojibake – A low-level Unicode library written in C
mojibake.zaerl.comI've written Mojibake because I don't like the other Unicode libraries for Unicode support.
It consists of only two amalgamation files: mojibake.h and mojibake.c. I've added all the most important Unicode algorithms, such as normalization, case conversion, segmentation, bidirectional text, collation, confusable, and others.
I regularly test it in these OSes: Linux, macOS, FreeBSD, OpenBSD, NetBSD, and Windows 11.
You can find a WASM demo on that site of all the public API functions and the documentation. If you want to participate, feel free to do it. Any kind of help is welcome. Check the CONTRIBUTING.md and API.md files in the GitHub repository for instructions on how to do it.
I assume the submitter is also the author. If so, can you share about your motivation to write this library? For example, do you use it professionally or in hobby projects? Did you look at other libraries and think that you could do better? These are honest questions -- no trolling from me. I browsed the code and it looks very clean.
The API documention is so nice! It looks like index.html from https://mojibake.zaerl.com/ uses JavaScript to generate the page. Very cool, indeed.
Yes, I'm the author. I began writing this library because I needed something to handle Unicode for another project of mine (that I didn't finish), and I didn't like ICU4C or the other libraries I found.
It's a hobby project I wrote because I find the Unicode standard (sometimes unnecessary) complexity fascinating. And for other people to use, if needed.
The HTML page is static and generated at compile time. Every function has a little form to test the WASM function.
How does it compare with utf8proc [1]? I'm aware that Mojibake does a bit more than utf8proc (e.g. bi-di) but that seems marginal to me.
[1] https://juliastrings.github.io/utf8proc/
Mojibake handle UTF-8, UTF-16, UTF-16LE, UTF-16BE, UTF-32, UTF-32LE, UTF-32BE in input and output. Can be used as a simple file and handle a wider number of algorithms, such as the "confusable" one.
Love the amalgamation approach—the C/C++ ecosystem desperately needs cleaner, lightweight Unicode support without pulling in massive dependencies... thanks for sharing
Thank you very much. I've also started writing a smaller C++ wrapper so the user can use std::string_view for this, but I'm not the best C++ guy out there.
Not to bikeshed, but isn't the word "mojibake" synonymous with "when character encoding breaks"?
I wouldn’t call it synonymous as much as I’d call it its literal meaning.
https://jisho.org/word/%E6%96%87%E5%AD%97%E5%8C%96%E3%81%91
Mojibake is a Japanese term, and yes, it's used there when a page displays a lot of gibberish text. I chose it for fun.
what's performance like compared to python ftfy module?
Mojibake is a C library, not Python, so comparing the two isn't ideal. You can see the performance in the GitHub workflows results. At the end of the logs, there is an "execution time" line.
I have come to the conclusion that the only Unicode support needed in C is supporting pointers to char and arrays but lightweight C libraries are always welcome.
I guess you never have to deal with text if you think that’s enough? What kind of software do you write in C?