Show HN: Distributed JMAP and IMAP Servers in Rust

github.com

190 points by StalwartLabs 14 days ago

I am happy to announce Stalwart JMAP [1], an open-source JSON Meta Application Protocol server that aims to be scalable, robust and secure. Some of its key features are:

- JMAP Core, JMAP Mail and JMAP over WebSocket full compliance. - IMAP4 rev2/1 support via Stalwart IMAP, an imap-to-jmap proxy [2]. - Scalable and fault tolerant: consensus over Raft, node autodiscovery over gossip and read-only replicas. - RocksDB backend with full-text search support in 17 languages. - OAuth 2.0 authorization code and device authorization flows. - Domain Keys Identified Mail (DKIM) message signing. - Written in Rust. - No third-party software required to run or scale.

The next item on the roadmap is to release an SMTP server in Rust with the goal of making self-hosting an e-mail server much simpler.

Any comments or suggestions are more than welcome!

[1]: https://github.com/stalwartlabs/jmap-server [2]: https://github.com/stalwartlabs/imap-server

superboum 14 days ago

Hi, I would like to mention that some work on a Rust SMTP server has been already done with the Kannader project[1] (Disclaimer: I have no contribution on it but I know the maintainer).

I also work on a Rust IMAP server that is far from being as feature complete as yours. I also chose your `mail-parser` library to parse RFC822/5822, but we observed that in many cases, we did not have enough information to build some BODY/BODYSTRUCTURE responses. We also discovered that line count and many details are not very obvious on IMAP, did you run some tests to compare your IMAP server outputs to existing servers? Or, more generally, what is your approach to ensure compatibility / integration with the existing email ecosystem?

In any case, congratulation for your project, we will follow it closely! I experimented how big these protocols became with all their extensions, this is an impressive work!

[1]: https://github.com/Ekleog/kannader

  • StalwartLabs 14 days ago

    > Or, more generally, what is your approach to ensure compatibility / integration with the existing email ecosystem?

    Yes, Stalwart IMAP server's compliance to the IMAP4 protocol was tested using Dovecot's ImapTest tool.

lxpz 14 days ago

Very interesting project and impressive work, thanks for sharing!

Can you talk a bit about how data is replicated between nodes when Stalwart is run in clustered mode, and what kind of data integrity/resilience properties we have when one, two, several nodes go down?

Also, have you considered implementing server-side encryption of e-mail messages so that a "honest but curious" system administrators could not read user's messages? (e.g. using the user's password to derive an encryption key). More generally, what are your thoughts on the "privacy" aspect?

  • StalwartLabs 14 days ago

    > Can you talk a bit about how data is replicated between nodes when Stalwart is run in clustered mode, and what kind of data integrity/resilience properties we have when one, two, several nodes go down?

    Data is replicated using the Raft consensus protocol and when multiple nodes go down the cluster will keep keep active unless there are not enough nodes to guarantee consistency. More details can be found on the documentation [1] but I plan to add more details on how replication works once the server passes the Jepsen tests.

    > Also, have you considered implementing server-side encryption of e-mail messages so that a "honest but curious" system administrators could not read user's messages? (e.g. using the user's password to derive an encryption key). More generally, what are your thoughts on the "privacy" aspect?

    Yes, in addition to server-side encryption also S/MIME and PGP are on the roadmap.

    [1] https://stalw.art/jmap/cluster/quick-start/

andrenth 13 days ago

I see that quota and sieve filtering are on the pipeline, which are two requirements I’d have for switching to a different IMAP server. Do you have any plans for remote authentication support (e.g. via SQL or maybe saslauthd)?

  • StalwartLabs 13 days ago

    Yes, SASL support is planned. SQL auth support could be added as well if there is interest.

    • jerrac 13 days ago

      I currently use Vimbadmin to manage domains and addresses for my personal email server. So I'd need to be able to keep using it, or be able to replace it for both the new jmap/imap server and postfix. SQL support would let me continue using it.

      That said, I think I'll find some time to give Stalwart JMAP a try. I've been curious about JMAP for a while, and there are a couple things about Dovecot I'm not too fond of...

    • zimpenfish 8 days ago

      Another vote for SQL support - I use sqlite as my user backend for Dovecot. Definitely interested in giving Stalwart a try though.

ibotty 14 days ago

This is looking great. Of course, this being new, there are some essential questions not answered yet.

What's the backup story like? Can the whole state be restored from the mail storage (I assume not)? Do you support "master users" (as in dovecot) so that I can do a continuous backup with dovecot sync?

Is it possible to use any OIDC server on the IMAP proxy? Do you support password authentication for legacy applications (preferably application and device specific passwords)?

Also, is it possible to export the mails *and* the state, so if this project does not work out, there is a way out? (Here dovecot sync might also work.)

  • StalwartLabs 13 days ago

    > What's the backup story like? Can the whole state be restored from the mail storage (I assume not)? Do you support "master users" (as in dovecot) so that I can do a continuous backup with dovecot sync?

    At the moment backing up the raw messages can be done by copying the blobs directory. However backing up the metadata (which is stored on RocksDB) is not yet supported but will be added on the next release. RocksDB has support for checkpoints and backups so adding the backup functionality is pretty straight-forward. In Stalwart JMAP there is a single master user which is the administrator. Continuous backup will be implemented as a housekeeper task which can be run on a schedule or manually triggered by the administrator.

    > Is it possible to use any OIDC server on the IMAP proxy?

    The IMAP proxy supports the OAUTHBEARER authentication scheme but using third-party OIDC server is not supported at the moment. However, once the SMTP server is out I plan to add support for other SASL mechanisms on the IMAP proxy (since some of the work will be shared with the SMTP Auth module).

    > Do you support password authentication for legacy applications (preferably application and device specific passwords)?

    Yes, the IMAP proxy supports both the LOGIN and AUTH=PLAIN mechanisms.

    > Also, is it possible to export the mails and the state, so if this project does not work out, there is a way out?

    E-mails can already be exported by copying the blobs directory (only the raw messages are stored under that directory). To export the metadata and folder structure any IMAP backup tool or service could be used.

    • ibotty 13 days ago

      Thank you for your answers.

      > The IMAP proxy supports the OAUTHBEARER authentication scheme but using third-party OIDC server is not supported at the moment. However, once the SMTP server is out I plan to add support for other SASL mechanisms on the IMAP proxy (since some of the work will be shared with the SMTP Auth module).

      It would be great to allow another OIDC server (to have single sign on and MFA) without having to run another SASL server.

      > Yes, the IMAP proxy supports both the LOGIN and AUTH=PLAIN mechanisms.

      How are application-device-specific passwords handled? Is there some documentation?

      • StalwartLabs 13 days ago

        > How are application-device-specific passwords handled? Is there some documentation?

        Not sure what do you mean with application device specific passwords? Currently Stalwart JMAP only allows registered accounts to login using a password which is stored encrypted with Argon2. Authentication can be done using the OAuth or Basic mechanisms. There are no additional passwords specific to a particular device. Not sure if this answers your question though.

egberts1 13 days ago

As someone who is literally an old-school hacker gone semi-retired, can someone give me the rationale for going all-JSON?

As a serializer/deserializer efficiency goes, wouldn’t Binary Serial Object Notation (BSON) have made more sense?

However, I am duly impressed with how Stalw.Art takes it to the next level with their consolidation of many protocols under one build tool by language and fewest toolset needed (no Autotool, grep, awk, automake, multi-C suppprt, Flex, Bison).

A very nice job.

  • calvinmorrison 13 days ago

    Tooling around json has universal tooling, sanely structured, and understood, basically the opposite of anything with IMAP.

    Probably any structured data like json would be fine, it's basically bikeshedding to argue about json when the real driver behind JMAP is the previous state of email's crustiness

    • egberts1 13 days ago

      And crustiness of legacy SMTP is very prevalent, notably the RFC 822 common mixed-text content (Text/HTML).

      Maybe the old foggity in me is yearning for the good ol’ Text-only SMTP days that were spam-free.

      Excuse the pessimism here but we are still perpetuating the weakness of SMTP-related backends, notably HTML inclusion into RFC 822, no?

      I see increasing prevalence of link tracking of emails by those who commercialized IMAP … and now JMAP. However, I hope to see more private mail servers but that future of privacy remains cloudy.

      • wtetzner 13 days ago

        Isn't link tracking orthogonal to IMAP? If you don't follow the links/load remote images/etc. on the client, then tracking doesn't occur. Unless I'm missing something?

  • danielheath 13 days ago

    Lots of people use webmail. JMAP was built with that in mind; shipping a BSON decoder isn't meaningfully more efficient than using the JSON encoder the client already has.

andrewshadura 13 days ago

It's great to see more server support for JMAP, but the client support doesn't seem to be catching up. Neither Thunderbird nor K-9 support it yet, and I don't know of any mainstream client that does, really.

As I user of Fastmail I don't care that much yet, as I can use the Fastmail app and any of the IMAP apps, but if I were to move to a self-hosted solution, that would be a big deal.

miracle2k 14 days ago

I'm grateful to everyone working on the JMAP ecosystem. Thank you, these look great.

alberth 13 days ago

Thanks you Fastmail for JMAP.

For those unaware, Fastmail was the party who created, built and open sourced JMAP (with many others contributing).

This took over 5-years of work for Fastmail to complete and projects like above wouldn’t have been possible without them.

We take for granted that no other company is advancing the development of email and its open standards, without it being behind some proprietary paywall (Eg Google, Microsoft, HEY).

If you haven’t tried Fastmail, I encourage you to do so. Their web mail client is fantastic (and unbelievably fast).

https://fastmail.blog/open-technologies/jmap-new-email-open-...

emacsen 13 days ago

This is a bit tangential, but I'm curious about the migration path from, say Dovecot to something like this.

I have a Dovecot mail store ~(10gb of mail) on Mailcow (which is Postfix and Dovecot under the covers) and tools like mbsync fail to pull all my mail without multiple invocations. I've been looking for a migration path off it.

When I migrated to this system, I found an IMAP to IMAP script that I had to fiddle with and hack, which tool roughly a day to complete. I'm hoping for a more straightforward migration path in the future.

  • StalwartLabs 13 days ago

    Migration instructions are covered here:

    https://stalw.art/jmap/migrate/overview/

    • emacsen 13 days ago

      This is great, and I'm genuinely excited by the prospect of a modern set of tools to replace some of these existing systems which, while rock solid for decades, are beginning to show their age.

      I think once things are a bit more settled, I'll be interested in the client end of things... That's something that I didn't see much on the Stalwart website. Yes, I'm sure I could run the same set of tools now, but that seems like a waste of an opportunity.

NetOpWibby 14 days ago

I’ve been looking for a Mail-in-a-Box for JMAP for years! Thank you for this. Looking forward to the SMTP server release.

  • sanxiyn 14 days ago

    For people who don't know: there is literally a project named "Mail-in-a-Box", which you may want to check out: https://mailinabox.email/

    • Gigachad 13 days ago

      Ran this for years. Works well. Eventually gave up because I couldn’t be bothered updating Ubuntu and my vps eventually ran out of space.

      • NetOpWibby 13 days ago

        Dang! I'm still running mine but boy, I sure wish I could play around with JMAP instead.

amaccuish 12 days ago

Maybe the wrong place to ask but... This seems perfect for us, but are there plans to support things like LDAP or even GSSAPI auth?

pingiun 13 days ago

This is very cool! I had always hoped someone would build a native JMAP server after I read the posts from fastmail

X-Istence 13 days ago

The key thing I am missing is how do I get my mail to flow from Postfix into this?

The docs are missing the most important part...

  • ibotty 13 days ago

    It has an LMTP endpoint.

robert_foss 14 days ago

This is really exciting stuff. I can't wait to switch over to JMAP.

ch71r22 13 days ago

Is this a one-developer passion project? What is Stalwart Labs?

  • StalwartLabs 13 days ago

    For the moment, it's just one person (myself). I've been working on this project for the last year. I started a company in order to be able to receive donations and hopefully raise some money from YCombinator or VCs.

    My long term plan is to make an open source alternative to Google Workspace (not just e-mails and calendars but also Documents, Spreadsheets, etc) but I won't be able to do all that by myself so that is why I'll start looking for funding in the near future.

    • nicoburns 12 days ago

      Have you considered going the donations / sponsorship (e.g. patreon) route. This seems like it leads to much more sustainable open source projects than VC money which often leads to a project flaming out when the investors come to collect (or the company gets acquired).

      • StalwartLabs 11 days ago

        Yes, I have set up Github Sponsors already. Hopefully it won't be necessary to seek VC funding.

    • singhrac 13 days ago

      As a data point (probably closer to anecdote), I'd be willing to pay for just an email client. I don't know about recurring payments yet, but something like Superhuman-but-cheaper is definitely something I'd consider. My only real option right now is Outlook, I think.

    • ch71r22 13 days ago

      Super cool. Excited to follow along!

    • tpmx 13 days ago

      Why anonymous? You are after all asking strangers to run your code.

0xbadcafebee 13 days ago

I'd use it if you can remove the distributed part.

onebot 14 days ago

Nice work! Very impressive.

rch 13 days ago

Solid work, but I haven't seen too many AGPL projects prosper over time. I'll set a reminder to check back in a year or so to look for contributor growth.

  • capableweb 13 days ago

    Bitwarden, Ghostscript, GNUnet, Grafana, lichess, Mastodon, MongoDB, NextCloud, PeerTube, ScyllaDB, Signal and more all use AGPL. Seems there are plenty of examples.

    I think which license you use has less to do with number of contributors or "prosperity" than other things happening in the project.

    • rch 13 days ago

      Wow, I had no idea some of those names were under that license. Thanks!

      I'm even a Signal subscriber, and just didn't think about that aspect of their shared source model.

      I mentioned number of contributors as that attribute is generally predictive for longer running, healthy projects.

traceroute66 14 days ago

> release an SMTP server in Rust with the goal of making self-hosting an e-mail server much simpler.

Some might say that if someone can't setup Postfix, they've no business self-hosting an email server in the first place.

  • capableweb 14 days ago

    > Some might say that if someone can't setup Postfix, they've no business self-hosting an email server in the first place.

    Some might say that if you cannot setup Linux manually yourself (without any installers), they've no business running Linux. Also, you are not allowed to use a library unless you can manually put together the compile command without any Makefiles.

    Or you know, we can make the whole process easier with less pitfalls, so more people can enjoy the beauty of open ecosystems :)

  • calvinmorrison 13 days ago

    > Some might say that if someone can't setup Postfix, they've no business self-hosting an email server in the first place.

    In my mind, the reason I chose nginx every time wasn't because of performance, it was because I hated apache's config files. Ergonomics are important

  • WHATDOESIT 14 days ago

    Everybody needs an email server and selfhosting must be made as easy as possible.