I love Datomic, think it can be revolutionary and use it in my side projects. I understand their need to make a living off of their (brilliant, unique) labor and so keep it closed source. But the big bananas at my company are reluctant to try something so different. With Clojure I can push them over that hump since it's open source. But they are not willing to introduce proprietary lock-in of such a fundamental piece to an otherwise OSS stack so Datomic is out.
It would be nice if Cognitect could do even better with a Red Hat business model, selling support for an open source product. I don't know if that is plausible for Datomic or not. I suppose that's a function of how popular an open source version would be compared to the current version, and I can't even guess.
If Ions is successful enough that could be their profit source, allowing them to set standalone Datomic free.
The biggest thing I have noticed, granted I don't have a ton experience with either system, is point 1 outlined in the "Other Differences" section.
>Datomic’s datom model provides a very granular and comprehensive interface for expressing novelty through the assertion and retraction of facts. Crux instead uses documents (i.e. schemaless EDN maps) which are atomically ingested and processed as groups of facts that correspond to top-level fields with each document. This design choice simplifies bitemporal indexing (i.e. the use of valid time + transaction time coordinates) whilst satisfying typical requirements and improving the ergonomics of integration with other document-oriented systems. Additionally, the ordering of fields using the same key in a document is naturally preserved and can be readily retrieved, whereas Datomic requires explicit modelling of order for cardinality-many attributes. The main downside of Crux’s document model is that re-transacting entire documents to update a single field can be considered inefficient, but this could be mitigated using lower-level compression techniques and content-addressable storage. Retractions in Crux are implicit and deleted documents are simply replaced with empty documents
So what this means in pratice is that in Datomic you don't really "have" an entity/document (this isn't entirely true but feel free to dig into the details https://docs.datomic.com/on-prem/entities.html) but rather a set of facts (rows/tuples) at any given time, or across time, generally gathered up under one id. You can query on one of those facts and get exactly that back, one or more specific facts. Crux has this ability to query parts/attributes of a document but in terms of updating its all or nothing. There are other, more niche, considerations. One example I can can cite is looking at an entity through time in Datomic can be alien at first whereas with Crux its just a document, but overall this is the biggest thing I have noticed.
Oh and Datomic is pretty well battle tested and has the full support of Cognitect behind it whereas Crux is a newer OSS product by Juxt. That is both a pro and con for both camps, just depends on your needs.
I'm a big fan of Rich's work and I like (the idea of) Datomic a lot and while I don't begrudge their decision to keep it closed source, I personally won't use a database that isn't open source. Its simply too important a piece of any modern software and having only one company that can make fixes or changes if you need something is too risky for my taste. My priorities won't always align with Cognitect's, but if they're the only ones that can provide support, fixes or features then that is too much risk for me.
Having said that, I don't know what the solution is. A Red Hat model would be great, but how many companies have managed to replicate that success? The usual "commercial gives you these extra features" model means you need non-critical but nice-to-have features that people want to pay for but that doesn't cripple the open source version, not exactly ideal either. Having paid support and consulting as the primary income seems difficult too (eg RethinkDB). So, perhaps keeping it closed is the best route for them, I just wish there were better options. Oh well, I still get to enjoy the majority of Rich & team's work, so I'm not complaining.
> If Ions is successful enough that could be their profit source, allowing them to set standalone Datomic free.
My current job (primarily a Java shop on the backend) is the first where I've been able to introduce a little Clojure after years of having it relegated to being my 'side project' language.
Datomic looks neat from a distance, but not being open source means I'm probably not even going to look at it (since I wouldn't be able to use it in any case).
There’s an open source ‘version’ of datomic with a lot of the same features (Datalog, immutability, as-of queries etc) called Crux, I’ve played around with it on side projects and think it’s pretty awesome
In my experience, companies don't care about introducing proprietary lock-in. They do it all the time with services and infrastructure. It's all excuses. They just don't want to pay for software.
Paying for software is one thing. Paying for software where the amount you pay may be all but completely dictated by the supplier is another. If you decided to pay for a relational database, you could, with some work, swap vendors. If you paid for datomic, you could not. That makes it a non-starter imo as a technology decision. The business model here is naive and will change or datomic will always be super-niche. Maybe that's be design, but it's a little sad to me as it sounds like the tech could do some real good.
Of course they do. I'm criticising those companies that claim that they cannot use certain products because they're not open source, while not open sourcing their own. It is just an oxymoron for not reaching to the wallet.
Opensourcing sounds good. The problem is the right license choice. If they use something like Apache/LGPL "business friendly" license then soon or later Amazon will take the product, pack it into its cloud offering and Datomic will get nothing (MongoDB case). Maybe dual license GPL 3 or AGPL plus commercial would work, the question is how it would affect their business. Maybe if they are doing well with current model there is not incentive to change anything.
I wonder how hard it would be to adapt eg: postgres to keep history around forever. There's temporal tables in ANSI SQL 2011, but I think the semantics are different (it's per table, I believe).
As much as I like datomic, I'd prefer an implementation in a lower level language. However, I'm pretty sure there are alot of clojure experts out there pushing datomic to its limits.
It's kind of incredible how, over the course of two decades, being open source went from making something radioactive to business clients to being a requirement. I've seen it over and over again.
It's still radioactive, just to a different group of people now and weaponized (see Mongo, Elastic, et al).
I also think GPL compliance is a looming major problem for organizations and most haven't realized it yet. We have seen a few folks try to monetize GPL compliance the last few years...netfilter, I'm looking at you.
And this is why many of us prefer open source software to free software.
Yes, and that's what you were complaining about and why it seems that what you really prefer is permissive licenses over Copyleft licenses.
But open source software/licenses and free software/licenses are essentially identical sets (essentially all open source licenses, whether permissive or copyleft, are also free software licenses, and vice versa.) So preferring open source over free software is meaningless, they are the same thing.
> The OSI and FSF are separate organizations for a reason.
For political reasons that have nothing to do with what licenses meet the definitions each organization establishes.
I'm sorry but in the context of this "A vs B" discussion, anyone with sense knows exactly what point of distinction I'm talking about and perhaps my choice of words is because I don't want to give in to the FSF's chosen term of Copyleft but perhaps I didn't want to be completely incendiary in the discussion and call it what it is, a viral license, either.
You're both just choosing to be pedantic about it. I was just trying to be nice, but I guess that's not possible.
Also, from this wikipedia article here [1]:
"The Free Software Foundation has related but distinct criteria for evaluating whether or not a license qualifies software as free software. Most free software licenses are also considered open-source software licenses." According to them, A does not always equal B here.
> You're both just choosing to be pedantic about it. I was just trying to be nice, but I guess that's not possible.
Sorry, I had no intention of being rude; I just dislike the misconception that permissive licenses are not Free Software, because it turns people who dislike copyleft against the whole concept.
Hey, fair enough. I get ya, even. I think the copyleft folks have gone to lengths to co-opt the term "Free Software" to mean their thing and I'm fine to let them have it.
I'd like to see folks get more educated about what their license options are and the implications of using them in general.
I love Clojure and like Datomic, but that thing needs some documentation. Really good documentation. I think that is one of the main reasons why it didn't achieve much penetration for something so good. Be different if you must, but document. Look at something like Kafka, where every little thing is documented. You can find documentation for every shell script! that ships with it, not to mention excellent api docs. They also specify how the clients work in detail, not to mention how Kafka works. With datomic, even though everything works well, most of the time I feel like I am driving blind. I don't know if something I wrote works in every case.
I upvoted to counter because the morally hazardous incentive does exist with proprietary tech that you become super dependent on. This risk may not apply to the founders (Rich, who we all love), but to a future acquisitor like Oracle, IBM or AWS.
I predict that AWS will buy Cognitect. I think they are aiming for that exit and is part of why Datomic Cloud is so closely integrated with DynamoDB and other AWS services.
So Rich's talk about Datomic's architecture is taking place at Oracle? I love the radical openness. There's something cheeky about going to your competitor's office and inviting their engineers to learn about how your product works and how they can make better databases.
Having dealt with Datomic at work I really have come around to the way thinking that Datomic promotes. Having history that you can query along with having datalog and a graph database as opposed to a relational database has its benefits.
I'm still considering Datomic, or DataScript at least. I'm sold on concepts, but having hard time using it. Note that I don't have prior DB experience so YMMV.
The state of documentation is a dealbreaker for me. I could not understand how to use Datomic just from docs. It lacked code snippets, examples of different ways to use it.
The ClojureDocs solved this in a beautiful way with crowd-sourced collection of use-cases. ClojureDocs API pages manage to seamlessly present the core principles along with user-contributed annotated code snippets. Learning from examples is most effective way of learning.
have always wondered why this didn't succeed on a larger scale
Maybe in a v0 release, most applications can get away with created & modified fields. Maybe the storage overhead of immutability scares people for some applications.
Maybe datomic is something that should be part of a powerful DB plugins ecosystem rather than its own database, or as an external audit log that can be written separately while keeping your main DB in SQL.
It seems to me that you need to be all-in on Datomic to get value from it. So that knocks out anyone working with a legacy system.
You also can’t really hire Datomic people in bulk so that knocks out anyone trying to scale
their eng org.
I can see it working well for companies who are trying to do a lot with a small very bright staff, but that’s a small category. And even then it would only be places where the staff has consensus around immutability as a value.
Which... immutability, like many forms of strictness, is both a PITA and not actually that helpful unless you go all in.
It’s just a major commitment and for what benefit?
Maybe for a product where you need rewind everywhere.
Maybe for a product where you need extreme auditability.
I don’t know... what’s the killer feature that would make someone sacrifice everything else to use it?
Personally I think Datomic needs to be paired with a hosting system, debugging tools, basically some kind of integrated development environment that’s immutable-first in every respect.
Something like that would be powerfully differentiated.
You don't need to be all-in on Datomic, it's unsuitable for some kinds of data (eg binary blobs, or generally text due to the 4k string limit, PII due to missing delete support, etc). It seems designed to be used along with ancillary data stores.
I think the benefits have been pretty well laid out and are quite attractive, your restatement of them is just damnation with faint praise :)
I love Datomic, think it can be revolutionary and use it in my side projects. I understand their need to make a living off of their (brilliant, unique) labor and so keep it closed source. But the big bananas at my company are reluctant to try something so different. With Clojure I can push them over that hump since it's open source. But they are not willing to introduce proprietary lock-in of such a fundamental piece to an otherwise OSS stack so Datomic is out.
It would be nice if Cognitect could do even better with a Red Hat business model, selling support for an open source product. I don't know if that is plausible for Datomic or not. I suppose that's a function of how popular an open source version would be compared to the current version, and I can't even guess.
If Ions is successful enough that could be their profit source, allowing them to set standalone Datomic free.
While it isn't the same https://juxt.pro/crux/docs/index.html is a good compromise in the space and is open source albeit a very freshly released project. One of the major blockers I had to adoption, needing Kafka, is no longer an issue with the release of https://github.com/juxt/crux/tree/master/crux-jdbc. A great starting point if anyone wants to check Crux out at a distance is https://nextjournal.com/crux-tutorial, starting https://nextjournal.com/crux-tutorial/start
Could you say anything more at all about how Crux is a compromise compared to Datomic?
It seems to have a lot of what the latter has, as well as some features that it doesn't (e.g. it's bitemporal).
(I'm reading https://juxt.pro/crux/docs/faq.html but if there's anything you've seen that'd be super helpful)
The biggest thing I have noticed, granted I don't have a ton experience with either system, is point 1 outlined in the "Other Differences" section.
>Datomic’s datom model provides a very granular and comprehensive interface for expressing novelty through the assertion and retraction of facts. Crux instead uses documents (i.e. schemaless EDN maps) which are atomically ingested and processed as groups of facts that correspond to top-level fields with each document. This design choice simplifies bitemporal indexing (i.e. the use of valid time + transaction time coordinates) whilst satisfying typical requirements and improving the ergonomics of integration with other document-oriented systems. Additionally, the ordering of fields using the same key in a document is naturally preserved and can be readily retrieved, whereas Datomic requires explicit modelling of order for cardinality-many attributes. The main downside of Crux’s document model is that re-transacting entire documents to update a single field can be considered inefficient, but this could be mitigated using lower-level compression techniques and content-addressable storage. Retractions in Crux are implicit and deleted documents are simply replaced with empty documents
So what this means in pratice is that in Datomic you don't really "have" an entity/document (this isn't entirely true but feel free to dig into the details https://docs.datomic.com/on-prem/entities.html) but rather a set of facts (rows/tuples) at any given time, or across time, generally gathered up under one id. You can query on one of those facts and get exactly that back, one or more specific facts. Crux has this ability to query parts/attributes of a document but in terms of updating its all or nothing. There are other, more niche, considerations. One example I can can cite is looking at an entity through time in Datomic can be alien at first whereas with Crux its just a document, but overall this is the biggest thing I have noticed.
Oh and Datomic is pretty well battle tested and has the full support of Cognitect behind it whereas Crux is a newer OSS product by Juxt. That is both a pro and con for both camps, just depends on your needs.
I'm a big fan of Rich's work and I like (the idea of) Datomic a lot and while I don't begrudge their decision to keep it closed source, I personally won't use a database that isn't open source. Its simply too important a piece of any modern software and having only one company that can make fixes or changes if you need something is too risky for my taste. My priorities won't always align with Cognitect's, but if they're the only ones that can provide support, fixes or features then that is too much risk for me.
Having said that, I don't know what the solution is. A Red Hat model would be great, but how many companies have managed to replicate that success? The usual "commercial gives you these extra features" model means you need non-critical but nice-to-have features that people want to pay for but that doesn't cripple the open source version, not exactly ideal either. Having paid support and consulting as the primary income seems difficult too (eg RethinkDB). So, perhaps keeping it closed is the best route for them, I just wish there were better options. Oh well, I still get to enjoy the majority of Rich & team's work, so I'm not complaining.
> If Ions is successful enough that could be their profit source, allowing them to set standalone Datomic free.
That would be a great future.
My current job (primarily a Java shop on the backend) is the first where I've been able to introduce a little Clojure after years of having it relegated to being my 'side project' language.
Datomic looks neat from a distance, but not being open source means I'm probably not even going to look at it (since I wouldn't be able to use it in any case).
There’s an open source ‘version’ of datomic with a lot of the same features (Datalog, immutability, as-of queries etc) called Crux, I’ve played around with it on side projects and think it’s pretty awesome
Amazing how the world has changed. Not too long ago it was hard to introduce code that wasn't closed source.
In my experience, companies don't care about introducing proprietary lock-in. They do it all the time with services and infrastructure. It's all excuses. They just don't want to pay for software.
Paying for software is one thing. Paying for software where the amount you pay may be all but completely dictated by the supplier is another. If you decided to pay for a relational database, you could, with some work, swap vendors. If you paid for datomic, you could not. That makes it a non-starter imo as a technology decision. The business model here is naive and will change or datomic will always be super-niche. Maybe that's be design, but it's a little sad to me as it sounds like the tech could do some real good.
edit: typo
Lot’s of companies pay for software. E.g. Windows Server and SQL Server.
Of course they do. I'm criticising those companies that claim that they cannot use certain products because they're not open source, while not open sourcing their own. It is just an oxymoron for not reaching to the wallet.
Blame it on decades old experienced marketing and salesmen department at Microsoft. I'm not sure Cognitect has any firepower here.
Opensourcing sounds good. The problem is the right license choice. If they use something like Apache/LGPL "business friendly" license then soon or later Amazon will take the product, pack it into its cloud offering and Datomic will get nothing (MongoDB case). Maybe dual license GPL 3 or AGPL plus commercial would work, the question is how it would affect their business. Maybe if they are doing well with current model there is not incentive to change anything.
I think if it ever happens it'll be Eclipse Public License 1.0 like everything else clojure.
I wonder how hard it would be to adapt eg: postgres to keep history around forever. There's temporal tables in ANSI SQL 2011, but I think the semantics are different (it's per table, I believe).
As much as I like datomic, I'd prefer an implementation in a lower level language. However, I'm pretty sure there are alot of clojure experts out there pushing datomic to its limits.
It's kind of incredible how, over the course of two decades, being open source went from making something radioactive to business clients to being a requirement. I've seen it over and over again.
Actually it is more of having a generation that doesn't want to pay for their tooling, while expecting nonetheless to be paid for their work.
I keep working for customers where paying for tools is business as usual.
It's still radioactive, just to a different group of people now and weaponized (see Mongo, Elastic, et al).
I also think GPL compliance is a looming major problem for organizations and most haven't realized it yet. We have seen a few folks try to monetize GPL compliance the last few years...netfilter, I'm looking at you.
And this is why many of us prefer open source software to free software.
You mean you prefer permissive over copyleft. Most open source licenses are free software and vice-versa.
Sorry, but I don't agree. Copyleft software compels certain things.
There's a history here and important points of distinction. The OSI and FSF are separate organizations for a reason.
> Sorry, but I don't agree.
You seem to agree, though.
> Copyleft software compels certain things.
Yes, and that's what you were complaining about and why it seems that what you really prefer is permissive licenses over Copyleft licenses.
But open source software/licenses and free software/licenses are essentially identical sets (essentially all open source licenses, whether permissive or copyleft, are also free software licenses, and vice versa.) So preferring open source over free software is meaningless, they are the same thing.
> The OSI and FSF are separate organizations for a reason.
For political reasons that have nothing to do with what licenses meet the definitions each organization establishes.
I'm sorry but in the context of this "A vs B" discussion, anyone with sense knows exactly what point of distinction I'm talking about and perhaps my choice of words is because I don't want to give in to the FSF's chosen term of Copyleft but perhaps I didn't want to be completely incendiary in the discussion and call it what it is, a viral license, either.
You're both just choosing to be pedantic about it. I was just trying to be nice, but I guess that's not possible.
Also, from this wikipedia article here [1]: "The Free Software Foundation has related but distinct criteria for evaluating whether or not a license qualifies software as free software. Most free software licenses are also considered open-source software licenses." According to them, A does not always equal B here.
[1]: https://en.wikipedia.org/wiki/Open-source_license
> You're both just choosing to be pedantic about it. I was just trying to be nice, but I guess that's not possible.
Sorry, I had no intention of being rude; I just dislike the misconception that permissive licenses are not Free Software, because it turns people who dislike copyleft against the whole concept.
Hey, fair enough. I get ya, even. I think the copyleft folks have gone to lengths to co-opt the term "Free Software" to mean their thing and I'm fine to let them have it.
I'd like to see folks get more educated about what their license options are and the implications of using them in general.
I love Clojure and like Datomic, but that thing needs some documentation. Really good documentation. I think that is one of the main reasons why it didn't achieve much penetration for something so good. Be different if you must, but document. Look at something like Kafka, where every little thing is documented. You can find documentation for every shell script! that ships with it, not to mention excellent api docs. They also specify how the clients work in detail, not to mention how Kafka works. With datomic, even though everything works well, most of the time I feel like I am driving blind. I don't know if something I wrote works in every case.
Having great documentation would mean less sales in support contracts.
I upvoted to counter because the morally hazardous incentive does exist with proprietary tech that you become super dependent on. This risk may not apply to the founders (Rich, who we all love), but to a future acquisitor like Oracle, IBM or AWS.
I predict that AWS will buy Cognitect. I think they are aiming for that exit and is part of why Datomic Cloud is so closely integrated with DynamoDB and other AWS services.
So Rich's talk about Datomic's architecture is taking place at Oracle? I love the radical openness. There's something cheeky about going to your competitor's office and inviting their engineers to learn about how your product works and how they can make better databases.
Having dealt with Datomic at work I really have come around to the way thinking that Datomic promotes. Having history that you can query along with having datalog and a graph database as opposed to a relational database has its benefits.
There is an open source alternative to Datomic being currently developped: Datahike.
https://github.com/replikativ/datahike
The "Java" Youtube channel has a few "new" videos from the Summit in 2012 (e.g. on Graal, Truffle, Lambda Expressions):
https://www.oracle.com/technetwork/java/javase/community/jvm...
https://www.youtube.com/channel/UCmRtPmgnQ04CMUpSUqPfhxQ
I'm still considering Datomic, or DataScript at least. I'm sold on concepts, but having hard time using it. Note that I don't have prior DB experience so YMMV.
The state of documentation is a dealbreaker for me. I could not understand how to use Datomic just from docs. It lacked code snippets, examples of different ways to use it.
The ClojureDocs solved this in a beautiful way with crowd-sourced collection of use-cases. ClojureDocs API pages manage to seamlessly present the core principles along with user-contributed annotated code snippets. Learning from examples is most effective way of learning.
My favorite: https://clojuredocs.org/clojure.core/reduce
https://clojuredocs.org/examples-styleguide
I'm using the AWS hosted version of this (Datomic ions) to talk with the Monzo API it's been a lot of fun so far
have always wondered why this didn't succeed on a larger scale
Maybe in a v0 release, most applications can get away with created & modified fields. Maybe the storage overhead of immutability scares people for some applications.
Maybe datomic is something that should be part of a powerful DB plugins ecosystem rather than its own database, or as an external audit log that can be written separately while keeping your main DB in SQL.
It seems to me that you need to be all-in on Datomic to get value from it. So that knocks out anyone working with a legacy system.
You also can’t really hire Datomic people in bulk so that knocks out anyone trying to scale their eng org.
I can see it working well for companies who are trying to do a lot with a small very bright staff, but that’s a small category. And even then it would only be places where the staff has consensus around immutability as a value.
Which... immutability, like many forms of strictness, is both a PITA and not actually that helpful unless you go all in.
It’s just a major commitment and for what benefit?
Maybe for a product where you need rewind everywhere.
Maybe for a product where you need extreme auditability.
I don’t know... what’s the killer feature that would make someone sacrifice everything else to use it?
Personally I think Datomic needs to be paired with a hosting system, debugging tools, basically some kind of integrated development environment that’s immutable-first in every respect.
Something like that would be powerfully differentiated.
You don't need to be all-in on Datomic, it's unsuitable for some kinds of data (eg binary blobs, or generally text due to the 4k string limit, PII due to missing delete support, etc). It seems designed to be used along with ancillary data stores.
I think the benefits have been pretty well laid out and are quite attractive, your restatement of them is just damnation with faint praise :)
> Maybe for a product where you need rewind everywhere.
> Maybe for a product where you need extreme auditability.
These are "nice to have"s or "wish I had it but it's too late now"s for a lot of products though